MergeMDFiles: try a couple of things to speed it up

[Janik Zikovsky]

My starting point: 1157 seconds to merge 20 files made from TOPAZ_3007.nxs:

import time
LoadEventNexus(Filename='/home/8oz/data/TOPAZ_3007_event.nxs',OutputWorkspace='TOPAZ_3007_NXS',CompressTolerance='0.10000000000000001')

start = time.time();
for omega in xrange(0, 20):
	print "Starting omega %03d degrees" % omega
	CreateMDWorkspace(Dimensions='3',Extents='-10,10,-10,10,-10,10',Names='Q_sample_x,Q_sample_y,Q_sample_z',Units='A,A,A',SplitInto='5',SplitThreshold='2000',MaxRecursionDepth='3',
		MinRecursionDepth='3',	OutputWorkspace='TOPAZ_3007_MD')

	# Convert events to MD events
	AddSampleLog("TOPAZ_3007_NXS", "omega", "%s" % omega, "Number Series")
	ConvertToDiffractionMDWorkspace(InputWorkspace='TOPAZ_3007_NXS',OutputWorkspace='TOPAZ_3007_MD',OutputDimensions='Q (sample frame)',LorentzCorrection='1', Append='1')
	
	SaveMD("TOPAZ_3007_MD", "/home/8oz/data/rotated_md/TOPAZ_3007_rotated_%03d.nxs" % omega)
	print time.time() - start, " seconds since start."

A total of near 1 billion neutron events.

[Janik Zikovsky]

Refs #4794: addEventUnsafe() method on MDBoxes

To try to speed up MergeMDFiles, however my second try is 1172 seconds which is slightly slower than before.

Changeset: 7aa491f8c2191b003699a6d3e11a91cb2f074523

[Janik Zikovsky]

Refs #4794: Parallel flag for MergeMDFiles can be faster

I got 15.75 minutes (down from 19.5) using it.

Changeset: d3413b97ad3547704035da38df70d768361a3cb4

[Janik Zikovsky]

I think I had 25 minutes or so when the coord_t type was doubles (so files were about twice the size) so getting down to 15 minutes is pretty good. This compares well to Toby Perring's benchmark of 30 minutes for a billion events.

[Janik Zikovsky]

Refs #4794: Parallel flag for MergeMDFiles can be faster

I got 15.75 minutes (down from 19.5) using it.

Changeset: d3413b97ad3547704035da38df70d768361a3cb4

[Michael Reuter]

When I was testing another ticket on this subject, I got a rate of 1.2 million events per second. This would give 2.1 billion events in 30 minutes. Good!

[Stuart Campbell]

This ticket has been transferred to github issue ​5641