MergeMDFiles algorithm does not work.

[Alex Buts]

Trying to merge set of MD files obtained running ConvertToMD (events) on Chadwick (500G of RAM) with script, provided below.

First part of the script (converting particular files into MD and saving Nexus workspaces) works fine, but MergeMDFiles fails.

Script reports 35,770,649 events (*28Bytes per event ~1G). Script reports incorrectly. It is about 1Tb events in this file.

After running Mantid for 6 hours and taking 90% of memory on Chadwick, Mantid reports 50% job done but no any target file present.

The algorithm seems just does not even try to do what it says it doing.

Build SQW script deploying MergeMDFiles:

from os.path import exists
print 'start convert to SQW\n'
cur_ws='wsn';
md_ws='mdws4D';
saveTumbTo='/home/wkc26243/ProcessedData/'
MDWSF='';

ic=0;
file_nums = range(15618,15834);
for n in file_nums:
	MD_Fname = 'MDMAP'+str(n)+'.nxs';
	ic = ic+1;
	if not(exists(saveTumbTo+MD_Fname)):
		inFile = 'MAP'+str(n)+'.nxspe';
		print 'processing file: #'+str(ic)+':of:'+str(len(file_nums)),'name: '+inFile		
		LoadNXSPE(Filename=inFile,OutputWorkspace=cur_ws)		
		SetUB(Workspace=cur_ws,a='2.87',b='2.87',c='2.87')
		print 'converting to md workspace '
		ConvertToMDEvents(InputWorkspace=cur_ws,OutputWorkspace=md_ws,QDimensions='Q3D',UsePreprocessedDetectors='1',MinValues='-4.2,-4.2,-4.2,-50',MaxValues='4.2,4.2,4.2,350',SplitInto='50',MaxRecursionDepth='1',MinRecursionDepth='1')
	        print 'saving to md file: '+MD_Fname
		SaveMD(md_ws,Filename=saveTumbTo+MD_Fname);
		DeleteWorkspace(md_ws);
		DeleteWorkspace(cur_ws);   
		print '\n'
	# buld filename list to merge in merge algorithm
	if (len(MDWSF) == 0):
		MDWSF = saveTumbTo+MD_Fname;
	else:
		MDWSF=MDWSF+','+saveTumbTo+MD_Fname;
	
print 'Merging '+str(len(file_nums))+' MD files:'
MergeMDFiles(MDWSF,OutputFilename='fe1400_8k.nxs',OutputWorkspace='WS4D',Parallel='0');

Homer script, which used to produce the nxspe files, used as source files in the conversion to MD. The homer script uses Illiad routine, available in qtiGenie in Mantid Users indirect scripts

from qtiGenie import *
iliad_setup('MAP')


#print 'SL:',sl
#
#
mapfile ="/home/wkc26243/ProcessedData/Instrument/4to1"
mskfile  ="/home/wkc26243/ProcessedData/Instrument/4to1_022.msk"

bins     ='-100, 2, 1352'
ei        = 1400
LoadRaw(Filename="MAP15527.raw",OutputWorkspace="wb_wksp")
ic=0;
for n in range(15618,15665):
	in_file = 'MAP'+str(n)+'.RAW';
	LoadRaw(Filename=in_file,OutputWorkspace="run_wksp")
	print '----->\n'
	w1=iliad("wb_wksp","run_wksp",ei,bins,mapfile,det_cal_file=in_file,fixei=False,norm_method='monitor-1',bkgd_range=[12000,18000],diag_sigma=3,diag_remove_zero=False)
	SaveNXSPE(w1,'MAP'+str(n)+'.nxspe',Psi=ic)
	ic=ic+1;

ic=0;
for n in range(15665,15711):
	in_file = 'MAP'+str(n)+'.RAW';
	LoadRaw(Filename=in_file,OutputWorkspace="run_wksp")
        print n,':----->\n'
	w1=iliad("wb_wksp","run_wksp",ei,bins,mapfile,det_cal_file=in_file,fixei=False,norm_method='monitor-1',bkgd_range=[12000,18000],diag_sigma=3,diag_remove_zero=False)
	SaveNXSPE(w1,'MAP'+str(n)+'.nxspe',Psi=ic)
	ic=ic-1;

ic=-44.5;
for n in range(15711,15740):
	in_file = 'MAP'+str(n)+'.RAW';
	LoadRaw(Filename=in_file,OutputWorkspace="run_wksp")
	print n,':----->\n'
	w1=iliad("wb_wksp","run_wksp",ei,bins,mapfile,det_cal_file=in_file,fixei=False,norm_method='monitor-1',bkgd_range=[12000,18000],diag_sigma=3,diag_remove_zero=False)
	SaveNXSPE(w1,'MAP'+str(n)+'.nxspe',Psi=ic)
	ic=ic+1;

ic=-15.5;
for n in range(15740,15801):
	in_file = 'MAP'+str(n)+'.RAW';
	LoadRaw(Filename=in_file,OutputWorkspace="run_wksp")
	print n,':----->\n'
	w1=iliad("wb_wksp","run_wksp",ei,bins,mapfile,det_cal_file=in_file,fixei=False,norm_method='monitor-1',bkgd_range=[12000,18000],diag_sigma=3,diag_remove_zero=False)
	SaveNXSPE(w1,'MAP'+str(n)+'.nxspe',Psi=ic)
	ic=ic+1;

ic=-31.5;
for n in range(15801,15816):
	in_file = 'MAP'+str(n)+'.RAW';
	LoadRaw(Filename=in_file,OutputWorkspace="run_wksp")
	print n,':----->\n'
	w1=iliad("wb_wksp","run_wksp",ei,bins,mapfile,det_cal_file=in_file,fixei=False,norm_method='monitor-1',bkgd_range=[12000,18000],diag_sigma=3,diag_remove_zero=False)
	SaveNXSPE(w1,'MAP'+str(n)+'.nxspe',Psi=ic)
	ic=ic+1;
	
ic=0;
for n in range(15816,15835):
	in_file = 'MAP'+str(n)+'.RAW';
	LoadRaw(Filename=in_file,OutputWorkspace="run_wksp")
	print n,':----->\n'
	w1=iliad("wb_wksp","run_wksp",ei,bins,mapfile,det_cal_file=in_file,fixei=False,norm_method='monitor-1',bkgd_range=[12000,18000],diag_sigma=3,diag_remove_zero=False)
	SaveNXSPE(w1,'MAP'+str(n)+'.nxspe',Psi=ic)
	ic=ic+1;
#
print "All done\n"

[Nick Draper]

moved to Release 2.4

[Alex Buts]

refs #5849 better regular points distribution from defaults.

Changeset: 27dae4e56d3fc6599f1d190d2d606d7edd917f41

[Alex Buts]

refs #5849 Better fake regular data

Changeset: 1827e91620c06ba845333c205a066b4744bab8b6

[Alex Buts]

refs #5849 Small changes to simplify MBBox structure->

flat box representation

Changeset: aa9fad78017d4f7cca5914d2af211e6f71eb9998

[Alex Buts]

refs #5849 reverted to const methods changing the class contents

extremely bad practice but have to keep it this way until good way of refactoring is identified.

Changeset: 1fc0ebcf895645b8bb7df719a3a416c423bb458d

[Alex Buts]

refs #5849 better regular points distribution from defaults.

Changeset: 1511574f46fecf136b79e7c532eadac046304eab

[Alex Buts]

refs #5849 Better fake regular data

Changeset: 5780c74186be4a175eb7e4e69187b6cf9c795664

[Alex Buts]

refs #5849 Small changes to simplify MBBox structure->

flat box representation

Changeset: e3c5a58f12d0791f6bd633d9f7aeeeb86168b5b1

[Alex Buts]

refs #5849 reverted to const methods changing the class contents

extremely bad practice but have to keep it this way until good way of refactoring is identified.

Changeset: da19b2b053ed3d478812b143c2189ad4ce94ea43

[Alex Buts]

refs #5849 better regular points distribution from defaults.

Changeset: 8c1e0b4be4c3fda0272a07bdb1c3505be8d23f19

[Alex Buts]

refs #5849 Small changes to simplify MBBox structure->

flat box representation

Changeset: 49e330d0925ed085a7e542331dc6f639bce49fa7

[Alex Buts]

refs #5849 better regular points distribution from defaults.

Changeset: 1e5e7f4b742fca6386bea97d17fbb189cf152ad6

[Alex Buts]

refs #5849 Small changes to simplify MBBox structure->

flat box representation

Changeset: 541689b7399e5d0fed9ca45a397dbe5104a203e0

[Alex Buts]

refs #5849 Added two files which is basis for common MD events class.

Changeset: 3ae6e69ddc9da14f53f33e775d9d433f3bb27ce2

[Alex Buts]

refs #5849 tests for flatTree and one useful? function

Changeset: 14fa0e3e97b5bf0a434fdfa6000786b43cdaf623

[Alex Buts]

refs #5849 Main part of SaveMD extracted into the FlatBoxTree class

Changeset: 424abc8897a02783e4b09cbad3ec0578ee04138f

[Alex Buts]

refs #5849 Both Save and Load MD work on the basis of FlatTree class

Changeset: fbe09caa5ebbf55ed02692c7592540c9deeba6a4

[Alex Buts]

refs #5849 Hopefully it fixes it.

Changeset: 457fa51c356a4a5bf1748d4408c7fa5fab60f7c3

[Alex Buts]

refs #5849 Fixed induced error.

Changeset: d770c3f1fe17cf274ce447e7cc8c10435c2777cc

[Alex Buts]

This ticket involved substantial changes to the structure of MD workspaces.

The changes involve all Load/Save/Merge MD algorithms so to test this ticket properly one has to look to all tests and check how file access to MD workspaces works and if there are any new unpleasant changes introduced.

To make formal test, just run testing script above, though I have already run it on windows 7 machine

[Alex Buts]

refs #5849 fixing Unix error with headers

Changeset: 92e968e018fa8505f20e668c02258c55628a09d3

[Alex Buts]

refs #5849 stupid syntax error

Changeset: b1943d259e2b05da4402045ee0248ac7f1c2ea3c

[Alex Buts]

refs #5849 fixing Unix warnings (and questioning tests)

Changeset: e50e4a5d77c396a6d4aecff8730f1904c33e8967

[Alex Buts]

refs #5849 One sanity check in MDBoxFlatTree.cpp

Changeset: 28ee7be0baaee5d032f93da35fa3a8679e76ba47

[Alex Buts]

refs #5849 Temporary disabling some failing tests

Changeset: 3d4ba9b98fe2469366212e4674e34ff79a6a2898

[Alex Buts]

refs #5849 attempt to disable failing tests

Changeset: e0dbabd0d4f686280ee02a959658462adbdd993f

[Alex Buts]

refs #5849 disabling failing tests

Changeset: b9f5254bff8021bd674a389108ed83bb673c76a6

[Martyn Gigg]

Fix build on systems where uint64_t != size_t Refs #5849

Changeset: 544fd2e598a0f2159e98b385899662be44207136

[Alex Buts]

refs #5849 filename made class parameter trying to fix problem

on build servers

Changeset: 6dfb30c9874d3d15602bd0fafacfe82e04bb37ac

[Alex Buts]

refs #5849 filename made class parameter trying to fix problem

on build servers

Changeset: 5223bd8846231b745ded5b65f85e591b5c8cd58e

[Alex Buts]

refs #5849 enable all disabled tests

as seems the previous commit have fixed all problem

Changeset: a0859076c6873eab57e8d59291189e48d33fa47f

[Nick Draper]

Tested with 3 files on my laptop win64

from os.path import exists
print 'start convert to SQW\n'
cur_ws='wsn';
md_ws='mdws4D';
saveTumbTo='c:/mantidinstall/data/'
MDWSF='';

ic=0;
file_nums = range(15628,15631);
for n in file_nums:
	MD_Fname = 'MDMAP'+str(n)+'.nxs';
	ic = ic+1;
	if not(exists(saveTumbTo+MD_Fname)):
		inFile = 'MAP'+str(n)+'.nxspe';
		print 'processing file: #'+str(ic)+':of:'+str(len(file_nums)),'name: '+inFile		
		LoadNXSPE(Filename=inFile,OutputWorkspace=cur_ws)		
		SetUB(Workspace=cur_ws,a='2.87',b='2.87',c='2.87')
		print 'converting to md workspace '
		ConvertToMDEvents(InputWorkspace=cur_ws,OutputWorkspace=md_ws,QDimensions='Q3D',UsePreprocessedDetectors='1',MinValues='-4.2,-4.2,-4.2,-50',MaxValues='4.2,4.2,4.2,350',SplitInto='50',MaxRecursionDepth='1',MinRecursionDepth='1')
	        print 'saving to md file: '+MD_Fname
		SaveMD(md_ws,Filename=saveTumbTo+MD_Fname);
		DeleteWorkspace(md_ws);
		DeleteWorkspace(cur_ws);   
		print '\n'
	# buld filename list to merge in merge algorithm
	if (len(MDWSF) == 0):
		MDWSF = saveTumbTo+MD_Fname;
	else:
		MDWSF=MDWSF+','+saveTumbTo+MD_Fname;
	
print 'Merging '+str(len(file_nums))+' MD files:'
MergeMDFiles(MDWSF,OutputFilename='MDResult.nxs',OutputWorkspace='WS4D',Parallel='0');

---------------------------------------------------------------------------
Tue 29. Jan 17:48:54 2013: Script execution started.
---------------------------------------------------------------------------
start convert to SQW

processing file: #1:of:3 name: MAP15628.nxspe
converting to md workspace 
saving to md file: MDMAP15628.nxs


processing file: #2:of:3 name: MAP15629.nxspe
converting to md workspace 
saving to md file: MDMAP15629.nxs


processing file: #3:of:3 name: MAP15630.nxspe
converting to md workspace 
saving to md file: MDMAP15630.nxs


Merging 3 MD files:
---------------------------------------------------------------------------
Tue 29. Jan 17:52:04 2013: Script execution finished.
---------------------------------------------------------------------------

Welcome to Mantid - Manipulation and Analysis Toolkit for Instrument Data
ParaView is not available
LoadNXSPE started
LoadNXSPE successful, Duration 6.45 seconds
SetUB started
SetUB successful, Duration 0.00 seconds
ConvertToMDEvents is deprecated. Use ConvertToMD instead.
ConvertToMDEvents started
ConvertToMDEvents successful, Duration 10.53 seconds
SaveMD started
SaveMD successful, Duration 5.95 seconds
DeleteWorkspace started
DeleteWorkspace successful, Duration 0.00 seconds
DeleteWorkspace started
DeleteWorkspace successful, Duration 0.00 seconds
LoadNXSPE started
LoadNXSPE successful, Duration 8.00 seconds
SetUB started
SetUB successful, Duration 0.00 seconds
ConvertToMDEvents is deprecated. Use ConvertToMD instead.
ConvertToMDEvents started
ConvertToMDEvents successful, Duration 10.49 seconds
SaveMD started
SaveMD successful, Duration 11.42 seconds
DeleteWorkspace started
DeleteWorkspace successful, Duration 0.00 seconds
DeleteWorkspace started
DeleteWorkspace successful, Duration 0.00 seconds
LoadNXSPE started
LoadNXSPE successful, Duration 7.97 seconds
SetUB started
SetUB successful, Duration 0.00 seconds
ConvertToMDEvents is deprecated. Use ConvertToMD instead.
ConvertToMDEvents started
ConvertToMDEvents successful, Duration 10.56 seconds
SaveMD started
SaveMD successful, Duration 15.89 seconds
DeleteWorkspace started
DeleteWorkspace successful, Duration 0.00 seconds
DeleteWorkspace started
DeleteWorkspace successful, Duration 0.00 seconds
MergeMDFiles started
Error loading instrument IDF file for 'NXSPE'.
Instrument search object NXSPE
Setting cache to 400 MB write.
4238571 events in 3 files.
Starting SaveMD to update the file back-end.
MergeMDFiles successful, Duration 88.59 seconds

[Alex Buts]

refs #5849 filename made class parameter trying to fix problem

on build servers

Changeset: 5223bd8846231b745ded5b65f85e591b5c8cd58e

[Alex Buts]

refs #5849 enable all disabled tests

as seems the previous commit have fixed all problem

Changeset: a0859076c6873eab57e8d59291189e48d33fa47f

[Alex Buts]

refs #5849 Added initial MDBoxFlatTreeTest to the build

Changeset: c6925a42ca0d039b67d75acb219085fbce5adbe5

[Stuart Campbell]

This ticket has been transferred to github issue ​6695