Ticket #5849 (closed: fixed)
MergeMDFiles algorithm does not work.
Reported by: | Alex Buts | Owned by: | Alex Buts |
---|---|---|---|
Priority: | blocker | Milestone: | Release 2.4 |
Component: | VATES | Keywords: | |
Cc: | Toby.Perring@… | Blocked By: | |
Blocking: | Tester: | Nick Draper |
Description (last modified by Alex Buts) (diff)
Trying to merge set of MD files obtained running ConvertToMD (events) on Chadwick (500G of RAM) with script, provided below.
First part of the script (converting particular files into MD and saving Nexus workspaces) works fine, but MergeMDFiles fails.
Script reports 35,770,649 events (*28Bytes per event ~1G). Script reports incorrectly. It is about 1Tb events in this file.
After running Mantid for 6 hours and taking 90% of memory on Chadwick, Mantid reports 50% job done but no any target file present.
The algorithm seems just does not even try to do what it says it doing.
Build SQW script deploying MergeMDFiles:
from os.path import exists print 'start convert to SQW\n' cur_ws='wsn'; md_ws='mdws4D'; saveTumbTo='/home/wkc26243/ProcessedData/' MDWSF=''; ic=0; file_nums = range(15618,15834); for n in file_nums: MD_Fname = 'MDMAP'+str(n)+'.nxs'; ic = ic+1; if not(exists(saveTumbTo+MD_Fname)): inFile = 'MAP'+str(n)+'.nxspe'; print 'processing file: #'+str(ic)+':of:'+str(len(file_nums)),'name: '+inFile LoadNXSPE(Filename=inFile,OutputWorkspace=cur_ws) SetUB(Workspace=cur_ws,a='2.87',b='2.87',c='2.87') print 'converting to md workspace ' ConvertToMDEvents(InputWorkspace=cur_ws,OutputWorkspace=md_ws,QDimensions='Q3D',UsePreprocessedDetectors='1',MinValues='-4.2,-4.2,-4.2,-50',MaxValues='4.2,4.2,4.2,350',SplitInto='50',MaxRecursionDepth='1',MinRecursionDepth='1') print 'saving to md file: '+MD_Fname SaveMD(md_ws,Filename=saveTumbTo+MD_Fname); DeleteWorkspace(md_ws); DeleteWorkspace(cur_ws); print '\n' # buld filename list to merge in merge algorithm if (len(MDWSF) == 0): MDWSF = saveTumbTo+MD_Fname; else: MDWSF=MDWSF+','+saveTumbTo+MD_Fname; print 'Merging '+str(len(file_nums))+' MD files:' MergeMDFiles(MDWSF,OutputFilename='fe1400_8k.nxs',OutputWorkspace='WS4D',Parallel='0');
Homer script, which used to produce the nxspe files, used as source files in the conversion to MD. The homer script uses Illiad routine, available in qtiGenie in Mantid Users indirect scripts
from qtiGenie import * iliad_setup('MAP') #print 'SL:',sl # # mapfile ="/home/wkc26243/ProcessedData/Instrument/4to1" mskfile ="/home/wkc26243/ProcessedData/Instrument/4to1_022.msk" bins ='-100, 2, 1352' ei = 1400 LoadRaw(Filename="MAP15527.raw",OutputWorkspace="wb_wksp") ic=0; for n in range(15618,15665): in_file = 'MAP'+str(n)+'.RAW'; LoadRaw(Filename=in_file,OutputWorkspace="run_wksp") print '----->\n' w1=iliad("wb_wksp","run_wksp",ei,bins,mapfile,det_cal_file=in_file,fixei=False,norm_method='monitor-1',bkgd_range=[12000,18000],diag_sigma=3,diag_remove_zero=False) SaveNXSPE(w1,'MAP'+str(n)+'.nxspe',Psi=ic) ic=ic+1; ic=0; for n in range(15665,15711): in_file = 'MAP'+str(n)+'.RAW'; LoadRaw(Filename=in_file,OutputWorkspace="run_wksp") print n,':----->\n' w1=iliad("wb_wksp","run_wksp",ei,bins,mapfile,det_cal_file=in_file,fixei=False,norm_method='monitor-1',bkgd_range=[12000,18000],diag_sigma=3,diag_remove_zero=False) SaveNXSPE(w1,'MAP'+str(n)+'.nxspe',Psi=ic) ic=ic-1; ic=-44.5; for n in range(15711,15740): in_file = 'MAP'+str(n)+'.RAW'; LoadRaw(Filename=in_file,OutputWorkspace="run_wksp") print n,':----->\n' w1=iliad("wb_wksp","run_wksp",ei,bins,mapfile,det_cal_file=in_file,fixei=False,norm_method='monitor-1',bkgd_range=[12000,18000],diag_sigma=3,diag_remove_zero=False) SaveNXSPE(w1,'MAP'+str(n)+'.nxspe',Psi=ic) ic=ic+1; ic=-15.5; for n in range(15740,15801): in_file = 'MAP'+str(n)+'.RAW'; LoadRaw(Filename=in_file,OutputWorkspace="run_wksp") print n,':----->\n' w1=iliad("wb_wksp","run_wksp",ei,bins,mapfile,det_cal_file=in_file,fixei=False,norm_method='monitor-1',bkgd_range=[12000,18000],diag_sigma=3,diag_remove_zero=False) SaveNXSPE(w1,'MAP'+str(n)+'.nxspe',Psi=ic) ic=ic+1; ic=-31.5; for n in range(15801,15816): in_file = 'MAP'+str(n)+'.RAW'; LoadRaw(Filename=in_file,OutputWorkspace="run_wksp") print n,':----->\n' w1=iliad("wb_wksp","run_wksp",ei,bins,mapfile,det_cal_file=in_file,fixei=False,norm_method='monitor-1',bkgd_range=[12000,18000],diag_sigma=3,diag_remove_zero=False) SaveNXSPE(w1,'MAP'+str(n)+'.nxspe',Psi=ic) ic=ic+1; ic=0; for n in range(15816,15835): in_file = 'MAP'+str(n)+'.RAW'; LoadRaw(Filename=in_file,OutputWorkspace="run_wksp") print n,':----->\n' w1=iliad("wb_wksp","run_wksp",ei,bins,mapfile,det_cal_file=in_file,fixei=False,norm_method='monitor-1',bkgd_range=[12000,18000],diag_sigma=3,diag_remove_zero=False) SaveNXSPE(w1,'MAP'+str(n)+'.nxspe',Psi=ic) ic=ic+1; # print "All done\n"
Change History
comment:3 Changed 8 years ago by Alex Buts
- Owner set to Alex Buts
- Status changed from new to accepted
comment:4 Changed 8 years ago by Nick Draper
- Milestone changed from Release 2.3 to Release 2.4
moved to Release 2.4
comment:6 Changed 8 years ago by Alex Buts
refs #5849 better regular points distribution from defaults.
Changeset: 27dae4e56d3fc6599f1d190d2d606d7edd917f41
comment:7 Changed 8 years ago by Alex Buts
refs #5849 Better fake regular data
Changeset: 1827e91620c06ba845333c205a066b4744bab8b6
comment:8 Changed 8 years ago by Alex Buts
refs #5849 Small changes to simplify MBBox structure->
flat box representation
Changeset: aa9fad78017d4f7cca5914d2af211e6f71eb9998
comment:9 Changed 8 years ago by Alex Buts
refs #5849 reverted to const methods changing the class contents
extremely bad practice but have to keep it this way until good way of refactoring is identified.
Changeset: 1fc0ebcf895645b8bb7df719a3a416c423bb458d
comment:10 Changed 8 years ago by Alex Buts
refs #5849 better regular points distribution from defaults.
Changeset: 1511574f46fecf136b79e7c532eadac046304eab
comment:11 Changed 8 years ago by Alex Buts
refs #5849 Better fake regular data
Changeset: 5780c74186be4a175eb7e4e69187b6cf9c795664
comment:12 Changed 8 years ago by Alex Buts
refs #5849 Small changes to simplify MBBox structure->
flat box representation
Changeset: e3c5a58f12d0791f6bd633d9f7aeeeb86168b5b1
comment:13 Changed 8 years ago by Alex Buts
refs #5849 reverted to const methods changing the class contents
extremely bad practice but have to keep it this way until good way of refactoring is identified.
Changeset: da19b2b053ed3d478812b143c2189ad4ce94ea43
comment:14 Changed 8 years ago by Alex Buts
refs #5849 better regular points distribution from defaults.
Changeset: 8c1e0b4be4c3fda0272a07bdb1c3505be8d23f19
comment:15 Changed 8 years ago by Alex Buts
refs #5849 Small changes to simplify MBBox structure->
flat box representation
Changeset: 49e330d0925ed085a7e542331dc6f639bce49fa7
comment:16 Changed 8 years ago by Alex Buts
refs #5849 better regular points distribution from defaults.
Changeset: 1e5e7f4b742fca6386bea97d17fbb189cf152ad6
comment:17 Changed 8 years ago by Alex Buts
refs #5849 Small changes to simplify MBBox structure->
flat box representation
Changeset: 541689b7399e5d0fed9ca45a397dbe5104a203e0
comment:18 Changed 8 years ago by Alex Buts
- Priority changed from major to blocker
- Blocked By 6362 removed
comment:19 Changed 8 years ago by Alex Buts
refs #5849 Added two files which is basis for common MD events class.
Changeset: 3ae6e69ddc9da14f53f33e775d9d433f3bb27ce2
comment:20 Changed 8 years ago by Alex Buts
refs #5849 tests for flatTree and one useful? function
Changeset: 14fa0e3e97b5bf0a434fdfa6000786b43cdaf623
comment:21 Changed 8 years ago by Alex Buts
refs #5849 Main part of SaveMD extracted into the FlatBoxTree class
Changeset: 424abc8897a02783e4b09cbad3ec0578ee04138f
comment:22 Changed 8 years ago by Alex Buts
refs #5849 Both Save and Load MD work on the basis of FlatTree class
Changeset: fbe09caa5ebbf55ed02692c7592540c9deeba6a4
comment:23 Changed 8 years ago by Alex Buts
refs #5849 Hopefully it fixes it.
Changeset: 457fa51c356a4a5bf1748d4408c7fa5fab60f7c3
comment:24 Changed 8 years ago by Alex Buts
refs #5849 Fixed induced error.
Changeset: d770c3f1fe17cf274ce447e7cc8c10435c2777cc
comment:25 Changed 8 years ago by Alex Buts
- Status changed from accepted to verify
- Resolution set to fixed
This ticket involved substantial changes to the structure of MD workspaces.
The changes involve all Load/Save/Merge MD algorithms so to test this ticket properly one has to look to all tests and check how file access to MD workspaces works and if there are any new unpleasant changes introduced.
To make formal test, just run testing script above, though I have already run it on windows 7 machine
comment:26 Changed 8 years ago by Alex Buts
refs #5849 fixing Unix error with headers
Changeset: 92e968e018fa8505f20e668c02258c55628a09d3
comment:27 Changed 8 years ago by Alex Buts
refs #5849 stupid syntax error
Changeset: b1943d259e2b05da4402045ee0248ac7f1c2ea3c
comment:28 Changed 8 years ago by Alex Buts
refs #5849 fixing Unix warnings (and questioning tests)
Changeset: e50e4a5d77c396a6d4aecff8730f1904c33e8967
comment:29 Changed 8 years ago by Alex Buts
refs #5849 One sanity check in MDBoxFlatTree.cpp
Changeset: 28ee7be0baaee5d032f93da35fa3a8679e76ba47
comment:30 Changed 8 years ago by Alex Buts
refs #5849 Temporary disabling some failing tests
Changeset: 3d4ba9b98fe2469366212e4674e34ff79a6a2898
comment:31 Changed 8 years ago by Alex Buts
refs #5849 attempt to disable failing tests
Changeset: e0dbabd0d4f686280ee02a959658462adbdd993f
comment:32 Changed 8 years ago by Alex Buts
refs #5849 disabling failing tests
Changeset: b9f5254bff8021bd674a389108ed83bb673c76a6
comment:33 Changed 8 years ago by Martyn Gigg
Fix build on systems where uint64_t != size_t Refs #5849
Changeset: 544fd2e598a0f2159e98b385899662be44207136
comment:34 Changed 8 years ago by Alex Buts
refs #5849 filename made class parameter trying to fix problem
on build servers
Changeset: 6dfb30c9874d3d15602bd0fafacfe82e04bb37ac
comment:35 Changed 8 years ago by Alex Buts
refs #5849 filename made class parameter trying to fix problem
on build servers
Changeset: 5223bd8846231b745ded5b65f85e591b5c8cd58e
comment:36 Changed 8 years ago by Alex Buts
refs #5849 enable all disabled tests
as seems the previous commit have fixed all problem
Changeset: a0859076c6873eab57e8d59291189e48d33fa47f
comment:37 Changed 8 years ago by Nick Draper
- Status changed from verify to verifying
- Tester set to Nick Draper
comment:38 Changed 8 years ago by Nick Draper
- Status changed from verifying to closed
Tested with 3 files on my laptop win64
from os.path import exists print 'start convert to SQW\n' cur_ws='wsn'; md_ws='mdws4D'; saveTumbTo='c:/mantidinstall/data/' MDWSF=''; ic=0; file_nums = range(15628,15631); for n in file_nums: MD_Fname = 'MDMAP'+str(n)+'.nxs'; ic = ic+1; if not(exists(saveTumbTo+MD_Fname)): inFile = 'MAP'+str(n)+'.nxspe'; print 'processing file: #'+str(ic)+':of:'+str(len(file_nums)),'name: '+inFile LoadNXSPE(Filename=inFile,OutputWorkspace=cur_ws) SetUB(Workspace=cur_ws,a='2.87',b='2.87',c='2.87') print 'converting to md workspace ' ConvertToMDEvents(InputWorkspace=cur_ws,OutputWorkspace=md_ws,QDimensions='Q3D',UsePreprocessedDetectors='1',MinValues='-4.2,-4.2,-4.2,-50',MaxValues='4.2,4.2,4.2,350',SplitInto='50',MaxRecursionDepth='1',MinRecursionDepth='1') print 'saving to md file: '+MD_Fname SaveMD(md_ws,Filename=saveTumbTo+MD_Fname); DeleteWorkspace(md_ws); DeleteWorkspace(cur_ws); print '\n' # buld filename list to merge in merge algorithm if (len(MDWSF) == 0): MDWSF = saveTumbTo+MD_Fname; else: MDWSF=MDWSF+','+saveTumbTo+MD_Fname; print 'Merging '+str(len(file_nums))+' MD files:' MergeMDFiles(MDWSF,OutputFilename='MDResult.nxs',OutputWorkspace='WS4D',Parallel='0');
--------------------------------------------------------------------------- Tue 29. Jan 17:48:54 2013: Script execution started. --------------------------------------------------------------------------- start convert to SQW processing file: #1:of:3 name: MAP15628.nxspe converting to md workspace saving to md file: MDMAP15628.nxs processing file: #2:of:3 name: MAP15629.nxspe converting to md workspace saving to md file: MDMAP15629.nxs processing file: #3:of:3 name: MAP15630.nxspe converting to md workspace saving to md file: MDMAP15630.nxs Merging 3 MD files: --------------------------------------------------------------------------- Tue 29. Jan 17:52:04 2013: Script execution finished. ---------------------------------------------------------------------------
Welcome to Mantid - Manipulation and Analysis Toolkit for Instrument Data ParaView is not available LoadNXSPE started LoadNXSPE successful, Duration 6.45 seconds SetUB started SetUB successful, Duration 0.00 seconds ConvertToMDEvents is deprecated. Use ConvertToMD instead. ConvertToMDEvents started ConvertToMDEvents successful, Duration 10.53 seconds SaveMD started SaveMD successful, Duration 5.95 seconds DeleteWorkspace started DeleteWorkspace successful, Duration 0.00 seconds DeleteWorkspace started DeleteWorkspace successful, Duration 0.00 seconds LoadNXSPE started LoadNXSPE successful, Duration 8.00 seconds SetUB started SetUB successful, Duration 0.00 seconds ConvertToMDEvents is deprecated. Use ConvertToMD instead. ConvertToMDEvents started ConvertToMDEvents successful, Duration 10.49 seconds SaveMD started SaveMD successful, Duration 11.42 seconds DeleteWorkspace started DeleteWorkspace successful, Duration 0.00 seconds DeleteWorkspace started DeleteWorkspace successful, Duration 0.00 seconds LoadNXSPE started LoadNXSPE successful, Duration 7.97 seconds SetUB started SetUB successful, Duration 0.00 seconds ConvertToMDEvents is deprecated. Use ConvertToMD instead. ConvertToMDEvents started ConvertToMDEvents successful, Duration 10.56 seconds SaveMD started SaveMD successful, Duration 15.89 seconds DeleteWorkspace started DeleteWorkspace successful, Duration 0.00 seconds DeleteWorkspace started DeleteWorkspace successful, Duration 0.00 seconds MergeMDFiles started Error loading instrument IDF file for 'NXSPE'. Instrument search object NXSPE Setting cache to 400 MB write. 4238571 events in 3 files. Starting SaveMD to update the file back-end. MergeMDFiles successful, Duration 88.59 seconds
comment:39 Changed 8 years ago by Alex Buts
refs #5849 filename made class parameter trying to fix problem
on build servers
Changeset: 5223bd8846231b745ded5b65f85e591b5c8cd58e
comment:40 Changed 8 years ago by Alex Buts
refs #5849 enable all disabled tests
as seems the previous commit have fixed all problem
Changeset: a0859076c6873eab57e8d59291189e48d33fa47f
comment:41 Changed 8 years ago by Alex Buts
refs #5849 Added initial MDBoxFlatTreeTest to the build
Changeset: c6925a42ca0d039b67d75acb219085fbce5adbe5
comment:42 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 6695