Ticket #5852 (closed: duplicate)
MD event workspace can not handle more then 0.3G events
Reported by: | Alex Buts | Owned by: | Alex Buts |
---|---|---|---|
Priority: | major | Milestone: | Release 2.3 |
Component: | VATES | Keywords: | |
Cc: | Blocked By: | ||
Blocking: | Tester: | Andrei Savici |
Description
I am attempting to create large MD file in memory on the machine with 0.5T of RAM.
The script to build such file is presented below, but similar Horace script (also supplied for comparison) generates 4Gb events, writes 151Gb file and takes (70min+3min+160min) to complete (3 stages of Horace processing -- converting single workspace, calculating indexes, writing combined file)
Horace and Mantid script expected to generate the same results (if they not, it is a bug of its own which I am investigating independently) but in fact the results are different.
The Mantid script (single thread version has been used) takes ~(30min+85min) to complete (2 stages -- load nxspe and convertToMD) (very approximate, as single threaded, takes substantial memory and strongly depends on the load of the machine (changes with load and during conversion time)) but it generates ~0.3G events in workspace and writes 17G file.
If may be that the Matnid script sets different conversion from Horace (different bug) but there is an malloc messages generated during conversion: tcmalloc: large alloc 3451092992 bytes == 0x7fd2cecd8000 @ 0x7fdb90c1ef66 0x7fdb829025c8 0x7fda8127a800
Which make me think that it is rather the workspace bug.
Mantid congersion script:
rint 'start converting to SQW\n' cur_ws='wsn'; md_ws='mdws4D'; saveTmpTo='/home/wkc26243/ProcessedData/' MDWSF=''; TargetWSName='WS4DMerged'; ic=0; file_nums1 = range(15654,15723); file_nums2 = range(15733,15834); file_nums = file_nums1+file_nums2 for n in file_nums: ic = ic+1; inFile = 'MAP'+str(n)+'.nxspe'; print 'processing file name: '+inFile+' step: #'+str(ic)+'/'+str(len(file_nums)) LoadNXSPE(Filename=inFile,OutputWorkspace=cur_ws) SetUB(Workspace=cur_ws,a='2.87',b='2.87',c='2.87',u='1,0,0',v='0,1,0') #SetGoniometer(cur_ws,Axis0='Psi,0,1,0,1'); print 'Adding to md workspace ' ConvertToMDEvents(InputWorkspace=cur_ws,OutputWorkspace=TargetWSName,QDimensions='Q3D',OverwriteExisting='0',QConversionScales='HKL',UsePreprocessedDetectors='1',MinValues='-50,-50,-50,-150',MaxValues='50,50,50,1300') print 'Saving Final workspace' SaveMD(TargetWSName,Filename=saveTmpTo+TargetWSName+'.nxs');
Horace Conversion script:
function hor1400K_script % where the script should be run demo_dir=pwd; % Give locations of input files indir=pwd; % source directory of spe files par_file=''; par_file=parse_path([indir '/Instrument/map_4to1_jul09.par']); % detector parameter file sqw_file=fullfile(indir,'fe_E1400_8K.sqw'); % output sqw file data_source =sqw_file; % Set incident energy, lattice parameters etc. efix=1371.5; emode=1; alatt=[2.87,2.87,2.87]; angdeg=[90,90,90]; u=[1,0,0]; v=[0,1,0]; omega=0;dpsi=0;gl=0;gs=0; % Create the list of file names and orientation angles % G1 [spe_file,psi]=build_fnames(indir,15654:15664,35,1,45); % G2 [spe_file,psi]=build_fnames(indir,15665:15710,0,-1,-45,spe_file,psi); % G3 [spe_file,psi]=build_fnames(indir,15711:15723,-44.5,1,-32.5,spe_file,psi); % G4 [spe_file,psi]=build_fnames(indir,15733:15739,-22.5,1,-16.5,spe_file,psi); % G5 [spe_file,psi]=build_fnames(indir,15740:15800,-15.5,1,44.5,spe_file,psi); % G6 [spe_file,psi]=build_fnames(indir,15801:15815,-31.5,1,-17.5,spe_file,psi); % G7 [spe_file,psi]=build_fnames(indir,15816:15834,0,1,18,spe_file,psi); % Create sqw file gen_sqw (spe_file, par_file, sqw_file, efix, emode, alatt, angdeg,... u, v, psi, omega, dpsi, gl, gs);