Ticket #5852 (closed: duplicate)

Opened 8 years ago

Last modified 5 years ago

MD event workspace can not handle more then 0.3G events

Reported by: Alex Buts Owned by: Alex Buts
Priority: major Milestone: Release 2.3
Component: VATES Keywords:
Cc: Blocked By:
Blocking: Tester: Andrei Savici

Description

I am attempting to create large MD file in memory on the machine with 0.5T of RAM.

The script to build such file is presented below, but similar Horace script (also supplied for comparison) generates 4Gb events, writes 151Gb file and takes (70min+3min+160min) to complete (3 stages of Horace processing -- converting single workspace, calculating indexes, writing combined file)

Horace and Mantid script expected to generate the same results (if they not, it is a bug of its own which I am investigating independently) but in fact the results are different.

The Mantid script (single thread version has been used) takes ~(30min+85min) to complete (2 stages -- load nxspe and convertToMD) (very approximate, as single threaded, takes substantial memory and strongly depends on the load of the machine (changes with load and during conversion time)) but it generates ~0.3G events in workspace and writes 17G file.

If may be that the Matnid script sets different conversion from Horace (different bug) but there is an malloc messages generated during conversion: tcmalloc: large alloc 3451092992 bytes == 0x7fd2cecd8000 @ 0x7fdb90c1ef66 0x7fdb829025c8 0x7fda8127a800

Which make me think that it is rather the workspace bug.

Mantid congersion script:

rint 'start converting to SQW\n'
cur_ws='wsn';
md_ws='mdws4D';
saveTmpTo='/home/wkc26243/ProcessedData/'
MDWSF='';
TargetWSName='WS4DMerged';

ic=0;
file_nums1 = range(15654,15723);
file_nums2 = range(15733,15834);
file_nums  = file_nums1+file_nums2
for n in file_nums:
	ic = ic+1;

	inFile = 'MAP'+str(n)+'.nxspe';
	print 'processing file name: '+inFile+' step: #'+str(ic)+'/'+str(len(file_nums))
	LoadNXSPE(Filename=inFile,OutputWorkspace=cur_ws)		
	SetUB(Workspace=cur_ws,a='2.87',b='2.87',c='2.87',u='1,0,0',v='0,1,0')
	#SetGoniometer(cur_ws,Axis0='Psi,0,1,0,1');
	print 'Adding to md workspace '
	ConvertToMDEvents(InputWorkspace=cur_ws,OutputWorkspace=TargetWSName,QDimensions='Q3D',OverwriteExisting='0',QConversionScales='HKL',UsePreprocessedDetectors='1',MinValues='-50,-50,-50,-150',MaxValues='50,50,50,1300')
	
	
print 'Saving Final workspace'
SaveMD(TargetWSName,Filename=saveTmpTo+TargetWSName+'.nxs');

Horace Conversion script:

function hor1400K_script
% where the script should be run
demo_dir=pwd;


% Give locations of input files
indir=pwd;     % source directory of spe files
par_file='';
par_file=parse_path([indir '/Instrument/map_4to1_jul09.par']);     % detector parameter file
sqw_file=fullfile(indir,'fe_E1400_8K.sqw');        % output sqw file
data_source =sqw_file;

% Set incident energy, lattice parameters etc.
efix=1371.5;
emode=1;
alatt=[2.87,2.87,2.87];
angdeg=[90,90,90];
u=[1,0,0];
v=[0,1,0];
omega=0;dpsi=0;gl=0;gs=0;

% Create the list of file names and orientation angles
% G1
[spe_file,psi]=build_fnames(indir,15654:15664,35,1,45);
% G2
[spe_file,psi]=build_fnames(indir,15665:15710,0,-1,-45,spe_file,psi);
% G3
[spe_file,psi]=build_fnames(indir,15711:15723,-44.5,1,-32.5,spe_file,psi);
% G4
[spe_file,psi]=build_fnames(indir,15733:15739,-22.5,1,-16.5,spe_file,psi);
% G5
[spe_file,psi]=build_fnames(indir,15740:15800,-15.5,1,44.5,spe_file,psi);
% G6
[spe_file,psi]=build_fnames(indir,15801:15815,-31.5,1,-17.5,spe_file,psi);
% G7
[spe_file,psi]=build_fnames(indir,15816:15834,0,1,18,spe_file,psi);


% Create sqw file
gen_sqw (spe_file, par_file, sqw_file, efix, emode, alatt, angdeg,...
         u, v, psi, omega, dpsi, gl, gs);

Change History

comment:1 Changed 8 years ago by Alex Buts

  • Status changed from new to verify
  • Owner set to Alex Buts
  • Resolution set to duplicate

comment:2 Changed 8 years ago by Alex Buts

this is dublixate for the ticket #5853

comment:3 Changed 8 years ago by Andrei Savici

  • Status changed from verify to verifying
  • Tester set to Andrei Savici

comment:4 Changed 8 years ago by Andrei Savici

  • Status changed from verifying to closed

comment:5 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 6698

Note: See TracTickets for help on using tickets.