Python Reduction does not change reduction argiments three times (thread/memory corruption suspected)

[Alex Buts]

The description below demonstrates generic problem -- a python script (reduction) does not accept third in a row change to the input parameters. To change parameters for the third time, Mantid restart is needed (which makes it the first change).

If one tries the reduction script found below, first time you request to run hardmaskPlus, it runs the whole diagnostics:

==== Diagnostic Test Summary from: start to: end ====
Hard mask:               mar11015.msk  11
First white beam test:   white_masks   2
Second white beam test:  None          None
Background test:         mask_bkgd     2
PSD Bleed test :         None          None

Second times it runs hadmask only as requested:

Diag Processed  workspace with 918 spectra and masked 11 bad spectra

But the third time, which should produce the run, equivalent to the first run it runs diagnostics only again:

Diag Processed  workspace with 918 spectra and masked 11 bad spectra

If you run this script again, it newer runs full diagnostics.

"""
Script to perform absolute units data reduction for MARI
"""
#from qtiGenie import *
import dgreduce
import time

dgreduce.setup('MAR')
ext='.raw'

# where to save resutls (usually specified in Mantid, data search directories)
save_dir = config.getString('defaultsave.directory')
if len(save_dir) ==0 :
    config['defaultsave.directory']=os.getcwd()
    save_dir = config.getString('defaultsave.directory')
    
print "Data will be saved into: ",save_dir
# map mask and cal file, again the values from Mantid, data search directories can be modified here
config.appendDataSearchDir('/usr/local/mprogs/InstrumentFiles/mari') 
config.appendDataSearchDir(r'c:\mprogs\ISIS\Libisis\InstrumentFiles\mari') 
# data (raw or nxs) run files -- values from data search directories can be modified here
config.appendDataSearchDir('/isisdatar55/NDXMARI/Instrument/data/cycle_05_1') 
config.appendDataSearchDir(r'd:/Data/MantidSystemTests/Data') 


maskfile='mar11015.msk' #'testMask2.msk'#hard mask out the edges of detectors, which tend to be noisy

#map file
mapfile='mari_res.map' # mapping file
#mapfile='/opt/Mantid/instrument/mapfiles/maps/parker_rings' #powder mapping file
mv_mapfile='4to1_mid_lowang'

# latest white beam vanadium file for bad detector diagnosis
wbvan=11060

#Run numbers can be specified as a list:
#runno=[17422,17423, etc]
runno=[11001] #[19399] #,[00004] 19402]

#Incident energy list e.g. ei=[20,30,40]
ei=12

#Sample info
sam_rmm = 435.96
#sam_mass= 2.465
sam_mass = 10
    
rebin_pars=[-11,0.05,11]
monovan=11015
argi = {};
argi['hardmaskPlus']=maskfile 
argi['sample_mass'] = sam_mass   
argi['sample_rmm']   =sam_rmm 


print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
for i in range(len(runno)):
        #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi)
        # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters
        w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi)

	save_file='MAR'+str(runno[i])+'_ei'+str(ei)  
	SaveNXSPE(w1,save_file+'_Abs.nxspe')
	#SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs')	
	DeleteWorkspace(w1)


argi['hardmaskOnly']=maskfile 
print 'HARD MASK ONLY ------------------------------------------------------------------------------------------------'
print 'HARD MASK ONLY-----------------------------------------------------------------------------------------------'
print 'HARD MASK ONLY---------------------------------------------------------------------------------------------'


for i in range(len(runno)):
        #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi)
        # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters
        w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi)

	save_file='MAR'+str(runno[i])+'_ei'+str(ei)  
	SaveNXSPE(w1,save_file+'_Abs.nxspe')
	#SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs')	
	DeleteWorkspace(w1)

argi['hardmaskPlus']=maskfile 

print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
for i in range(len(runno)):
        #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi)
        # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters
        w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi)

	save_file='MAR'+str(runno[i])+'_ei'+str(ei)  
	SaveNXSPE(w1,save_file+'_Abs.nxspe')
	#SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs')	
	DeleteWorkspace(w1)

[Alex Buts]

The script above is not fully illustrates the problem , as uses once initiated reducer in all subsequent calls to reducers. If I change script to the one below (plus do some other minor change to the reducer committed), wrong behaviour does not happens in the script, run separately in a debugger, but still continues in the script window if one set hardmaskPlus, and then change to harmaskOnly and then hardmaskPlus again.

This can happen if the script window does not re-initiates internal variables (reducer in this case) on each RunAll call.

"""
Script to perform absolute units data reduction for MARI
"""
#from qtiGenie import *
from mantid import config
from mantid.simpleapi import *
import dgreduce
import time
import os

dgreduce.setup('MAR')
ext='.raw'

# where to save resutls (usually specified in Mantid, data search directories)
save_dir = config.getString('defaultsave.directory')
if len(save_dir) ==0 :
    config['defaultsave.directory']=os.getcwd()
    save_dir = config.getString('defaultsave.directory')
    
print "Data will be saved into: ",save_dir
# map mask and cal file, again the values from Mantid, data search directories can be modified here
config.appendDataSearchDir('/usr/local/mprogs/InstrumentFiles/mari') 
config.appendDataSearchDir(r'c:\mprogs\ISIS\Libisis\InstrumentFiles\mari') 
# data (raw or nxs) run files -- values from data search directories can be modified here
config.appendDataSearchDir('/isisdatar55/NDXMARI/Instrument/data/cycle_05_1') 
config.appendDataSearchDir(r'd:/Data/MantidSystemTests/Data') 


maskfile='mar11015.msk' #'testMask2.msk'#hard mask out the edges of detectors, which tend to be noisy

#map file
mapfile='mari_res.map' # mapping file
#mapfile='/opt/Mantid/instrument/mapfiles/maps/parker_rings' #powder mapping file
mv_mapfile='4to1_mid_lowang'

# latest white beam vanadium file for bad detector diagnosis
wbvan=11060

#Run numbers can be specified as a list:
#runno=[17422,17423, etc]
runno=[11001] #[19399] #,[00004] 19402]

#Incident energy list e.g. ei=[20,30,40]
ei=12

#Sample info
sam_rmm = 435.96
#sam_mass= 2.465
sam_mass = 10
    
rebin_pars=[-11,0.05,11]
monovan=11015
argi = {};
argi['hardmaskPlus']=maskfile 
argi['sample_mass'] = sam_mass   
argi['sample_rmm']   =sam_rmm 


print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
for i in range(len(runno)):
        #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi)
        # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters
        w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi)

        save_file='MAR'+str(runno[i])+'_ei'+str(ei)  
        SaveNXSPE(w1,save_file+'_Abs.nxspe')
        #SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs')	
        DeleteWorkspace(w1)

argi1 = {};
argi1['sample_mass'] = sam_mass   
argi1['sample_rmm']   =sam_rmm 
argi1['hardmaskOnly'] =maskfile 
print 'HARD MASK ONLY ------------------------------------------------------------------------------------------------'
print 'HARD MASK ONLY-----------------------------------------------------------------------------------------------'
print 'HARD MASK ONLY---------------------------------------------------------------------------------------------'


for i in range(len(runno)):
        #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi)
        # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters
        w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi1)

        save_file='MAR'+str(runno[i])+'_ei'+str(ei)  
        SaveNXSPE(w1,save_file+'_Abs.nxspe')
        #SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs')	
        DeleteWorkspace(w1)


argi2 = {};
argi2['sample_mass'] = sam_mass   
argi2['sample_rmm']   =sam_rmm 
argi2['hardmaskPlus']=maskfile 

print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
for i in range(len(runno)): 
        #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi)
        # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters
        w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi)

        save_file='MAR'+str(runno[i])+'_ei'+str(ei)  
        SaveNXSPE(w1,save_file+'_Abs.nxspe')
    #SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs')	
        DeleteWorkspace(w1)

[Alex Buts]

the changes were committed to wrong ticket number. They are in the branch:

bugfix/8578_storedPythonParams

[Alex Buts]

After fixing issue with legacy parameters I can not reproduce the issue with any other parameters any more

Git branch exists at ​https://github.com/mantidproject/mantid/tree/bugfix/8578_storedPythonParams and shows the changes with presumably fix the changes.

Tester can run the script above and see that no unexpected behaviour with diagnostics occur. This is trivial change which easy to understand and test.

I am not investigating this any further in anticipation to changes in reduction workflow algorithms.

[Alex Buts]

refs #8578 Fixed unit test DgreduceTest

Changeset: c20e7d76fe9572e141e6f315c44db3bd719fdf19

[Alex Buts]

This is other script which illustrates the problem which still exists.

When script below is run with testToRun = MARIReductionSum? only, MARIReductionSum passes.

When one runs two tests in the row, e.g.: testToRun = ['MARIReductionFromFile','MARIReductionSum']

MARIReductionSum fails.

'''
This is provided for testing purposes. It allows you to run the single system test
within Mantid environent
It can be useful for debugging because the errors do not alway 'get out' of
the sub-process used for running the tests in the regular way
'''
from mantid.simpleapi import *
from mantid import config
import sys
import os


this_dir = sys.path[0]
print "Script resides in : ",this_dir
#python directories
os.chdir(r'd:\Data\MantidSystemTests\SystemTests')
stressmodule_dir = r'd:\Data\MantidSystemTests\StressTestFramework'
tests_dir = r'd:/Data/MantidSystemTests/SystemTests/AnalysisTests'


sys.path.insert(0,tests_dir)
sys.path.append(stressmodule_dir)
sys.path.append(r'd:\Data\Mantid_GIT_test\Code\builds\br_master\bin\Release')



maps_dir = 'd:/Data/MantidSystemTests/Data'
data_dir ='d:/Data/Mantid_Testing/13_12_09'
ref_data_dir = 'd:/Data/MantidSystemTests/SystemTests/AnalysisTests/ReferenceResults' 
config.setDataSearchDirs('{0};{1};{2}'.format(data_dir,maps_dir,ref_data_dir))
config['defaultsave.directory'] = data_dir # folder to save resulting spe/nxspe files. Defaults are in

# data search directories



# Find these first
modlToRun = ['ISISDirectInelastic']

#testToRun = ['MARIReductionSum']
testToRun = ['MARIReductionFromFile','MARIReductionSum']




module = __import__(modlToRun[0])
reload(module)
#reload(sys.modules['isis_reduction_steps'])

for className in testToRun:
    try:
        testClass = getattr(module, className)()
        testClass.execute()
    except:
        raise
        #os.chdir(this_dir)
        #raise

    outcome = testClass.doValidation()
    print 'Test result: ' + str(outcome)

os.chdir(this_dir)

[Alex Buts]

refs #8578 Should fix the issue

Changeset: b03c78598688911d0f0939bf941a6a193b7c3214

[Alex Buts]

refs #8578 fixing some weird brunching error

which cause unit test to fail.

Changeset: b630bdd31b8329522748b0cd2470aba0dbb75d30

[Alex Buts]

Should be simple ticket to test.

2 scripts above should pass after the merge and changes to code are small and should be easy to understand.

[Alex Buts]

refs #8578 Trivial change allows to define hard mask file in any form

imaginable (or rather define that hard mask file is absent)

Changeset: e7fe8921ac26982d0dbb35365133e95ebdedcd51

[Alex Buts]

(In #8640) The System tests will fail until #8578 is applied, so I am merging it to the ticket.

[Martyn Gigg]

The system tests script seems to function correctly now and resetting the IDF parameters seems sensible, especially if the object is going to be global.

[Stuart Campbell]

This ticket has been transferred to github issue ​9422