Ticket #8578 (closed: fixed)
Python Reduction does not change reduction argiments three times (thread/memory corruption suspected)
Reported by: | Alex Buts | Owned by: | Alex Buts |
---|---|---|---|
Priority: | critical | Milestone: | Release 3.1 |
Component: | Direct Inelastic | Keywords: | |
Cc: | martyn.gigg@…, nick.draper@…, jon.taylor@… | Blocked By: | |
Blocking: | #8640 | Tester: | Martyn Gigg |
Description (last modified by Alex Buts) (diff)
The description below demonstrates generic problem -- a python script (reduction) does not accept third in a row change to the input parameters. To change parameters for the third time, Mantid restart is needed (which makes it the first change).
If one tries the reduction script found below, first time you request to run hardmaskPlus, it runs the whole diagnostics:
==== Diagnostic Test Summary from: start to: end ==== Hard mask: mar11015.msk 11 First white beam test: white_masks 2 Second white beam test: None None Background test: mask_bkgd 2 PSD Bleed test : None None
Second times it runs hadmask only as requested:
Diag Processed workspace with 918 spectra and masked 11 bad spectra
But the third time, which should produce the run, equivalent to the first run it runs diagnostics only again:
Diag Processed workspace with 918 spectra and masked 11 bad spectra
If you run this script again, it newer runs full diagnostics.
""" Script to perform absolute units data reduction for MARI """ #from qtiGenie import * import dgreduce import time dgreduce.setup('MAR') ext='.raw' # where to save resutls (usually specified in Mantid, data search directories) save_dir = config.getString('defaultsave.directory') if len(save_dir) ==0 : config['defaultsave.directory']=os.getcwd() save_dir = config.getString('defaultsave.directory') print "Data will be saved into: ",save_dir # map mask and cal file, again the values from Mantid, data search directories can be modified here config.appendDataSearchDir('/usr/local/mprogs/InstrumentFiles/mari') config.appendDataSearchDir(r'c:\mprogs\ISIS\Libisis\InstrumentFiles\mari') # data (raw or nxs) run files -- values from data search directories can be modified here config.appendDataSearchDir('/isisdatar55/NDXMARI/Instrument/data/cycle_05_1') config.appendDataSearchDir(r'd:/Data/MantidSystemTests/Data') maskfile='mar11015.msk' #'testMask2.msk'#hard mask out the edges of detectors, which tend to be noisy #map file mapfile='mari_res.map' # mapping file #mapfile='/opt/Mantid/instrument/mapfiles/maps/parker_rings' #powder mapping file mv_mapfile='4to1_mid_lowang' # latest white beam vanadium file for bad detector diagnosis wbvan=11060 #Run numbers can be specified as a list: #runno=[17422,17423, etc] runno=[11001] #[19399] #,[00004] 19402] #Incident energy list e.g. ei=[20,30,40] ei=12 #Sample info sam_rmm = 435.96 #sam_mass= 2.465 sam_mass = 10 rebin_pars=[-11,0.05,11] monovan=11015 argi = {}; argi['hardmaskPlus']=maskfile argi['sample_mass'] = sam_mass argi['sample_rmm'] =sam_rmm print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------' print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------' print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------' for i in range(len(runno)): #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi) # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi) save_file='MAR'+str(runno[i])+'_ei'+str(ei) SaveNXSPE(w1,save_file+'_Abs.nxspe') #SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs') DeleteWorkspace(w1) argi['hardmaskOnly']=maskfile print 'HARD MASK ONLY ------------------------------------------------------------------------------------------------' print 'HARD MASK ONLY-----------------------------------------------------------------------------------------------' print 'HARD MASK ONLY---------------------------------------------------------------------------------------------' for i in range(len(runno)): #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi) # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi) save_file='MAR'+str(runno[i])+'_ei'+str(ei) SaveNXSPE(w1,save_file+'_Abs.nxspe') #SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs') DeleteWorkspace(w1) argi['hardmaskPlus']=maskfile print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------' print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------' print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------' for i in range(len(runno)): #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi) # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi) save_file='MAR'+str(runno[i])+'_ei'+str(ei) SaveNXSPE(w1,save_file+'_Abs.nxspe') #SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs') DeleteWorkspace(w1)
Change History
comment:1 Changed 7 years ago by Alex Buts
- Priority changed from major to critical
- type changed from enhancement to defect
- Description modified (diff)
- Milestone changed from Backlog to Release 3.1
comment:3 Changed 7 years ago by Alex Buts
- Description modified (diff)
- Summary changed from Python Reduction does not change reduction argiments thirt dimes (thread corruption suspected) to Python Reduction does not change reduction argiments three times (thread/memory corruption suspected)
comment:4 Changed 7 years ago by Alex Buts
The script above is not fully illustrates the problem , as uses once initiated reducer in all subsequent calls to reducers. If I change script to the one below (plus do some other minor change to the reducer committed), wrong behaviour does not happens in the script, run separately in a debugger, but still continues in the script window if one set hardmaskPlus, and then change to harmaskOnly and then hardmaskPlus again.
This can happen if the script window does not re-initiates internal variables (reducer in this case) on each RunAll call.
""" Script to perform absolute units data reduction for MARI """ #from qtiGenie import * from mantid import config from mantid.simpleapi import * import dgreduce import time import os dgreduce.setup('MAR') ext='.raw' # where to save resutls (usually specified in Mantid, data search directories) save_dir = config.getString('defaultsave.directory') if len(save_dir) ==0 : config['defaultsave.directory']=os.getcwd() save_dir = config.getString('defaultsave.directory') print "Data will be saved into: ",save_dir # map mask and cal file, again the values from Mantid, data search directories can be modified here config.appendDataSearchDir('/usr/local/mprogs/InstrumentFiles/mari') config.appendDataSearchDir(r'c:\mprogs\ISIS\Libisis\InstrumentFiles\mari') # data (raw or nxs) run files -- values from data search directories can be modified here config.appendDataSearchDir('/isisdatar55/NDXMARI/Instrument/data/cycle_05_1') config.appendDataSearchDir(r'd:/Data/MantidSystemTests/Data') maskfile='mar11015.msk' #'testMask2.msk'#hard mask out the edges of detectors, which tend to be noisy #map file mapfile='mari_res.map' # mapping file #mapfile='/opt/Mantid/instrument/mapfiles/maps/parker_rings' #powder mapping file mv_mapfile='4to1_mid_lowang' # latest white beam vanadium file for bad detector diagnosis wbvan=11060 #Run numbers can be specified as a list: #runno=[17422,17423, etc] runno=[11001] #[19399] #,[00004] 19402] #Incident energy list e.g. ei=[20,30,40] ei=12 #Sample info sam_rmm = 435.96 #sam_mass= 2.465 sam_mass = 10 rebin_pars=[-11,0.05,11] monovan=11015 argi = {}; argi['hardmaskPlus']=maskfile argi['sample_mass'] = sam_mass argi['sample_rmm'] =sam_rmm print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------' print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------' print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------' for i in range(len(runno)): #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi) # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi) save_file='MAR'+str(runno[i])+'_ei'+str(ei) SaveNXSPE(w1,save_file+'_Abs.nxspe') #SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs') DeleteWorkspace(w1) argi1 = {}; argi1['sample_mass'] = sam_mass argi1['sample_rmm'] =sam_rmm argi1['hardmaskOnly'] =maskfile print 'HARD MASK ONLY ------------------------------------------------------------------------------------------------' print 'HARD MASK ONLY-----------------------------------------------------------------------------------------------' print 'HARD MASK ONLY---------------------------------------------------------------------------------------------' for i in range(len(runno)): #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi) # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi1) save_file='MAR'+str(runno[i])+'_ei'+str(ei) SaveNXSPE(w1,save_file+'_Abs.nxspe') #SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs') DeleteWorkspace(w1) argi2 = {}; argi2['sample_mass'] = sam_mass argi2['sample_rmm'] =sam_rmm argi2['hardmaskPlus']=maskfile print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------' print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------' print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------' for i in range(len(runno)): #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi) # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi) save_file='MAR'+str(runno[i])+'_ei'+str(ei) SaveNXSPE(w1,save_file+'_Abs.nxspe') #SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs') DeleteWorkspace(w1)
comment:5 Changed 7 years ago by Alex Buts
the changes were committed to wrong ticket number. They are in the branch:
bugfix/8578_storedPythonParams
comment:6 Changed 7 years ago by Alex Buts
- Status changed from new to inprogress
- Owner set to Alex Buts
comment:7 Changed 7 years ago by Alex Buts
After fixing issue with legacy parameters I can not reproduce the issue with any other parameters any more
Git branch exists at https://github.com/mantidproject/mantid/tree/bugfix/8578_storedPythonParams and shows the changes with presumably fix the changes.
Tester can run the script above and see that no unexpected behaviour with diagnostics occur. This is trivial change which easy to understand and test.
I am not investigating this any further in anticipation to changes in reduction workflow algorithms.
comment:8 Changed 7 years ago by Alex Buts
- Status changed from inprogress to verify
- Resolution set to fixed
comment:9 Changed 7 years ago by Alex Buts
refs #8578 Fixed unit test DgreduceTest
Changeset: c20e7d76fe9572e141e6f315c44db3bd719fdf19
comment:10 Changed 7 years ago by Alex Buts
- Status changed from verify to reopened
- Resolution fixed deleted
This is other script which illustrates the problem which still exists.
When script below is run with testToRun = MARIReductionSum? only, MARIReductionSum passes.
When one runs two tests in the row, e.g.: testToRun = ['MARIReductionFromFile','MARIReductionSum']
MARIReductionSum fails.
''' This is provided for testing purposes. It allows you to run the single system test within Mantid environent It can be useful for debugging because the errors do not alway 'get out' of the sub-process used for running the tests in the regular way ''' from mantid.simpleapi import * from mantid import config import sys import os this_dir = sys.path[0] print "Script resides in : ",this_dir #python directories os.chdir(r'd:\Data\MantidSystemTests\SystemTests') stressmodule_dir = r'd:\Data\MantidSystemTests\StressTestFramework' tests_dir = r'd:/Data/MantidSystemTests/SystemTests/AnalysisTests' sys.path.insert(0,tests_dir) sys.path.append(stressmodule_dir) sys.path.append(r'd:\Data\Mantid_GIT_test\Code\builds\br_master\bin\Release') maps_dir = 'd:/Data/MantidSystemTests/Data' data_dir ='d:/Data/Mantid_Testing/13_12_09' ref_data_dir = 'd:/Data/MantidSystemTests/SystemTests/AnalysisTests/ReferenceResults' config.setDataSearchDirs('{0};{1};{2}'.format(data_dir,maps_dir,ref_data_dir)) config['defaultsave.directory'] = data_dir # folder to save resulting spe/nxspe files. Defaults are in # data search directories # Find these first modlToRun = ['ISISDirectInelastic'] #testToRun = ['MARIReductionSum'] testToRun = ['MARIReductionFromFile','MARIReductionSum'] module = __import__(modlToRun[0]) reload(module) #reload(sys.modules['isis_reduction_steps']) for className in testToRun: try: testClass = getattr(module, className)() testClass.execute() except: raise #os.chdir(this_dir) #raise outcome = testClass.doValidation() print 'Test result: ' + str(outcome) os.chdir(this_dir)
comment:11 Changed 7 years ago by Alex Buts
- Status changed from reopened to inprogress
refs #8578 Should fix the issue
Changeset: b03c78598688911d0f0939bf941a6a193b7c3214
comment:12 Changed 7 years ago by Alex Buts
refs #8578 fixing some weird brunching error
which cause unit test to fail.
Changeset: b630bdd31b8329522748b0cd2470aba0dbb75d30
comment:13 Changed 7 years ago by Alex Buts
Should be simple ticket to test.
2 scripts above should pass after the merge and changes to code are small and should be easy to understand.
comment:14 Changed 7 years ago by Alex Buts
refs #8578 Trivial change allows to define hard mask file in any form
imaginable (or rather define that hard mask file is absent)
Changeset: e7fe8921ac26982d0dbb35365133e95ebdedcd51
comment:15 Changed 7 years ago by Alex Buts
- Status changed from inprogress to verify
- Resolution set to fixed
comment:16 Changed 7 years ago by Alex Buts
- Blocking 8640 added
comment:21 Changed 7 years ago by Alex Buts
- Blocking 8640 added; 8578 removed
- Blocked By 8578 removed
comment:23 Changed 7 years ago by Martyn Gigg
- Status changed from verify to verifying
- Tester set to Martyn Gigg
comment:24 Changed 7 years ago by Martyn Gigg
- Status changed from verifying to closed
The system tests script seems to function correctly now and resetting the IDF parameters seems sensible, especially if the object is going to be global.
comment:25 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 9422