Ticket #8578 (closed: fixed)

Opened 7 years ago

Last modified 5 years ago

Python Reduction does not change reduction argiments three times (thread/memory corruption suspected)

Reported by: Alex Buts Owned by: Alex Buts
Priority: critical Milestone: Release 3.1
Component: Direct Inelastic Keywords:
Cc: martyn.gigg@…, nick.draper@…, jon.taylor@… Blocked By:
Blocking: #8640 Tester: Martyn Gigg

Description (last modified by Alex Buts) (diff)

The description below demonstrates generic problem -- a python script (reduction) does not accept third in a row change to the input parameters. To change parameters for the third time, Mantid restart is needed (which makes it the first change).

If one tries the reduction script found below, first time you request to run hardmaskPlus, it runs the whole diagnostics:

==== Diagnostic Test Summary from: start to: end ====
Hard mask:               mar11015.msk  11
First white beam test:   white_masks   2
Second white beam test:  None          None
Background test:         mask_bkgd     2
PSD Bleed test :         None          None

Second times it runs hadmask only as requested:

Diag Processed  workspace with 918 spectra and masked 11 bad spectra

But the third time, which should produce the run, equivalent to the first run it runs diagnostics only again:

Diag Processed  workspace with 918 spectra and masked 11 bad spectra

If you run this script again, it newer runs full diagnostics.

"""
Script to perform absolute units data reduction for MARI
"""
#from qtiGenie import *
import dgreduce
import time

dgreduce.setup('MAR')
ext='.raw'

# where to save resutls (usually specified in Mantid, data search directories)
save_dir = config.getString('defaultsave.directory')
if len(save_dir) ==0 :
    config['defaultsave.directory']=os.getcwd()
    save_dir = config.getString('defaultsave.directory')
    
print "Data will be saved into: ",save_dir
# map mask and cal file, again the values from Mantid, data search directories can be modified here
config.appendDataSearchDir('/usr/local/mprogs/InstrumentFiles/mari') 
config.appendDataSearchDir(r'c:\mprogs\ISIS\Libisis\InstrumentFiles\mari') 
# data (raw or nxs) run files -- values from data search directories can be modified here
config.appendDataSearchDir('/isisdatar55/NDXMARI/Instrument/data/cycle_05_1') 
config.appendDataSearchDir(r'd:/Data/MantidSystemTests/Data') 


maskfile='mar11015.msk' #'testMask2.msk'#hard mask out the edges of detectors, which tend to be noisy

#map file
mapfile='mari_res.map' # mapping file
#mapfile='/opt/Mantid/instrument/mapfiles/maps/parker_rings' #powder mapping file
mv_mapfile='4to1_mid_lowang'

# latest white beam vanadium file for bad detector diagnosis
wbvan=11060

#Run numbers can be specified as a list:
#runno=[17422,17423, etc]
runno=[11001] #[19399] #,[00004] 19402]

#Incident energy list e.g. ei=[20,30,40]
ei=12

#Sample info
sam_rmm = 435.96
#sam_mass= 2.465
sam_mass = 10
    
rebin_pars=[-11,0.05,11]
monovan=11015
argi = {};
argi['hardmaskPlus']=maskfile 
argi['sample_mass'] = sam_mass   
argi['sample_rmm']   =sam_rmm 


print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
for i in range(len(runno)):
        #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi)
        # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters
        w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi)

	save_file='MAR'+str(runno[i])+'_ei'+str(ei)  
	SaveNXSPE(w1,save_file+'_Abs.nxspe')
	#SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs')	
	DeleteWorkspace(w1)


argi['hardmaskOnly']=maskfile 
print 'HARD MASK ONLY ------------------------------------------------------------------------------------------------'
print 'HARD MASK ONLY-----------------------------------------------------------------------------------------------'
print 'HARD MASK ONLY---------------------------------------------------------------------------------------------'


for i in range(len(runno)):
        #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi)
        # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters
        w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi)

	save_file='MAR'+str(runno[i])+'_ei'+str(ei)  
	SaveNXSPE(w1,save_file+'_Abs.nxspe')
	#SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs')	
	DeleteWorkspace(w1)

argi['hardmaskPlus']=maskfile 

print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
for i in range(len(runno)):
        #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi)
        # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters
        w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi)

	save_file='MAR'+str(runno[i])+'_ei'+str(ei)  
	SaveNXSPE(w1,save_file+'_Abs.nxspe')
	#SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs')	
	DeleteWorkspace(w1)




Change History

comment:1 Changed 7 years ago by Alex Buts

  • Priority changed from major to critical
  • type changed from enhancement to defect
  • Description modified (diff)
  • Milestone changed from Backlog to Release 3.1

comment:2 Changed 7 years ago by Alex Buts

  • Description modified (diff)

comment:3 Changed 7 years ago by Alex Buts

  • Description modified (diff)
  • Summary changed from Python Reduction does not change reduction argiments thirt dimes (thread corruption suspected) to Python Reduction does not change reduction argiments three times (thread/memory corruption suspected)

comment:4 Changed 7 years ago by Alex Buts

The script above is not fully illustrates the problem , as uses once initiated reducer in all subsequent calls to reducers. If I change script to the one below (plus do some other minor change to the reducer committed), wrong behaviour does not happens in the script, run separately in a debugger, but still continues in the script window if one set hardmaskPlus, and then change to harmaskOnly and then hardmaskPlus again.

This can happen if the script window does not re-initiates internal variables (reducer in this case) on each RunAll call.

"""
Script to perform absolute units data reduction for MARI
"""
#from qtiGenie import *
from mantid import config
from mantid.simpleapi import *
import dgreduce
import time
import os

dgreduce.setup('MAR')
ext='.raw'

# where to save resutls (usually specified in Mantid, data search directories)
save_dir = config.getString('defaultsave.directory')
if len(save_dir) ==0 :
    config['defaultsave.directory']=os.getcwd()
    save_dir = config.getString('defaultsave.directory')
    
print "Data will be saved into: ",save_dir
# map mask and cal file, again the values from Mantid, data search directories can be modified here
config.appendDataSearchDir('/usr/local/mprogs/InstrumentFiles/mari') 
config.appendDataSearchDir(r'c:\mprogs\ISIS\Libisis\InstrumentFiles\mari') 
# data (raw or nxs) run files -- values from data search directories can be modified here
config.appendDataSearchDir('/isisdatar55/NDXMARI/Instrument/data/cycle_05_1') 
config.appendDataSearchDir(r'd:/Data/MantidSystemTests/Data') 


maskfile='mar11015.msk' #'testMask2.msk'#hard mask out the edges of detectors, which tend to be noisy

#map file
mapfile='mari_res.map' # mapping file
#mapfile='/opt/Mantid/instrument/mapfiles/maps/parker_rings' #powder mapping file
mv_mapfile='4to1_mid_lowang'

# latest white beam vanadium file for bad detector diagnosis
wbvan=11060

#Run numbers can be specified as a list:
#runno=[17422,17423, etc]
runno=[11001] #[19399] #,[00004] 19402]

#Incident energy list e.g. ei=[20,30,40]
ei=12

#Sample info
sam_rmm = 435.96
#sam_mass= 2.465
sam_mass = 10
    
rebin_pars=[-11,0.05,11]
monovan=11015
argi = {};
argi['hardmaskPlus']=maskfile 
argi['sample_mass'] = sam_mass   
argi['sample_rmm']   =sam_rmm 


print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
for i in range(len(runno)):
        #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi)
        # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters
        w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi)

        save_file='MAR'+str(runno[i])+'_ei'+str(ei)  
        SaveNXSPE(w1,save_file+'_Abs.nxspe')
        #SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs')	
        DeleteWorkspace(w1)

argi1 = {};
argi1['sample_mass'] = sam_mass   
argi1['sample_rmm']   =sam_rmm 
argi1['hardmaskOnly'] =maskfile 
print 'HARD MASK ONLY ------------------------------------------------------------------------------------------------'
print 'HARD MASK ONLY-----------------------------------------------------------------------------------------------'
print 'HARD MASK ONLY---------------------------------------------------------------------------------------------'


for i in range(len(runno)):
        #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi)
        # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters
        w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi1)

        save_file='MAR'+str(runno[i])+'_ei'+str(ei)  
        SaveNXSPE(w1,save_file+'_Abs.nxspe')
        #SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs')	
        DeleteWorkspace(w1)


argi2 = {};
argi2['sample_mass'] = sam_mass   
argi2['sample_rmm']   =sam_rmm 
argi2['hardmaskPlus']=maskfile 

print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
print 'HARD MASK PLUS ------------------------------------------------------------------------------------------------'
for i in range(len(runno)): 
        #w1=iliad_abs(wbvan,runno[i],monovan,wbvan,sam_rmm,sam_mass,ei[i],rebin_pars,mapfile,mv_mapfile,**argi)
        # this does absolute units normalization as far as monovan is not None. Uses default map file, provided in MAPS_Parameters.xml file. Any changes from defaults should be provided here or above as parameters
        w1=dgreduce.arb_units(wbvan,runno[i],ei,rebin_pars,None,monovan,**argi)

        save_file='MAR'+str(runno[i])+'_ei'+str(ei)  
        SaveNXSPE(w1,save_file+'_Abs.nxspe')
    #SaveNexus(w1,save_file+'newDgrd_NewQTG_NewDirectConv.nxs')	
        DeleteWorkspace(w1)





comment:5 Changed 7 years ago by Alex Buts

the changes were committed to wrong ticket number. They are in the branch:

bugfix/8578_storedPythonParams

comment:6 Changed 7 years ago by Alex Buts

  • Status changed from new to inprogress
  • Owner set to Alex Buts

comment:7 Changed 7 years ago by Alex Buts

After fixing issue with legacy parameters I can not reproduce the issue with any other parameters any more

Git branch exists at https://github.com/mantidproject/mantid/tree/bugfix/8578_storedPythonParams and shows the changes with presumably fix the changes.

Tester can run the script above and see that no unexpected behaviour with diagnostics occur. This is trivial change which easy to understand and test.

I am not investigating this any further in anticipation to changes in reduction workflow algorithms.

Last edited 7 years ago by Alex Buts (previous) (diff)

comment:8 Changed 7 years ago by Alex Buts

  • Status changed from inprogress to verify
  • Resolution set to fixed

comment:9 Changed 7 years ago by Alex Buts

refs #8578 Fixed unit test DgreduceTest

Changeset: c20e7d76fe9572e141e6f315c44db3bd719fdf19

comment:10 Changed 7 years ago by Alex Buts

  • Status changed from verify to reopened
  • Resolution fixed deleted

This is other script which illustrates the problem which still exists.

When script below is run with testToRun = MARIReductionSum? only, MARIReductionSum passes.

When one runs two tests in the row, e.g.: testToRun = ['MARIReductionFromFile','MARIReductionSum']

MARIReductionSum fails.

'''
This is provided for testing purposes. It allows you to run the single system test
within Mantid environent
It can be useful for debugging because the errors do not alway 'get out' of
the sub-process used for running the tests in the regular way
'''
from mantid.simpleapi import *
from mantid import config
import sys
import os


this_dir = sys.path[0]
print "Script resides in : ",this_dir
#python directories
os.chdir(r'd:\Data\MantidSystemTests\SystemTests')
stressmodule_dir = r'd:\Data\MantidSystemTests\StressTestFramework'
tests_dir = r'd:/Data/MantidSystemTests/SystemTests/AnalysisTests'


sys.path.insert(0,tests_dir)
sys.path.append(stressmodule_dir)
sys.path.append(r'd:\Data\Mantid_GIT_test\Code\builds\br_master\bin\Release')



maps_dir = 'd:/Data/MantidSystemTests/Data'
data_dir ='d:/Data/Mantid_Testing/13_12_09'
ref_data_dir = 'd:/Data/MantidSystemTests/SystemTests/AnalysisTests/ReferenceResults' 
config.setDataSearchDirs('{0};{1};{2}'.format(data_dir,maps_dir,ref_data_dir))
config['defaultsave.directory'] = data_dir # folder to save resulting spe/nxspe files. Defaults are in

# data search directories



# Find these first
modlToRun = ['ISISDirectInelastic']

#testToRun = ['MARIReductionSum']
testToRun = ['MARIReductionFromFile','MARIReductionSum']




module = __import__(modlToRun[0])
reload(module)
#reload(sys.modules['isis_reduction_steps'])

for className in testToRun:
    try:
        testClass = getattr(module, className)()
        testClass.execute()
    except:
        raise
        #os.chdir(this_dir)
        #raise

    outcome = testClass.doValidation()
    print 'Test result: ' + str(outcome)

os.chdir(this_dir)

comment:11 Changed 7 years ago by Alex Buts

  • Status changed from reopened to inprogress

refs #8578 Should fix the issue

Changeset: b03c78598688911d0f0939bf941a6a193b7c3214

comment:12 Changed 7 years ago by Alex Buts

refs #8578 fixing some weird brunching error

which cause unit test to fail.

Changeset: b630bdd31b8329522748b0cd2470aba0dbb75d30

comment:13 Changed 7 years ago by Alex Buts

Should be simple ticket to test.

2 scripts above should pass after the merge and changes to code are small and should be easy to understand.

comment:14 Changed 7 years ago by Alex Buts

refs #8578 Trivial change allows to define hard mask file in any form

imaginable (or rather define that hard mask file is absent)

Changeset: e7fe8921ac26982d0dbb35365133e95ebdedcd51

comment:15 Changed 7 years ago by Alex Buts

  • Status changed from inprogress to verify
  • Resolution set to fixed

comment:16 Changed 7 years ago by Alex Buts

  • Blocking 8640 added

(In #8640) The System tests will fail until #8578 is applied, so I am merging it to the ticket.

comment:17 Changed 7 years ago by Alex Buts

  • Blocking 8640 removed

comment:18 Changed 7 years ago by Alex Buts

  • Blocking 8640 added
Last edited 7 years ago by Alex Buts (previous) (diff)

comment:19 Changed 7 years ago by Alex Buts

  • Blocking 8640 removed

comment:20 Changed 7 years ago by Alex Buts

  • Blocking 8640 added

comment:20 Changed 7 years ago by Alex Buts

  • Blocking 8578 added; 8640 removed
  • Blocked By 8578 added

comment:21 Changed 7 years ago by Alex Buts

  • Blocking 8640 added; 8578 removed
  • Blocked By 8578 removed

comment:22 Changed 7 years ago by Nick Draper

  • Component changed from Framework to Direct Inelastic

comment:23 Changed 7 years ago by Martyn Gigg

  • Status changed from verify to verifying
  • Tester set to Martyn Gigg

comment:24 Changed 7 years ago by Martyn Gigg

  • Status changed from verifying to closed

The system tests script seems to function correctly now and resetting the IDF parameters seems sensible, especially if the object is going to be global.

comment:25 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 9422

Note: See TracTickets for help on using tickets.