Ticket #9955 (closed: fixed)

Opened 6 years ago

Last modified 5 years ago

Strange NaN issues with Stitch1D

Reported by: Owen Arnold Owned by: Owen Arnold
Priority: blocker Milestone: Release 3.2.1
Component: Framework Keywords:
Cc: Blocked By: #9956
Blocking: Tester: Anders Markvardsen

Description (last modified by Owen Arnold) (diff)

The following is a script reported by an scientist. 

# Load SURF00110504, SURF00110503. SURF00110505 from ICAT
RenameWorkspace('SURF00110504', OutputWorkspace='_110504')
RenameWorkspace('SURF00110503', OutputWorkspace='_110503')
RenameWorkspace('SURF00110505', OutputWorkspace='_110505')
CreateSingleValuedWorkspace(OutputWorkspace='__tmp_binary_operation_double')
Plus(LHSWorkspace='__tmp_binary_operation_double',RHSWorkspace='_110504',OutputWorkspace='__python_op_tmp0')
ConvertUnits(InputWorkspace='__python_op_tmp0',OutputWorkspace='sum_wavelength',Target='Wavelength',AlignBins='1')
CropWorkspace(InputWorkspace='sum_wavelength',OutputWorkspace='_monitor_ws',StartWorkspaceIndex='1',EndWorkspaceIndex='1')
CloneWorkspace(InputWorkspace='sum_wavelength',OutputWorkspace='_in_rng')
CropWorkspace(InputWorkspace='_in_rng',OutputWorkspace='_in_rng',StartWorkspaceIndex='2',EndWorkspaceIndex='2')
CropWorkspace(InputWorkspace='_in_rng',OutputWorkspace='_detector_ws',XMin='0.5',XMax='9')
CalculateFlatBackground(InputWorkspace='_monitor_ws',OutputWorkspace='_monitor_ws',StartX='7.5999999999999996',EndX='8.5',WorkspaceIndexList='0')
RebinToWorkspace(WorkspaceToRebin='_monitor_ws',WorkspaceToMatch='_detector_ws',OutputWorkspace='_I0P')
Integration(InputWorkspace='_I0P',OutputWorkspace='_monInt',RangeLower='4',RangeUpper='6')
Divide(LHSWorkspace='_detector_ws',RHSWorkspace='_monInt',OutputWorkspace='IvsLam')
CreateSingleValuedWorkspace(OutputWorkspace='__tmp_binary_operation_double')
Plus(LHSWorkspace='__tmp_binary_operation_double',RHSWorkspace='_110503',OutputWorkspace='__python_op_tmp1')
ConvertUnits(InputWorkspace='__python_op_tmp1',OutputWorkspace='sum_wavelength',Target='Wavelength',AlignBins='1')
CropWorkspace(InputWorkspace='sum_wavelength',OutputWorkspace='_monitor_ws',StartWorkspaceIndex='1',EndWorkspaceIndex='1')
CloneWorkspace(InputWorkspace='sum_wavelength',OutputWorkspace='_in_rng')
CropWorkspace(InputWorkspace='_in_rng',OutputWorkspace='_in_rng',StartWorkspaceIndex='2',EndWorkspaceIndex='2')
CropWorkspace(InputWorkspace='_in_rng',OutputWorkspace='_detector_ws',XMin='0.5',XMax='9')
CalculateFlatBackground(InputWorkspace='_monitor_ws',OutputWorkspace='_monitor_ws',StartX='7.5999999999999996',EndX='8.5',WorkspaceIndexList='0')
RebinToWorkspace(WorkspaceToRebin='_monitor_ws',WorkspaceToMatch='_detector_ws',OutputWorkspace='_i0p_trans')
Integration(InputWorkspace='_i0p_trans',OutputWorkspace='_mon_int_trans',RangeLower='4',RangeUpper='6')
Divide(LHSWorkspace='_detector_ws',RHSWorkspace='_mon_int_trans',OutputWorkspace='transWS')
RebinToWorkspace(WorkspaceToRebin='transWS',WorkspaceToMatch='IvsLam',OutputWorkspace='_i_vs_lam_trans')
Divide(LHSWorkspace='IvsLam',RHSWorkspace='_i_vs_lam_trans',OutputWorkspace='_i_vs_lam_corrected')
RenameWorkspace(InputWorkspace='_i_vs_lam_corrected',OutputWorkspace='IvsLam')
MoveInstrumentComponent(Workspace='IvsLam',ComponentName='point-detector',Y='0.031287739139067995',Z='11.43',RelativePosition='0')
ConvertUnits(InputWorkspace='IvsLam',OutputWorkspace='IvsQ',Target='MomentumTransfer')
RenameWorkspace(InputWorkspace='IvsQ',OutputWorkspace='110504_IvsQ')
CreateSingleValuedWorkspace(OutputWorkspace='__tmp_binary_operation_double')
Plus(LHSWorkspace='__tmp_binary_operation_double',RHSWorkspace='_110505',OutputWorkspace='__python_op_tmp0')
ConvertUnits(InputWorkspace='__python_op_tmp0',OutputWorkspace='sum_wavelength',Target='Wavelength',AlignBins='1')
CropWorkspace(InputWorkspace='sum_wavelength',OutputWorkspace='_monitor_ws',StartWorkspaceIndex='1',EndWorkspaceIndex='1')
CloneWorkspace(InputWorkspace='sum_wavelength',OutputWorkspace='_in_rng')
CropWorkspace(InputWorkspace='_in_rng',OutputWorkspace='_in_rng',StartWorkspaceIndex='2',EndWorkspaceIndex='2')
CropWorkspace(InputWorkspace='_in_rng',OutputWorkspace='_detector_ws',XMin='0.5',XMax='9')
CalculateFlatBackground(InputWorkspace='_monitor_ws',OutputWorkspace='_monitor_ws',StartX='7.5999999999999996',EndX='8.5',WorkspaceIndexList='0')
RebinToWorkspace(WorkspaceToRebin='_monitor_ws',WorkspaceToMatch='_detector_ws',OutputWorkspace='_I0P')
Integration(InputWorkspace='_I0P',OutputWorkspace='_monInt',RangeLower='4',RangeUpper='6')
Divide(LHSWorkspace='_detector_ws',RHSWorkspace='_monInt',OutputWorkspace='IvsLam')
RebinToWorkspace(WorkspaceToRebin='transWS',WorkspaceToMatch='IvsLam',OutputWorkspace='_i_vs_lam_trans')
Divide(LHSWorkspace='IvsLam',RHSWorkspace='_i_vs_lam_trans',OutputWorkspace='_i_vs_lam_corrected')
RenameWorkspace(InputWorkspace='_i_vs_lam_corrected',OutputWorkspace='IvsLam')
MoveInstrumentComponent(Workspace='IvsLam',ComponentName='point-detector',Y='0.058102261279893055',Z='11.43',RelativePosition='0')
ConvertUnits(InputWorkspace='IvsLam',OutputWorkspace='IvsQ',Target='MomentumTransfer')
RenameWorkspace(InputWorkspace='IvsQ',OutputWorkspace='110505_IvsQ')
Rebin(InputWorkspace='110504_IvsQ',OutputWorkspace='110504_IvsQ_binned',Params='0.011,-0.0376509,0.051')
Rebin(InputWorkspace='110505_IvsQ',OutputWorkspace='110505_IvsQ_binned',Params='0.02,-0.0376509,0.142471')
CloneWorkspace(InputWorkspace='110504_IvsQ_binned',OutputWorkspace='currentSum')
Stitch1D(LHSWorkspace='currentSum',RHSWorkspace='110505_IvsQ_binned',OutputWorkspace='currentSum')
RenameWorkspace(InputWorkspace='currentSum',OutputWorkspace='110504_50')
plotSpectrum('110504_50')

The plot is bad because of the first y-value in 110505, which is a NAN. We should be ignoring this! Anyway, changing 

Rebin(InputWorkspace='110505_IvsQ',OutputWorkspace='110505_IvsQ_binned',Params='0.02,-0.0376509,0.142471')

to 

Rebin(InputWorkspace='110505_IvsQ',OutputWorkspace='110505_IvsQ_binned',Params='0.0206,-0.0376509,0.142471')

Fixes the issue of the discontinuity at the intersection, but we should really be keeping Nans post processing.

Change History

comment:1 Changed 6 years ago by Owen Arnold

  • Blocked By 9956 added

comment:2 Changed 6 years ago by Nick Draper

  • Status changed from new to assigned

comment:3 Changed 6 years ago by Owen Arnold

The fix that the instrument scientist have asked for in Stitch1D is as follows:

  • After rebinning the data, find the positions of the special values (NaNs, Infs)
  • Perform the processing including determining the stitch overlap
  • Right at the end of the processing, put the special values back.
  • As a corollary to this, we may not need to perform the current call to ReplaceSpecialValues since the rebinned workspaces being integrated will already be missing the special values.

comment:4 Changed 6 years ago by Owen Arnold

  • Status changed from assigned to inprogress

comment:5 Changed 6 years ago by Owen Arnold

refs #9955. No need for replace special values.

Replace special values done as part of finding and recording those indexes.

Changeset: 0c5642237c428bd5ca151ef7435ba6cf8c7882c4

comment:6 Changed 6 years ago by Owen Arnold

refs #9955. Add test and introduce replace functionality.

Changeset: 1355491d395009ea197ae1fe7f7cab181e6baee3

comment:7 Changed 6 years ago by Owen Arnold

refs #9955. Special treatment for errors too

Changeset: 346f9eaa54e593bf89b3cd94243ae670b6e35efe

comment:8 Changed 6 years ago by Owen Arnold

refs #9955. Document and format.

Changeset: 7f5d87212f8baaedf134c648f50fff8434fd9702

comment:9 Changed 6 years ago by Owen Arnold

Tester:

  • Check the new documentation
  • New unit test covers this functionality
  • Running the following should include the perturbation due to the nan, but (a) the scale factors (error) and (value) with log=debug for Stitch1D should be real numbers, the continuation of the curve on both sides of the perturbation should look ok. 
SURF00110503=Load('SURF00110503')
SURF00110504=Load('SURF00110504')
SURF00110505=Load('SURF00110505')



RenameWorkspace('SURF00110504', OutputWorkspace='_110504')
RenameWorkspace('SURF00110503', OutputWorkspace='_110503')
RenameWorkspace('SURF00110505', OutputWorkspace='_110505')
CreateSingleValuedWorkspace(OutputWorkspace='__tmp_binary_operation_double')
Plus(LHSWorkspace='__tmp_binary_operation_double',RHSWorkspace='_110504',OutputWorkspace='__python_op_tmp0')
ConvertUnits(InputWorkspace='__python_op_tmp0',OutputWorkspace='sum_wavelength',Target='Wavelength',AlignBins='1')
CropWorkspace(InputWorkspace='sum_wavelength',OutputWorkspace='_monitor_ws',StartWorkspaceIndex='1',EndWorkspaceIndex='1')
CloneWorkspace(InputWorkspace='sum_wavelength',OutputWorkspace='_in_rng')
CropWorkspace(InputWorkspace='_in_rng',OutputWorkspace='_in_rng',StartWorkspaceIndex='2',EndWorkspaceIndex='2')
CropWorkspace(InputWorkspace='_in_rng',OutputWorkspace='_detector_ws',XMin='0.5',XMax='9')
CalculateFlatBackground(InputWorkspace='_monitor_ws',OutputWorkspace='_monitor_ws',StartX='7.5999999999999996',EndX='8.5',WorkspaceIndexList='0')
RebinToWorkspace(WorkspaceToRebin='_monitor_ws',WorkspaceToMatch='_detector_ws',OutputWorkspace='_I0P')
Integration(InputWorkspace='_I0P',OutputWorkspace='_monInt',RangeLower='4',RangeUpper='6')
Divide(LHSWorkspace='_detector_ws',RHSWorkspace='_monInt',OutputWorkspace='IvsLam')
CreateSingleValuedWorkspace(OutputWorkspace='__tmp_binary_operation_double')
Plus(LHSWorkspace='__tmp_binary_operation_double',RHSWorkspace='_110503',OutputWorkspace='__python_op_tmp1')
ConvertUnits(InputWorkspace='__python_op_tmp1',OutputWorkspace='sum_wavelength',Target='Wavelength',AlignBins='1')
CropWorkspace(InputWorkspace='sum_wavelength',OutputWorkspace='_monitor_ws',StartWorkspaceIndex='1',EndWorkspaceIndex='1')
CloneWorkspace(InputWorkspace='sum_wavelength',OutputWorkspace='_in_rng')
CropWorkspace(InputWorkspace='_in_rng',OutputWorkspace='_in_rng',StartWorkspaceIndex='2',EndWorkspaceIndex='2')
CropWorkspace(InputWorkspace='_in_rng',OutputWorkspace='_detector_ws',XMin='0.5',XMax='9')
CalculateFlatBackground(InputWorkspace='_monitor_ws',OutputWorkspace='_monitor_ws',StartX='7.5999999999999996',EndX='8.5',WorkspaceIndexList='0')
RebinToWorkspace(WorkspaceToRebin='_monitor_ws',WorkspaceToMatch='_detector_ws',OutputWorkspace='_i0p_trans')
Integration(InputWorkspace='_i0p_trans',OutputWorkspace='_mon_int_trans',RangeLower='4',RangeUpper='6')
Divide(LHSWorkspace='_detector_ws',RHSWorkspace='_mon_int_trans',OutputWorkspace='transWS')
RebinToWorkspace(WorkspaceToRebin='transWS',WorkspaceToMatch='IvsLam',OutputWorkspace='_i_vs_lam_trans')
Divide(LHSWorkspace='IvsLam',RHSWorkspace='_i_vs_lam_trans',OutputWorkspace='_i_vs_lam_corrected')
RenameWorkspace(InputWorkspace='_i_vs_lam_corrected',OutputWorkspace='IvsLam')
MoveInstrumentComponent(Workspace='IvsLam',ComponentName='point-detector',Y='0.031287739139067995',Z='11.43',RelativePosition='0')
ConvertUnits(InputWorkspace='IvsLam',OutputWorkspace='IvsQ',Target='MomentumTransfer')
RenameWorkspace(InputWorkspace='IvsQ',OutputWorkspace='110504_IvsQ')
CreateSingleValuedWorkspace(OutputWorkspace='__tmp_binary_operation_double')
Plus(LHSWorkspace='__tmp_binary_operation_double',RHSWorkspace='_110505',OutputWorkspace='__python_op_tmp0')
ConvertUnits(InputWorkspace='__python_op_tmp0',OutputWorkspace='sum_wavelength',Target='Wavelength',AlignBins='1')
CropWorkspace(InputWorkspace='sum_wavelength',OutputWorkspace='_monitor_ws',StartWorkspaceIndex='1',EndWorkspaceIndex='1')
CloneWorkspace(InputWorkspace='sum_wavelength',OutputWorkspace='_in_rng')
CropWorkspace(InputWorkspace='_in_rng',OutputWorkspace='_in_rng',StartWorkspaceIndex='2',EndWorkspaceIndex='2')
CropWorkspace(InputWorkspace='_in_rng',OutputWorkspace='_detector_ws',XMin='0.5',XMax='9')
CalculateFlatBackground(InputWorkspace='_monitor_ws',OutputWorkspace='_monitor_ws',StartX='7.5999999999999996',EndX='8.5',WorkspaceIndexList='0')
RebinToWorkspace(WorkspaceToRebin='_monitor_ws',WorkspaceToMatch='_detector_ws',OutputWorkspace='_I0P')
Integration(InputWorkspace='_I0P',OutputWorkspace='_monInt',RangeLower='4',RangeUpper='6')
Divide(LHSWorkspace='_detector_ws',RHSWorkspace='_monInt',OutputWorkspace='IvsLam')
RebinToWorkspace(WorkspaceToRebin='transWS',WorkspaceToMatch='IvsLam',OutputWorkspace='_i_vs_lam_trans')
Divide(LHSWorkspace='IvsLam',RHSWorkspace='_i_vs_lam_trans',OutputWorkspace='_i_vs_lam_corrected')
RenameWorkspace(InputWorkspace='_i_vs_lam_corrected',OutputWorkspace='IvsLam')
MoveInstrumentComponent(Workspace='IvsLam',ComponentName='point-detector',Y='0.058102261279893055',Z='11.43',RelativePosition='0')
ConvertUnits(InputWorkspace='IvsLam',OutputWorkspace='IvsQ',Target='MomentumTransfer')
RenameWorkspace(InputWorkspace='IvsQ',OutputWorkspace='110505_IvsQ')
Rebin(InputWorkspace='110504_IvsQ',OutputWorkspace='110504_IvsQ_binned',Params='0.011,-0.0376509,0.051')
Rebin(InputWorkspace='110505_IvsQ',OutputWorkspace='110505_IvsQ_binned',Params='0.02,-0.0376509,0.142471')
CloneWorkspace(InputWorkspace='110504_IvsQ_binned',OutputWorkspace='currentSum')
Stitch1D(LHSWorkspace='currentSum',RHSWorkspace='110505_IvsQ_binned',OutputWorkspace='currentSum')
RenameWorkspace(InputWorkspace='currentSum',OutputWorkspace='110504_50')
graph_spec=plotSpectrum('110504_50', 0)
l = graph_spec.activeLayer()
l.logLogAxes()

comment:10 Changed 6 years ago by Owen Arnold

  • Description modified (diff)

comment:11 Changed 6 years ago by Owen Arnold

  • Status changed from inprogress to verify
  • Resolution set to fixed

comment:12 Changed 6 years ago by Anders Markvardsen

  • Status changed from verify to verifying
  • Tester set to Anders Markvardsen

comment:13 Changed 6 years ago by Anders Markvardsen

  • Status changed from verifying to closed

Merge remote-tracking branch 'origin/feature/9955_nans'

Full changeset: ​41410b10f418e639101ebabda460fb916867ff44

comment:14 Changed 6 years ago by Anders Markvardsen

update to doc matches changes made to code and script runs with continuation on both sides

comment:15 Changed 6 years ago by Peter Peterson

  • Keywords PatchCandidate removed
  • Milestone changed from Release 3.3 to Release 3.2.1

comment:16 Changed 6 years ago by Martyn Gigg

Although not technically blocked by #9954, it there was some changes to Stitch1D there that mean this will only merge cleanly of #9954 goes in first.

comment:17 Changed 6 years ago by Owen Arnold

Cherry-pick fixes from feature/9955_nans into next

Changes:

refs #9955. No need for replace special values.

Replace special values done as part of finding and recording those indexes. (cherry picked from commit 0c5642237c428bd5ca151ef7435ba6cf8c7882c4)

refs #9955. Add test and introduce replace functionality. (cherry picked from commit 1355491d395009ea197ae1fe7f7cab181e6baee3)

refs #9955. Special treatment for errors too (cherry picked from commit 346f9eaa54e593bf89b3cd94243ae670b6e35efe)

refs #9955. Document and format. (cherry picked from commit 7f5d87212f8baaedf134c648f50fff8434fd9702)

Changeset: e834c2a0c1141e3ee258d6d756bcd3aa2e853203

comment:18 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue ​10797

Note: See TracTickets for help on using tickets.