Quick to Algorithms

[Owen Arnold]

Users want the quick script to be turned into a decent set of Mantid algorithms. Then gut quick. Later on kill the quick script altogether.

1 Week

[Owen Arnold]

Latest idea is to have this algorithm as a c++ workflow algorithm with a top level algorithm called something like "RefelectometryReduction" to be implemented shortly.

My feeling is that we shouldn't call this algorithm 'Quick' even though It would be the easiest go-to for the users. I think something like ReflectometryRecuctionOne, to highlight that this algorithm is only handling the reduction for a single run. I'm sure the users will want me to rename it to something else in the near future, but I'll go with that for now.

[Owen Arnold]

refs #8371. Template for quick algorithm.

Changeset: 29e776f5d82254fc0bf7c95b26edc22ba57403ea

[Owen Arnold]

refs #8371. Sketch out workflow inputs.

Changeset: 8346d00fef1b606258b58715908a091ca283671a

[Owen Arnold]

refs #8371. Extend inputs and test to make more like quick.

A new builder has been added to the test class, which can dynamically create builders around mantid algorithms. This is used to create a standard algorithm for testing, which can be copied around before executing. This is probably generally useful for python unit testing.

Changeset: bcc568323fa958817b0708ef08086b4dd4bf6fef

[Owen Arnold]

refs #8371. Background and integration limit inputs.

These are very different to the overall wavelength min and max inputs already provided. Extend test coverage for these.

Changeset: dffbce2599ee7180d2b44381c66127a3cfa6eb66

[Owen Arnold]

refs #8371. Better logic around transmission runs.

More tests, and also testing around workspace index lists.

Changeset: 609be63ed10ba16186c98e9685071b6679bf6504

[Owen Arnold]

refs #8371. Region of interest direct beam.

Read property if not point detector mode. Quit if point detector mode.

Changeset: 74f76827a805a3b1ecb09376ca68daeb22209f8b

[Owen Arnold]

refs #8371. Extend and refactor inputs.

Extend checking for roi and db inputs. Refactor validation and setting of inputs out into separate methods where possible to separate from future processing logic.

Changeset: 5bd84ec3b496acf4acab0daf918af24ee81c373c

[Owen Arnold]

refs #8371. toLam functionality implemented.

Start work on toLam functionality using convert_to_wavlength.py as guide. Only the detector side of things has been implemented thus far, but test coverage is quite good, and I have implemented the pieces that mean that detector id ranges can be used.

Changeset: a7ce35c3bcb469943bfb66163ddfbfbfbb82ea5d

[Owen Arnold]

refs #8371. Handle detectors and monitors in toLam.

Extend tests and functionality to handle monitor conversion and flat background correction.

Changeset: 7b6a36bd8d7da39791dc52cbad30a80be97729e7

[Owen Arnold]

refs #8371. Refactor toLam code. Method too large.

Better to break this function up into its monitor and detector components.

Changeset: a0ca7a54c5601edcdc8033b6da958933360a7aae

[Owen Arnold]

refs #8371. Transmission correction function added.

Basic transmission correction functionality. Will need refactoring as we go along. Only working for point detector mode with a single transmission workspace at the moment. No stitching yet.

Changeset: 3adcfcd773fa4844983a44820fdbfa5a3b5db5a9

[Owen Arnold]

refs #8371. Transmission correction with two corrections.

This involves using Stitch1D (currently a python algorithm). In order to do this I need to work around this interface a little by pushing the input workspaces into the ADS prior to algorithm execution, and cleaning them out again afterwards. When I move Stitch1D to be a c++ algorithm, I will no longer need to do this. More refactoring work is required around these new features.

Changeset: 4d78e0a2023c3a200580d9bbaa09fc95719c7657

[Owen Arnold]

refs #8371. Make StartOverlap and EndOverlap properties.

Previously these were hard-coded in the quick scripts, but it's very important that they are exposed. Add checking and tests around these features. Refactoring step required to separate validation from setting of these optional parameters.

Changeset: ed4858b34fee1abb6abc606157d94ef9aa2d5abf

[Owen Arnold]

refs #8371. Intermediate refactoring.

Refactoring before I start tackling the rest of the multi-detector options, and IvsQ aspects of the conversion. Improve the documentation and make the methods const where possible.

Changeset: 1b7e2bcd85fe689577b79a8564d8d27091b9060a

[Owen Arnold]

refs #8371. Explicit rebin and fix tests.

I have used the existing quick functionality to check that everything up to and including the transmission correction functionality for point detectors is working (Have a script for this). Remove the Align bins and explicitly renin, although the wavelength step is hard-coded, and should probably be exposed.

Changeset: 3f14bed1160c2c85d0b8b9eeb3c758fb5a7d6224

[Owen Arnold]

class ReflectometryQuick(object):
  
    def runTest(self):
        defaultInstKey = 'default.instrument'
        defaultInstrument = config[defaultInstKey]
        
        config[defaultInstKey] = 'INTER'
        LoadISISNexus(Filename='13463', OutputWorkspace='13463')
        LoadISISNexus(Filename='13464', OutputWorkspace='13464')
        LoadISISNexus(Filename='13460', OutputWorkspace='13460')
    
        transmissionRuns = '13463,13464'
        #transmissionRuns = '13463'
        runNo = '13460'
        incidentAngle = 0.7
        quick.quick(runNo, trans=transmissionRuns, theta=incidentAngle) 
        
        defaults =quick.get_defaults(mtd['13460'])
        print defaults

        config[defaultInstKey] = defaultInstrument
        
        # new is in lambda! later it will be in IvsQ
        new_IvsLam = ReflectometryReductionOne(InputWorkspace='13460', WavelengthMin=1.0, WavelengthMax=17.0, MonitorBackgroundWavelengthMin=15.0, MonitorBackgroundWavelengthMax=17.0, I0MonitorIndex=2, MonitorIntegrationWavelengthMin=4.0, MonitorIntegrationWavelengthMax=10.0, WorkspaceIndexList=[3,4], Params=[1.5, 0.02, 17], StartOverlapQ=10.0, EndOverlapQ=12.0, FirstTransmissionRun='13463', SecondTransmissionRun='13464')
        
        plotSpectrum([new_IvsLam,'13460_IvsLam'], [0])
        
        
test = ReflectometryQuick()
test.runTest()

This has been used to demonstrate the lambda side of the functionality working.

[Owen Arnold]

refs #8371. Allow the rebinning step to be specified.

This has been hard-coded previously. Better that we allow it to be externally controlled.

Changeset: 2ce37cdbeaed2d6aa1d2f9bdd0b9fdb50b2c8bc5

[Owen Arnold]

refs #8371. Layout for toQ and pos corrections

This functionality was previously wrapped up in the l2q script. However, I'm going to need to merge in changes from master to fully get this working, and i'm also going to need to provide some testing over these features.

Changeset: f1965fc38b19e88e78525af085ed6a93d8f39c58

[Owen Arnold]

refs #8371. Template for quick algorithm.

Changeset: a080c455547a113fdeab59ebe7414faf8b44e41a

[Owen Arnold]

refs #8371. Sketch out workflow inputs.

Changeset: 22db2a13d33d6fad982f8d97d483b74889288bd7

[Owen Arnold]

refs #8371. Extend inputs and test to make more like quick.

A new builder has been added to the test class, which can dynamically create builders around mantid algorithms. This is used to create a standard algorithm for testing, which can be copied around before executing. This is probably generally useful for python unit testing.

Changeset: d7fda0ac11e0dca864fe750371ff85cedf641077

[Owen Arnold]

refs #8371. Background and integration limit inputs.

These are very different to the overall wavelength min and max inputs already provided. Extend test coverage for these.

Changeset: 62565b75ae7f51b9639e304ab2311b499cc8a62b

[Owen Arnold]

refs #8371. Better logic around transmission runs.

More tests, and also testing around workspace index lists.

Changeset: 248ad715437865e90efa4015fbf0cf90d46b38e8

[Owen Arnold]

refs #8371. Region of interest direct beam.

Read property if not point detector mode. Quit if point detector mode.

Changeset: 974968931477aba474aa335c5e20c222dfe63d58

[Owen Arnold]

refs #8371. Extend and refactor inputs.

Extend checking for roi and db inputs. Refactor validation and setting of inputs out into separate methods where possible to separate from future processing logic.

Changeset: fa3b19ff5af50bc08718c8e8e05864c2951199b1

[Owen Arnold]

refs #8371. toLam functionality implemented.

Start work on toLam functionality using convert_to_wavlength.py as guide. Only the detector side of things has been implemented thus far, but test coverage is quite good, and I have implemented the pieces that mean that detector id ranges can be used.

Changeset: 08826bed084801bbd5bf217d9f692057bc9f96bb

[Owen Arnold]

refs #8371. Handle detectors and monitors in toLam.

Extend tests and functionality to handle monitor conversion and flat background correction.

Changeset: fde013a2118f85959e56f9658f770c81d8fceafc

[Owen Arnold]

refs #8371. Refactor toLam code. Method too large.

Better to break this function up into its monitor and detector components.

Changeset: 2857bc92990dd3c429a246c1ab42e8b9d1082ce4

[Owen Arnold]

refs #8371. Transmission correction function added.

Basic transmission correction functionality. Will need refactoring as we go along. Only working for point detector mode with a single transmission workspace at the moment. No stitching yet.

Changeset: 48400b677592cc989fe522b7efe18cdbcbd158ce

[Owen Arnold]

refs #8371. Transmission correction with two corrections.

This involves using Stitch1D (currently a python algorithm). In order to do this I need to work around this interface a little by pushing the input workspaces into the ADS prior to algorithm execution, and cleaning them out again afterwards. When I move Stitch1D to be a c++ algorithm, I will no longer need to do this. More refactoring work is required around these new features.

Changeset: 79e85b5d14522ab22951632afa875e15c9fed777

[Owen Arnold]

refs #8371. Make StartOverlap and EndOverlap properties.

Previously these were hard-coded in the quick scripts, but it's very important that they are exposed. Add checking and tests around these features. Refactoring step required to separate validation from setting of these optional parameters.

Changeset: 3b45a3c021af7525f2b5871f378397abbb6b14ab

[Owen Arnold]

refs #8371. Intermediate refactoring.

Refactoring before I start tackling the rest of the multi-detector options, and IvsQ aspects of the conversion. Improve the documentation and make the methods const where possible.

Changeset: a02c311f047b873543c6864c8cb89d10d0b3e819

[Owen Arnold]

refs #8371. Explicit rebin and fix tests.

I have used the existing quick functionality to check that everything up to and including the transmission correction functionality for point detectors is working (Have a script for this). Remove the Align bins and explicitly renin, although the wavelength step is hard-coded, and should probably be exposed.

Changeset: f572122a565901cf337c081d12c5c97b42937561

[Owen Arnold]

refs #8371. Allow the rebinning step to be specified.

This has been hard-coded previously. Better that we allow it to be externally controlled.

Changeset: 34cbcbbe88683032e0a89a471b4e06c46504f01a

[Owen Arnold]

refs #8371. Layout for toQ and pos corrections

This functionality was previously wrapped up in the l2q script. However, I'm going to need to merge in changes from master to fully get this working, and i'm also going to need to provide some testing over these features.

Changeset: de39927f6070e22de7b53153821be63fc4fa1c84

[Owen Arnold]

refs #8371. Get working with move instrument components.

Comments on what needs doing next for the point detector run.

Changeset: 9b8a7e07a827026458d6ff83a635c2729081188c

[Owen Arnold]

refs #8371. Multiple output workspaces supported.

There is a need to fetch out the full q-workspace as well as the intermediate lam-workspace.

Changeset: b5755fe33aa21587e0cffba3a9c799cffc68aa8a

[Owen Arnold]

The following python script shows this working in point detector mode with a supplied theta value.

from mantid.simpleapi import *
from isis_reflectometry import quick
reload(quick)

class ReflectometryQuick(object):
  
    def runTest(self):
        defaultInstKey = 'default.instrument'
        defaultInstrument = config[defaultInstKey]
        
        config[defaultInstKey] = 'INTER'
        LoadISISNexus(Filename='13463', OutputWorkspace='13463')
        LoadISISNexus(Filename='13464', OutputWorkspace='13464')
        LoadISISNexus(Filename='13460', OutputWorkspace='13460')
    
        transmissionRuns = '13463,13464'
        #transmissionRuns = '13463'
        runNo = '13460'
        incidentAngle = 0.7
        quick.quick(runNo, trans=transmissionRuns, theta=incidentAngle) 
        
        defaults =quick.get_defaults(mtd['13460'])
        print defaults

        config[defaultInstKey] = defaultInstrument
        
        # new is in lambda! later it will be in IvsQ
        new_IvsQ, new_IvsLam = ReflectometryReductionOne(InputWorkspace='13460', Theta=incidentAngle, WavelengthMin=1.0, WavelengthMax=17.0, WavelengthStep=0.05, MonitorBackgroundWavelengthMin=15.0, MonitorBackgroundWavelengthMax=17.0, I0MonitorIndex=2, MonitorIntegrationWavelengthMin=4.0, MonitorIntegrationWavelengthMax=10.0, WorkspaceIndexList=[3,4], Params=[1.5, 0.02, 17], StartOverlapQ=10.0, EndOverlapQ=12.0, FirstTransmissionRun='13463', SecondTransmissionRun='13464')
        
        plotSpectrum([new_IvsLam,'13460_IvsLam'], [0])
        plotSpectrum([new_IvsQ, '13460_IvsQ'], [0])
        
        
test = ReflectometryQuick()
test.runTest()

[Owen Arnold]

refs #8371. Possible injection of component names.

The component names may be specified externally. That way we have no hard-coded values. The algorithm will guess at component names to use, but will fail if they don't exist and the user can provide these names via the algorithm inputs.

Changeset: 3da1bf40f7efcf951c96047316e69b9c2de81fb8

[Owen Arnold]

refs #8371. Calculate theta if not given.

This replicates the current functionality of quick. Tests show this working. Results have been compared to existing quick script.

Changeset: 48da2d18e699032c633a493d7ed0661e71798fe7

[Owen Arnold]

refs #8371. Ready for multi-detector implementation.

Disable properties that are not relevant in the gui between multi detector and line detector analysis modes. Move code to common blocks where possible.

Changeset: cea4f06eacfdb65c1afed5b24c4b028716a74547

[Owen Arnold]

refs #8371. Multi-detector processing.

Multidetector processing using roi and db. The normalisation step seems to be missing from the original quick script, but it probably should have been there. This will require further validation.

Changeset: dfb6d52774f404de64db2dcf8112b12f1fcb48b0

[Owen Arnold]

refs #8371. Re-adjust input layout.

This is a workflow algorithm with a log of input properties. Lets help the user narrow down the choices that can be made.

Changeset: cdd7505a277f9266985c47cb10a5a0683f95bc5b

[Owen Arnold]

refs #8371. Build on windows and fix warnings.

Changeset: 410924fdd0d5c10759a953f752140d4839deb9f8

[Owen Arnold]

refs #8371. Fix unused variable warning.

Changeset: 536b7a9a2333608c5181fd1fcb5beac92d525756

[Owen Arnold]

Tester: The following script shows this algorithm working against quick. There are some minor differences owing to the fact that we are now providing an explicit rebin rather than using the AlignBins flag. The plots will show that this new algorithm is doing the same job as the old scripts.

• I have provided a large number of unit tests for this functionality.
• The code changes are very large (apologies), but isolated to a few files. I don't think there's much point trying to do a thorough code review of these changes.

from mantid.simpleapi import *
from isis_reflectometry import quick
reload(quick)

class ReflectometryQuick(object):
  
    def runTest(self):
        defaultInstKey = 'default.instrument'
        defaultInstrument = config[defaultInstKey]
        
        config[defaultInstKey] = 'INTER'
        LoadISISNexus(Filename='13463', OutputWorkspace='13463')
        LoadISISNexus(Filename='13464', OutputWorkspace='13464')
        LoadISISNexus(Filename='13460', OutputWorkspace='13460')
    
        transmissionRuns = '13463,13464'
        #transmissionRuns = '13463'
        runNo = '13460'
        incidentAngle = 0.7
        quick.quick(runNo, trans=transmissionRuns) 
        
        defaults =quick.get_defaults(mtd['13460'])
        print defaults

        config[defaultInstKey] = defaultInstrument
        
        # new is in lambda! later it will be in IvsQ
        new_IvsQ, new_IvsLam, thetaOut = ReflectometryReductionOne(InputWorkspace='13460', 
                                                                                                                WavelengthMin=1.0, 
                                                                                                                WavelengthMax=17.0, 
                                                                                                                WavelengthStep=0.05, 
                                                                                                                MonitorBackgroundWavelengthMin=15.0, 
                                                                                                                MonitorBackgroundWavelengthMax=17.0, 
                                                                                                                I0MonitorIndex=2, 
                                                                                                                MonitorIntegrationWavelengthMin=4.0, 
                                                                                                                MonitorIntegrationWavelengthMax=10.0, 
                                                                                                                WorkspaceIndexList=[3,4], 
                                                                                                                Params=[1.5, 0.02, 17], 
                                                                                                                StartOverlapQ=10.0, 
                                                                                                                EndOverlapQ=12.0, 
                                                                                                                FirstTransmissionRun='13463', 
                                                                                                                SecondTransmissionRun='13464')
        
        plotSpectrum([new_IvsLam,'13460_IvsLam'], [0])
        plotSpectrum([new_IvsQ, '13460_IvsQ'], [0])
        
        
test = ReflectometryQuick()
test.runTest()

[Owen Arnold]

refs #8371. Wiki documentation provided.

Changeset: 306866d0e860318f3a4ff2666c4b28db7c75fa64

[Peter Parker]

Merge remote-tracking branch 'origin/feature/8371_quick_algorithm'

Full changeset: ​82332bfe4397401d936c3cfde1415cf57f930ac5

[Peter Parker]

The script does indeed show the very similar results of running the new algorithm against the old quick code. The unit tests seem ample and are passing.

I've raised #8591 to move the checkNotPositive helper function into the anonymous namespace.

[Martyn Gigg]

Added note to #8591 about addressing compiler warning in ReflectometryReductionOne.cpp.

[Stuart Campbell]

This ticket has been transferred to github issue ​9216