Direct reduction -- Summation with wait for files and diagnostics for list of files

[Alex Buts]

Unit test for DirectEnergyConversionTest: test_multirep_abs_units_mode will fail if diagnostics is enabled. In addition to that, file summation does not work if run from local Mantid and wait for files is enabled.

The purpose of this ticket is to provide reliable workflow of the autoreduction script for multiple files when run from Mantid or proof that current workflow is correct for any data source (workspace, file or run-number)

(I am more or less confident that it works fine for autoreduction when single run is processed, but the issues above suggest that there are still problems when run locally multiple times with only some files changing)

[Alex Buts]

Re #11096 Streamlining various set options in RunDescriptor

Paying special attention to the case, when list of files is assigned. (nothing still works properly)

Changeset: 1cc6ada69562815d4ab6942533a13b391cccb3ba

[Alex Buts]

Re #11096 fixed autoreduce workflow with summation

when refactored to new summation class

Changeset: ff9ab4c04d9f9ebd6547fad852eed2262c799413

[Alex Buts]

Re #11096 Working sums cash and unit test for it

Changeset: 588439eac81bc70ddba923077a167777a3846886

[Alex Buts]

Re #11096 Stub for a cached RunDescriptor mask and its applications

Changeset: fdd160dc1cce228c5010c191da1fb9735bfcc58e

[Alex Buts]

Re #11096 minor bugs and typos

Changeset: f8fa765063866b5352829dc3a52af649aada7e0c

[Alex Buts]

Re #11096 Minor changes and typo to fix system test

in accordance with changes to main reduction

Changeset: 68b3d6d4027516ffa885fca7589cc03e58d58aaa

[Alex Buts]

Re #11096 Implemented reduce_sum while waiting for file

all seems works but couple of additional system tests would be useful

Changeset: bd1b8c0d7e2e7e9dce19be9338e72f0b38c6c47a

[Alex Buts]

Re #11096 should fix system tests (some of them at least)

Changeset: c2f88b09f3118e6ca5f1b3b94d16f39eb5c758b9

[Alex Buts]

Re #11096 Implemented cached masks

and all seems works though system test would be useful

Changeset: c920db4278a73bcc6fddc3997acbbd78a97d8212

[Alex Buts]

Re #11096 All cashes work

Changeset: 3aa8ca48edc6d30deb1adef21d345ab329980fdf

[Alex Buts]

Re #11096 Minor typos and bug fixes

also test and method to provide custom file format procedure

Changeset: 91285bc4d93dd8fe4cdca9d0b5dd98a0413b034d

[Alex Buts]

Re #11096 One more system test and better clean up

Changeset: d0f7672c1c5522b7fe825d2d876dba9b10e7f7cf

[Alex Buts]

Re #11096 fixing conflicts causes by pylint changes

Merge branch 'master' into 11096/SumFilesAndDiagMultipleRuns

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py Code/Mantid/scripts/Inelastic/Direct/PropertiesDescriptors.py Code/Mantid/scripts/Inelastic/Direct/PropertyManager.py Code/Mantid/scripts/Inelastic/Direct/ReductionWrapper.py Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py Code/Mantid/scripts/Inelastic/Direct/diagnostics.py

Changeset: 0d251fe3b250c82ccb8eebd50dc297540c553df2

[Alex Buts]

Re #11096 Bugs induced by merging master with pylint changes

Changeset: 79584187198a47da04592d996100433aebdb89bb

[Alex Buts]

Re #11096 Merge branch '11096/SumFilesAndDiagMultipleRuns'into develop

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py Code/Mantid/scripts/Inelastic/Direct/PropertiesDescriptors.py Code/Mantid/scripts/Inelastic/Direct/PropertyManager.py Code/Mantid/scripts/Inelastic/Direct/ReductionHelpers.py Code/Mantid/scripts/Inelastic/Direct/ReductionWrapper.py Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py Code/Mantid/scripts/Inelastic/Direct/diagnostics.py

Changeset: 692a654587406165fee38aae607abd57d346cd35

[Alex Buts]

Re #11096 fixed bugs in wait_for_file procedure

As the procedure without tests have deteriorated. Also framework for tests to avoid it in a future and tests themselves (in system tests branch)

Changeset: a19885787da56aee8f2e8cc146b87efd146e2bc8

[Alex Buts]

Re #11096 Tests for reduction with wait for files workflow

Changeset: 9636db41fbd38f014823f712c6528e6c6c9eeab7

[Alex Buts]

Should be tested only after #11063 is merged to master. (spawned from that branch, though diverged due to pylint changes later)

The changes are both in System tests and Mantid branches, so no pull request for that yet.

The essence of the ticket is to improve file summation and running range of files one after another -- the way which is the same regardless if script is run from web service or Mantid script window.

Purpose is achieved, unit tests fixed and number of additional system test are written. The tests confirm the validity of changes. Otherwise -- test by code review

[Alex Buts]

Re #11096 Interface function to provide custom file name procedure

Changeset: c933ff28602de9d57042c97bc8c749f4e361b7dd

[Alex Buts]

Re #11096 Typo in ReductionWrapper

Changeset: de9777925ac0fcb76808ceb22128700a91ee2e4b

[Alex Buts]

Re #11096 some conflicts with develop

Merge branch '11096/SumFilesAndDiagMultipleRuns' into develop

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/ReductionWrapper.py

Changeset: b0ff1a9d3e27e340f69619f9f593e4c53c316af5

[Alex Buts]

Re #11096 minor changes to internal save_results routine

do not split internal file-name to name and extension if . is present in this file name

Changeset: 312d4c84244a9381c11d0910649e004ceb919c95

[Alex Buts]

Re #11096 conflicts with develop

Merge branch '11096/SumFilesAndDiagMultipleRuns' into develop

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py

Changeset: a511884a7739300755f08f847e090bc9a32b66c6

[Alex Buts]

Re #11096 Merge branch 'master' into 11096/SumFilesAndDiagMultipleRuns

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py Code/Mantid/scripts/test/DirectEnergyConversionTest.py Code/Mantid/scripts/test/DirectPropertyManagerTest.py Code/Mantid/scripts/test/MariReduction.py Code/Mantid/scripts/test/RunDescriptorTest.py

Changeset: 3865d944ca6324946174fbb99d04f3d3d7f1af7a

[Alex Buts]

Re #11096 Merge branch 'master' into 11096/SumFilesAndDiagMultipleRuns

and fixing bugs introduced by merge

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py Code/Mantid/scripts/Inelastic/Direct/PropertiesDescriptors.py Code/Mantid/scripts/Inelastic/Direct/PropertyManager.py Code/Mantid/scripts/Inelastic/Direct/ReductionWrapper.py Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py Code/Mantid/scripts/test/DirectEnergyConversionTest.py Code/Mantid/scripts/test/MariReduction.py Code/Mantid/scripts/test/RunDescriptorTest.py

Changeset: b2d505a974a9482e6c05b8d197634643bafba6fa

[Alex Buts]

Re #11096 Fixing bugs caused by merge to master

(with pylint changes)

Changeset: 01f053d6f97c5d1e9e97a5cfcf41258fb3db6450

[Alex Buts]

Re #11096 Merge branch '11096/SumFilesAndDiagMultipleRuns' into dev

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py Code/Mantid/scripts/Inelastic/Direct/PropertiesDescriptors.py Code/Mantid/scripts/Inelastic/Direct/PropertyManager.py Code/Mantid/scripts/Inelastic/Direct/ReductionHelpers.py Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py

Changeset: 21c6276c74ad7700d3bc2c6cfa442037a8b7f9d3

[Alex Buts]

Re #11096 Adding file accidentally deleted earlier

Changeset: 9a7ebcd38abbb0b5b1fded4082b77ea1519d7de3

[Alex Buts]

Re #11096 restoring deleted file mode

Changeset: 04dab9ddd1b7d72b1067b7c2fa6309103658fa32

[Dan Nixon]

Merge remote-tracking branch 'origin/11096/SumFilesAndDiagMultipleRuns'

Full changeset: ​caf71182f0ccabd2a5cb0f442b6329c04081d8d3

[Dan Nixon]

Merge remote-tracking branch 'origin/11096/SumFilesAndDiagMultipleRuns'

Full changeset: ​9bef2e6b842b0be8a6012b777c963b466a16fbb4

[Mathieu Doucet]

Re #11096 Major refactoring in progress

Changeset: 0e378c6ffd677f912a6f8a86edd6be0d74575019

[Mathieu Doucet]

Re #11096 First version without wks_utility

Changeset: 4266609f989916c5e32dc4a9fa0b26dc5d32ecc5

[Mathieu Doucet]

Re #11096 First version without wks_utility

Changeset: 8a8eb8066c5726ade3cb5534da0a6a90d1f1ff6b

[Mathieu Doucet]

Re #11096 First cut of refactored algorithm

Changeset: f3eed89fc23aad9f607b4eddbd98f64d799ff67a

[Mathieu Doucet]

Re #11096 Backward compatibility issue

Changeset: 51b5d33da534d4645402134cdafbfddbb62a6472

[Mathieu Doucet]

Re #11096 Add data file MD5

Changeset: 7c84372341ac6c9537a3cae0af0bc2c3b993de4a

[Mathieu Doucet]

Re #11096 Add ref data MD5

Changeset: a8a5288bb19960ade0d03e8840eeb0f6ee3f9e2f

[Mathieu Doucet]

Re #11096 Add test data MD5

Changeset: 294494e971330edb34344b47dd23fc6dc27d7e79

[Mathieu Doucet]

Re #11096 Add test data MD5

Changeset: 98e92d0db4b287e9b2cdeaa06634c969b01735eb

[Stuart Campbell]

This ticket has been transferred to github issue ​11935