Ticket #11096 (closed: fixed)
Direct reduction -- Summation with wait for files and diagnostics for list of files
Reported by: | Alex Buts | Owned by: | Alex Buts |
---|---|---|---|
Priority: | major | Milestone: | Release 3.3.1 |
Component: | Framework | Keywords: | |
Cc: | Blocked By: | #11063 | |
Blocking: | #11155, #11177 | Tester: | Dan Nixon |
Description (last modified by Alex Buts) (diff)
Unit test for DirectEnergyConversionTest: test_multirep_abs_units_mode will fail if diagnostics is enabled. In addition to that, file summation does not work if run from local Mantid and wait for files is enabled.
The purpose of this ticket is to provide reliable workflow of the autoreduction script for multiple files when run from Mantid or proof that current workflow is correct for any data source (workspace, file or run-number)
(I am more or less confident that it works fine for autoreduction when single run is processed, but the issues above suggest that there are still problems when run locally multiple times with only some files changing)
Change History
comment:1 Changed 6 years ago by Alex Buts
- Status changed from new to assigned
- Description modified (diff)
comment:2 Changed 6 years ago by Alex Buts
- Status changed from assigned to inprogress
Re #11096 Streamlining various set options in RunDescriptor
Paying special attention to the case, when list of files is assigned. (nothing still works properly)
Changeset: 1cc6ada69562815d4ab6942533a13b391cccb3ba
comment:3 Changed 6 years ago by Alex Buts
Re #11096 fixed autoreduce workflow with summation
when refactored to new summation class
Changeset: ff9ab4c04d9f9ebd6547fad852eed2262c799413
comment:4 Changed 6 years ago by Alex Buts
Re #11096 Working sums cash and unit test for it
Changeset: 588439eac81bc70ddba923077a167777a3846886
comment:5 Changed 6 years ago by Alex Buts
Re #11096 Stub for a cached RunDescriptor mask and its applications
Changeset: fdd160dc1cce228c5010c191da1fb9735bfcc58e
comment:6 Changed 6 years ago by Alex Buts
Re #11096 minor bugs and typos
Changeset: f8fa765063866b5352829dc3a52af649aada7e0c
comment:7 Changed 6 years ago by Alex Buts
Re #11096 Minor changes and typo to fix system test
in accordance with changes to main reduction
Changeset: 68b3d6d4027516ffa885fca7589cc03e58d58aaa
comment:8 Changed 6 years ago by Alex Buts
Re #11096 Implemented reduce_sum while waiting for file
all seems works but couple of additional system tests would be useful
Changeset: bd1b8c0d7e2e7e9dce19be9338e72f0b38c6c47a
comment:9 Changed 6 years ago by Alex Buts
Re #11096 should fix system tests (some of them at least)
Changeset: c2f88b09f3118e6ca5f1b3b94d16f39eb5c758b9
comment:10 Changed 6 years ago by Alex Buts
Re #11096 Implemented cached masks
and all seems works though system test would be useful
Changeset: c920db4278a73bcc6fddc3997acbbd78a97d8212
comment:11 Changed 6 years ago by Alex Buts
Re #11096 All cashes work
Changeset: 3aa8ca48edc6d30deb1adef21d345ab329980fdf
comment:12 Changed 6 years ago by Alex Buts
Re #11096 Minor typos and bug fixes
also test and method to provide custom file format procedure
Changeset: 91285bc4d93dd8fe4cdca9d0b5dd98a0413b034d
comment:13 Changed 6 years ago by Alex Buts
Re #11096 One more system test and better clean up
Changeset: d0f7672c1c5522b7fe825d2d876dba9b10e7f7cf
comment:15 Changed 6 years ago by Alex Buts
Re #11096 fixing conflicts causes by pylint changes
Merge branch 'master' into 11096/SumFilesAndDiagMultipleRuns
Conflicts:
Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py Code/Mantid/scripts/Inelastic/Direct/PropertiesDescriptors.py Code/Mantid/scripts/Inelastic/Direct/PropertyManager.py Code/Mantid/scripts/Inelastic/Direct/ReductionWrapper.py Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py Code/Mantid/scripts/Inelastic/Direct/diagnostics.py
Changeset: 0d251fe3b250c82ccb8eebd50dc297540c553df2
comment:16 Changed 6 years ago by Alex Buts
Re #11096 Bugs induced by merging master with pylint changes
Changeset: 79584187198a47da04592d996100433aebdb89bb
comment:17 Changed 6 years ago by Alex Buts
Re #11096 Merge branch '11096/SumFilesAndDiagMultipleRuns'into develop
Conflicts:
Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py Code/Mantid/scripts/Inelastic/Direct/PropertiesDescriptors.py Code/Mantid/scripts/Inelastic/Direct/PropertyManager.py Code/Mantid/scripts/Inelastic/Direct/ReductionHelpers.py Code/Mantid/scripts/Inelastic/Direct/ReductionWrapper.py Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py Code/Mantid/scripts/Inelastic/Direct/diagnostics.py
Changeset: 692a654587406165fee38aae607abd57d346cd35
comment:18 Changed 6 years ago by Alex Buts
Re #11096 fixed bugs in wait_for_file procedure
As the procedure without tests have deteriorated. Also framework for tests to avoid it in a future and tests themselves (in system tests branch)
Changeset: a19885787da56aee8f2e8cc146b87efd146e2bc8
comment:19 Changed 6 years ago by Alex Buts
Re #11096 Tests for reduction with wait for files workflow
Changeset: 9636db41fbd38f014823f712c6528e6c6c9eeab7
comment:20 Changed 6 years ago by Alex Buts
- Status changed from inprogress to verify
- Resolution set to fixed
- Milestone changed from Release 3.4 to Release 3.3.1
Should be tested only after #11063 is merged to master. (spawned from that branch, though diverged due to pylint changes later)
The changes are both in System tests and Mantid branches, so no pull request for that yet.
The essence of the ticket is to improve file summation and running range of files one after another -- the way which is the same regardless if script is run from web service or Mantid script window.
Purpose is achieved, unit tests fixed and number of additional system test are written. The tests confirm the validity of changes. Otherwise -- test by code review
comment:22 Changed 6 years ago by Alex Buts
Re #11096 Interface function to provide custom file name procedure
Changeset: c933ff28602de9d57042c97bc8c749f4e361b7dd
comment:23 Changed 6 years ago by Alex Buts
Re #11096 Typo in ReductionWrapper
Changeset: de9777925ac0fcb76808ceb22128700a91ee2e4b
comment:24 Changed 6 years ago by Alex Buts
Re #11096 some conflicts with develop
Merge branch '11096/SumFilesAndDiagMultipleRuns' into develop
Conflicts:
Code/Mantid/scripts/Inelastic/Direct/ReductionWrapper.py
Changeset: b0ff1a9d3e27e340f69619f9f593e4c53c316af5
comment:25 Changed 6 years ago by Alex Buts
Re #11096 minor changes to internal save_results routine
do not split internal file-name to name and extension if . is present in this file name
Changeset: 312d4c84244a9381c11d0910649e004ceb919c95
comment:26 Changed 6 years ago by Alex Buts
Re #11096 conflicts with develop
Merge branch '11096/SumFilesAndDiagMultipleRuns' into develop
Conflicts:
Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py
Changeset: a511884a7739300755f08f847e090bc9a32b66c6
comment:28 Changed 6 years ago by Alex Buts
Re #11096 Merge branch 'master' into 11096/SumFilesAndDiagMultipleRuns
Conflicts:
Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py Code/Mantid/scripts/test/DirectEnergyConversionTest.py Code/Mantid/scripts/test/DirectPropertyManagerTest.py Code/Mantid/scripts/test/MariReduction.py Code/Mantid/scripts/test/RunDescriptorTest.py
Changeset: 3865d944ca6324946174fbb99d04f3d3d7f1af7a
comment:29 Changed 6 years ago by Alex Buts
Re #11096 Merge branch 'master' into 11096/SumFilesAndDiagMultipleRuns
and fixing bugs introduced by merge
Conflicts:
Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py Code/Mantid/scripts/Inelastic/Direct/PropertiesDescriptors.py Code/Mantid/scripts/Inelastic/Direct/PropertyManager.py Code/Mantid/scripts/Inelastic/Direct/ReductionWrapper.py Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py Code/Mantid/scripts/test/DirectEnergyConversionTest.py Code/Mantid/scripts/test/MariReduction.py Code/Mantid/scripts/test/RunDescriptorTest.py
Changeset: b2d505a974a9482e6c05b8d197634643bafba6fa
comment:30 Changed 6 years ago by Alex Buts
Re #11096 Fixing bugs caused by merge to master
(with pylint changes)
Changeset: 01f053d6f97c5d1e9e97a5cfcf41258fb3db6450
comment:31 Changed 6 years ago by Alex Buts
Re #11096 Merge branch '11096/SumFilesAndDiagMultipleRuns' into dev
Conflicts:
Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py Code/Mantid/scripts/Inelastic/Direct/PropertiesDescriptors.py Code/Mantid/scripts/Inelastic/Direct/PropertyManager.py Code/Mantid/scripts/Inelastic/Direct/ReductionHelpers.py Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py
Changeset: 21c6276c74ad7700d3bc2c6cfa442037a8b7f9d3
comment:32 Changed 6 years ago by Dan Nixon
- Status changed from verify to verifying
- Tester set to Dan Nixon
comment:33 Changed 6 years ago by Alex Buts
Re #11096 Adding file accidentally deleted earlier
Changeset: 9a7ebcd38abbb0b5b1fded4082b77ea1519d7de3
comment:34 Changed 6 years ago by Alex Buts
Re #11096 restoring deleted file mode
Changeset: 04dab9ddd1b7d72b1067b7c2fa6309103658fa32
comment:35 Changed 6 years ago by Dan Nixon
- Status changed from verifying to closed
Merge remote-tracking branch 'origin/11096/SumFilesAndDiagMultipleRuns'
Full changeset: caf71182f0ccabd2a5cb0f442b6329c04081d8d3
comment:36 Changed 6 years ago by Dan Nixon
Merge remote-tracking branch 'origin/11096/SumFilesAndDiagMultipleRuns'
Full changeset: 9bef2e6b842b0be8a6012b777c963b466a16fbb4
comment:37 Changed 6 years ago by Mathieu Doucet
Re #11096 Major refactoring in progress
Changeset: 0e378c6ffd677f912a6f8a86edd6be0d74575019
comment:38 Changed 6 years ago by Mathieu Doucet
Re #11096 First version without wks_utility
Changeset: 4266609f989916c5e32dc4a9fa0b26dc5d32ecc5
comment:39 Changed 6 years ago by Mathieu Doucet
Re #11096 First version without wks_utility
Changeset: 8a8eb8066c5726ade3cb5534da0a6a90d1f1ff6b
comment:40 Changed 6 years ago by Mathieu Doucet
Re #11096 First cut of refactored algorithm
Changeset: f3eed89fc23aad9f607b4eddbd98f64d799ff67a
comment:41 Changed 6 years ago by Mathieu Doucet
Re #11096 Backward compatibility issue
Changeset: 51b5d33da534d4645402134cdafbfddbb62a6472
comment:42 Changed 6 years ago by Mathieu Doucet
Re #11096 Add data file MD5
Changeset: 7c84372341ac6c9537a3cae0af0bc2c3b993de4a
comment:43 Changed 6 years ago by Mathieu Doucet
Re #11096 Add ref data MD5
Changeset: a8a5288bb19960ade0d03e8840eeb0f6ee3f9e2f
comment:44 Changed 6 years ago by Mathieu Doucet
Re #11096 Add test data MD5
Changeset: 294494e971330edb34344b47dd23fc6dc27d7e79
comment:45 Changed 6 years ago by Mathieu Doucet
Re #11096 Add test data MD5
Changeset: 98e92d0db4b287e9b2cdeaa06634c969b01735eb
comment:46 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 11935