Ticket #11096 (closed: fixed)

Opened 6 years ago

Last modified 5 years ago

Direct reduction -- Summation with wait for files and diagnostics for list of files

Reported by: Alex Buts Owned by: Alex Buts
Priority: major Milestone: Release 3.3.1
Component: Framework Keywords:
Cc: Blocked By: #11063
Blocking: #11155, #11177 Tester: Dan Nixon

Description (last modified by Alex Buts) (diff)

Unit test for DirectEnergyConversionTest: test_multirep_abs_units_mode will fail if diagnostics is enabled. In addition to that, file summation does not work if run from local Mantid and wait for files is enabled.

The purpose of this ticket is to provide reliable workflow of the autoreduction script for multiple files when run from Mantid or proof that current workflow is correct for any data source (workspace, file or run-number)

(I am more or less confident that it works fine for autoreduction when single run is processed, but the issues above suggest that there are still problems when run locally multiple times with only some files changing)

Change History

comment:1 Changed 6 years ago by Alex Buts

  • Status changed from new to assigned
  • Description modified (diff)

comment:2 Changed 6 years ago by Alex Buts

  • Status changed from assigned to inprogress

Re #11096 Streamlining various set options in RunDescriptor

Paying special attention to the case, when list of files is assigned. (nothing still works properly)

Changeset: 1cc6ada69562815d4ab6942533a13b391cccb3ba

comment:3 Changed 6 years ago by Alex Buts

Re #11096 fixed autoreduce workflow with summation

when refactored to new summation class

Changeset: ff9ab4c04d9f9ebd6547fad852eed2262c799413

comment:4 Changed 6 years ago by Alex Buts

Re #11096 Working sums cash and unit test for it

Changeset: 588439eac81bc70ddba923077a167777a3846886

comment:5 Changed 6 years ago by Alex Buts

Re #11096 Stub for a cached RunDescriptor mask and its applications

Changeset: fdd160dc1cce228c5010c191da1fb9735bfcc58e

comment:6 Changed 6 years ago by Alex Buts

Re #11096 minor bugs and typos

Changeset: f8fa765063866b5352829dc3a52af649aada7e0c

comment:7 Changed 6 years ago by Alex Buts

Re #11096 Minor changes and typo to fix system test

in accordance with changes to main reduction

Changeset: 68b3d6d4027516ffa885fca7589cc03e58d58aaa

comment:8 Changed 6 years ago by Alex Buts

Re #11096 Implemented reduce_sum while waiting for file

all seems works but couple of additional system tests would be useful

Changeset: bd1b8c0d7e2e7e9dce19be9338e72f0b38c6c47a

comment:9 Changed 6 years ago by Alex Buts

Re #11096 should fix system tests (some of them at least)

Changeset: c2f88b09f3118e6ca5f1b3b94d16f39eb5c758b9

comment:10 Changed 6 years ago by Alex Buts

Re #11096 Implemented cached masks

and all seems works though system test would be useful

Changeset: c920db4278a73bcc6fddc3997acbbd78a97d8212

comment:11 Changed 6 years ago by Alex Buts

Re #11096 All cashes work

Changeset: 3aa8ca48edc6d30deb1adef21d345ab329980fdf

comment:12 Changed 6 years ago by Alex Buts

Re #11096 Minor typos and bug fixes

also test and method to provide custom file format procedure

Changeset: 91285bc4d93dd8fe4cdca9d0b5dd98a0413b034d

comment:13 Changed 6 years ago by Alex Buts

Re #11096 One more system test and better clean up

Changeset: d0f7672c1c5522b7fe825d2d876dba9b10e7f7cf

comment:14 Changed 6 years ago by Alex Buts

  • Blocking 11155 added

comment:15 Changed 6 years ago by Alex Buts

Re #11096 fixing conflicts causes by pylint changes

Merge branch 'master' into 11096/SumFilesAndDiagMultipleRuns

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py Code/Mantid/scripts/Inelastic/Direct/PropertiesDescriptors.py Code/Mantid/scripts/Inelastic/Direct/PropertyManager.py Code/Mantid/scripts/Inelastic/Direct/ReductionWrapper.py Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py Code/Mantid/scripts/Inelastic/Direct/diagnostics.py

Changeset: 0d251fe3b250c82ccb8eebd50dc297540c553df2

comment:16 Changed 6 years ago by Alex Buts

Re #11096 Bugs induced by merging master with pylint changes

Changeset: 79584187198a47da04592d996100433aebdb89bb

comment:17 Changed 6 years ago by Alex Buts

Re #11096 Merge branch '11096/SumFilesAndDiagMultipleRuns'into develop

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py Code/Mantid/scripts/Inelastic/Direct/PropertiesDescriptors.py Code/Mantid/scripts/Inelastic/Direct/PropertyManager.py Code/Mantid/scripts/Inelastic/Direct/ReductionHelpers.py Code/Mantid/scripts/Inelastic/Direct/ReductionWrapper.py Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py Code/Mantid/scripts/Inelastic/Direct/diagnostics.py

Changeset: 692a654587406165fee38aae607abd57d346cd35

comment:18 Changed 6 years ago by Alex Buts

Re #11096 fixed bugs in wait_for_file procedure

As the procedure without tests have deteriorated. Also framework for tests to avoid it in a future and tests themselves (in system tests branch)

Changeset: a19885787da56aee8f2e8cc146b87efd146e2bc8

comment:19 Changed 6 years ago by Alex Buts

Re #11096 Tests for reduction with wait for files workflow

Changeset: 9636db41fbd38f014823f712c6528e6c6c9eeab7

comment:20 Changed 6 years ago by Alex Buts

  • Status changed from inprogress to verify
  • Resolution set to fixed
  • Milestone changed from Release 3.4 to Release 3.3.1

Should be tested only after #11063 is merged to master. (spawned from that branch, though diverged due to pylint changes later)

The changes are both in System tests and Mantid branches, so no pull request for that yet.

The essence of the ticket is to improve file summation and running range of files one after another -- the way which is the same regardless if script is run from web service or Mantid script window.

Purpose is achieved, unit tests fixed and number of additional system test are written. The tests confirm the validity of changes. Otherwise -- test by code review

comment:21 Changed 6 years ago by Alex Buts

  • Blocked By 11063 added

comment:22 Changed 6 years ago by Alex Buts

Re #11096 Interface function to provide custom file name procedure

Changeset: c933ff28602de9d57042c97bc8c749f4e361b7dd

comment:23 Changed 6 years ago by Alex Buts

Re #11096 Typo in ReductionWrapper

Changeset: de9777925ac0fcb76808ceb22128700a91ee2e4b

comment:24 Changed 6 years ago by Alex Buts

Re #11096 some conflicts with develop

Merge branch '11096/SumFilesAndDiagMultipleRuns' into develop

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/ReductionWrapper.py

Changeset: b0ff1a9d3e27e340f69619f9f593e4c53c316af5

comment:25 Changed 6 years ago by Alex Buts

Re #11096 minor changes to internal save_results routine

do not split internal file-name to name and extension if . is present in this file name

Changeset: 312d4c84244a9381c11d0910649e004ceb919c95

comment:26 Changed 6 years ago by Alex Buts

Re #11096 conflicts with develop

Merge branch '11096/SumFilesAndDiagMultipleRuns' into develop

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py

Changeset: a511884a7739300755f08f847e090bc9a32b66c6

comment:27 Changed 6 years ago by Alex Buts

  • Blocking 11177 added

comment:28 Changed 6 years ago by Alex Buts

Re #11096 Merge branch 'master' into 11096/SumFilesAndDiagMultipleRuns

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py Code/Mantid/scripts/test/DirectEnergyConversionTest.py Code/Mantid/scripts/test/DirectPropertyManagerTest.py Code/Mantid/scripts/test/MariReduction.py Code/Mantid/scripts/test/RunDescriptorTest.py

Changeset: 3865d944ca6324946174fbb99d04f3d3d7f1af7a

comment:29 Changed 6 years ago by Alex Buts

Re #11096 Merge branch 'master' into 11096/SumFilesAndDiagMultipleRuns

and fixing bugs introduced by merge

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py Code/Mantid/scripts/Inelastic/Direct/PropertiesDescriptors.py Code/Mantid/scripts/Inelastic/Direct/PropertyManager.py Code/Mantid/scripts/Inelastic/Direct/ReductionWrapper.py Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py Code/Mantid/scripts/test/DirectEnergyConversionTest.py Code/Mantid/scripts/test/MariReduction.py Code/Mantid/scripts/test/RunDescriptorTest.py

Changeset: b2d505a974a9482e6c05b8d197634643bafba6fa

comment:30 Changed 6 years ago by Alex Buts

Re #11096 Fixing bugs caused by merge to master

(with pylint changes)

Changeset: 01f053d6f97c5d1e9e97a5cfcf41258fb3db6450

comment:31 Changed 6 years ago by Alex Buts

Re #11096 Merge branch '11096/SumFilesAndDiagMultipleRuns' into dev

Conflicts:

Code/Mantid/scripts/Inelastic/Direct/DirectEnergyConversion.py Code/Mantid/scripts/Inelastic/Direct/PropertiesDescriptors.py Code/Mantid/scripts/Inelastic/Direct/PropertyManager.py Code/Mantid/scripts/Inelastic/Direct/ReductionHelpers.py Code/Mantid/scripts/Inelastic/Direct/RunDescriptor.py

Changeset: 21c6276c74ad7700d3bc2c6cfa442037a8b7f9d3

comment:32 Changed 6 years ago by Dan Nixon

  • Status changed from verify to verifying
  • Tester set to Dan Nixon

comment:33 Changed 6 years ago by Alex Buts

Re #11096 Adding file accidentally deleted earlier

Changeset: 9a7ebcd38abbb0b5b1fded4082b77ea1519d7de3

comment:34 Changed 6 years ago by Alex Buts

Re #11096 restoring deleted file mode

Changeset: 04dab9ddd1b7d72b1067b7c2fa6309103658fa32

comment:35 Changed 6 years ago by Dan Nixon

  • Status changed from verifying to closed

Merge remote-tracking branch 'origin/11096/SumFilesAndDiagMultipleRuns'

Full changeset: caf71182f0ccabd2a5cb0f442b6329c04081d8d3

comment:36 Changed 6 years ago by Dan Nixon

Merge remote-tracking branch 'origin/11096/SumFilesAndDiagMultipleRuns'

Full changeset: 9bef2e6b842b0be8a6012b777c963b466a16fbb4

comment:37 Changed 6 years ago by Mathieu Doucet

Re #11096 Major refactoring in progress

Changeset: 0e378c6ffd677f912a6f8a86edd6be0d74575019

comment:38 Changed 6 years ago by Mathieu Doucet

Re #11096 First version without wks_utility

Changeset: 4266609f989916c5e32dc4a9fa0b26dc5d32ecc5

comment:39 Changed 6 years ago by Mathieu Doucet

Re #11096 First version without wks_utility

Changeset: 8a8eb8066c5726ade3cb5534da0a6a90d1f1ff6b

comment:40 Changed 6 years ago by Mathieu Doucet

Re #11096 First cut of refactored algorithm

Changeset: f3eed89fc23aad9f607b4eddbd98f64d799ff67a

comment:41 Changed 6 years ago by Mathieu Doucet

Re #11096 Backward compatibility issue

Changeset: 51b5d33da534d4645402134cdafbfddbb62a6472

comment:42 Changed 6 years ago by Mathieu Doucet

Re #11096 Add data file MD5

Changeset: 7c84372341ac6c9537a3cae0af0bc2c3b993de4a

comment:43 Changed 6 years ago by Mathieu Doucet

Re #11096 Add ref data MD5

Changeset: a8a5288bb19960ade0d03e8840eeb0f6ee3f9e2f

comment:44 Changed 6 years ago by Mathieu Doucet

Re #11096 Add test data MD5

Changeset: 294494e971330edb34344b47dd23fc6dc27d7e79

comment:45 Changed 6 years ago by Mathieu Doucet

Re #11096 Add test data MD5

Changeset: 98e92d0db4b287e9b2cdeaa06634c969b01735eb

comment:46 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 11935

Note: See TracTickets for help on using tickets.