Implement multiple file loading/summing for all data loading algorithms

[Nick Draper]

Rob,

This is very interesting, and I hadn't thought about it too much, but I think there is something we can do.

What I want to avoid is having every load routine written to handle this as it would be tedious and repeated code. However I think there is a way of handling this in a base class.

How about if worked out what you wanted from the file property to load several files you would enter: CSP2011, 2012

If you wanted to load two files and not sum them

CSP2011 + 2012

If you wanted to load an sum two files

CSP2011-2021

If you wanted to load eleven files

+CSP2011-2012

If you wanted to sum 11 files

I'm not too sure about the last one and I'm open to suggestions.

I don't think this will get in for the August release, but it is a great idea.

Regards, Nick Draper

---

From: mantid-help-bounces@… mantid-help-bounces@… On Behalf Of robert.dalgliesh@… Sent: 22 July 2010 13:24 To: Mantid Help Cc: Charlton, Timothy (STFC,RAL,ISIS) Subject: [Mantid-help] load raw and load nexus add runs

Hi Nick, Martyn, Russell,

Are there and plans to allow the load data routines to accept more than 1 run number or data file?

We quite regularly add data on the fly by simply adding whole data sets on a spectrum by spectrum basis.

[Nick Draper]

I've added a hopefully final suggestion from me:

Add a list of files:
2010+2012
Add a range of files:
2010-2012
2010-2012:2 (in steps of 2)

Load a list of files:
2010,2012
Load a range of files:
2010:2012
2010:2012:2 (steps of 2)

[Nick Draper]

Bulk move of tickets to iteration 27, if your ticket is essential for Iteration 26 then move it back.

[Nick Draper]

Bulk move of tickets at the end of iteration 27

[Nick Draper]

Bulk move of tickets at the end of iteration 28

[Nick Draper]

Taken from Sofia Antony's list after she left

[Nick Draper]

"New" tickets moved at the code freeze of iteration 29

[Nick Draper]

Bulk move of tickets to iteration 31 at the iteration 30 code freeze

[Peter Parker]

Amendments to fix compiler errors on non-win7 builds. Refs #1419

Changeset: 859e741453003dc9b15c3c4a856d99bf8a3650ec

[Peter Parker]

Further fixes for compiler warnings on non-win7 builds ... Refs #1419

Changeset: 95cc1ca14a1d9d4289e628b5402d9df7c7556ba4

[Peter Parker]

Further fixes for compiler warnings on non-win7 builds ... Refs #1419

Changeset: 2682403ea0a5cf6c536acf2608ac46ebafaba6c9

[Nick Draper]

Moved to iteration 33 at iteration 32 code freeze

[Peter Parker]

Refs #1419 - MultiFileValidator class added, along with unit tests.

Further work to eventually support multiple loading/summing in the Load algorithm.

Changeset: a16cbccf581de9baa4f6a40f571d19f182182d9d

[Peter Parker]

Refs #1419 - Add run parser. Temp remove multifilename parser.

Added a more robust run string parser, with better validation / error messages. Will make implementation of multifilename parser simpler, which is to follow shortly.

Changeset: 68dc2ba3acea5dcd9cb0f32f2d4f99ace688e081

[Peter Parker]

Refs #1419 - Add multi filename parser.

Changeset: 1e29bdba4b273e1a8d81614cd78c540a89de42a3

[Peter Parker]

Refs #1419 - Fix error on templated function, as well as warnings.

Changeset: 30626fd446adc275a8bb582b694051f2c5cd5f99

[Peter Parker]

Refs #1419 - And again ...

Changeset: a61a23f2aa6adb4fb2fb5753b9eb09aef15b3703

[Peter Parker]

Refs #1419 - Once more.

Changeset: 96b518157f37162c33f4c62d8d42a7d136222595

[Peter Parker]

Refs #1419 - Removing commented unit tests.

Changeset: da6ad2559d00cd34c4b5be08e18801b7df7640c6

[Peter Parker]

Refs #1419 - Changes to finally support multi file loading.

MultiFileProperty changed to store a vector of vectors of file names. File names that share a vector are to be added. Old functionality can be produced by calling "flattenFileNames". See source code for more info.

MultiFileProperty was used in one place previously (MDMergeFiles), this has been modified to relect these changes.

Load now implemented in terms of MultiFileProperty, though its interface is unchanged. "Filename" input is now parsed as a multi file, and if that fails old functionality is attempted. Error reports will output the result of both attempts.

Some additional template specialisations / "toString" overloads surrounding the Property class heirachy were necessary to properly support std::vector<std::vector<std::string> >.

Changeset: 6039c89ac2e9cb95fa9b379333fa07a0476ac831

[Peter Parker]

Refs #1419 - Fix error.

Changeset: a5a5a88dbf26210e1de5a1e8a3187bde55a064a5

[Peter Parker]

Refs #1419 - Fix for failed FilePropertyTest.py

Changeset: ddb74efc65b2987fea4879cffd39323bd0388e1a

[Peter Parker]

Refs #1419 - Fix for System test and broken MergeMDFiles.

Changeset: c61f6e01d7b9f52c409540b92d5c5b8492ff867c

[Peter Parker]

Refs #1419 - Fix a further system test.

Added a further specialised version of IPropertymanager::setProperty to explicitly handle std::string.

Changeset: baef962acad52cef0a7643989768caf24315546d

[Peter Parker]

Refs #1419 - Support loading of incorrectly padded runs.

In the case where a user wants to load a file that had an "inccorrect" zero padding according to Facilities.xml, he/she should be allowed to if that file exists!

If the user only specifies a single file, then we should use the string they provided in its untampered form.

Note, files with incorrect zero padding will not work with multi file loading in its present form.

Changeset: 66e7d9ca2e12454cbeaa4916374ba18051b146c7

[Peter Parker]

Refs #1419 - Changes to finally support multi file loading.

MultiFileProperty changed to store a vector of vectors of file names. File names that share a vector are to be added. Old functionality can be produced by calling "flattenFileNames". See source code for more info.

MultiFileProperty was used in one place previously (MDMergeFiles), this has been modified to relect these changes.

Load now implemented in terms of MultiFileProperty, though its interface is unchanged. "Filename" input is now parsed as a multi file, and if that fails old functionality is attempted. Error reports will output the result of both attempts.

Some additional template specialisations / "toString" overloads surrounding the Property class heirachy were necessary to properly support std::vector<std::vector<std::string> >.

Changeset: 6039c89ac2e9cb95fa9b379333fa07a0476ac831

[Peter Parker]

Refs #1419 - Fix error.

Changeset: a5a5a88dbf26210e1de5a1e8a3187bde55a064a5

[Peter Parker]

Refs #1419 - Fix for failed FilePropertyTest.py

Changeset: ddb74efc65b2987fea4879cffd39323bd0388e1a

[Peter Parker]

Refs #1419 - Fix for System test and broken MergeMDFiles.

Changeset: c61f6e01d7b9f52c409540b92d5c5b8492ff867c

[Peter Parker]

Refs #1419 - Fix a further system test.

Added a further specialised version of IPropertymanager::setProperty to explicitly handle std::string.

Changeset: baef962acad52cef0a7643989768caf24315546d

[Peter Parker]

Refs #1419 - Support loading of incorrectly padded runs.

In the case where a user wants to load a file that had an "inccorrect" zero padding according to Facilities.xml, he/she should be allowed to if that file exists!

If the user only specifies a single file, then we should use the string they provided in its untampered form.

Note, files with incorrect zero padding will not work with multi file loading in its present form.

Changeset: 66e7d9ca2e12454cbeaa4916374ba18051b146c7

[Peter Parker]

Refs #1419 - Changing a log message priority from error to debug.

In the case where a user wishes to load a file with a name which is not of the "standard" format (including a valid instrument), our call to the ConfigService getInstrument method should not report an error.

A debug message is sufficient, and there is still an exception thrown which a class user can then interpret in any way he/she sees fit.

Changeset: 8b391d76bb058833bd689716831ae2905dddefe2

[Peter Parker]

Dev on this ticket seems to have died down, so closing.

To test, use the Load algo as per the examples on the Load Wiki page. Testing can be done in Python, but it is worth checking out the Load Dialog too. Files that load into group workspaces (i.e. MUSR files) should be tested as well as those that load into single workspaces.

Furthermore, helpful error messages should originate from the multi-run parser if a user types an invalid string (e.g. "TSC92273-92273-.raw").

[Peter Parker]

Reopening this to have Mantid try the default instrument in the case where none is specified by the user.

[Peter Parker]

Extra properties given to Load (e.g. "LoadLogFiles") are ignored during multi file loading - the parent "controller" instance of Load currently neglects to pass them on to the child instances that it spawns. This needs to be rectified.

(Note that in the case of single file loading, extra properties still work the way they always have.)

[Peter Parker]

Refs #1419 - MultiFileNameParser now uses default instrument.

Where no instrument is specified in the string to be parsed, the default is taken from the ConfigService and used instead.

Changeset: c6a9ae0c238ebcaaaa039aa788d1f359be129b34

[Peter Parker]

Refs #1419 - All properties now passed to spawned Load algs.

In the case of multiple files being passed to Load, all extra (non-base) properties are passed on to the spawned individual instances of Load that handle each file.

Changeset: 25c0394ad95f4bec9810b3133f90a756912dfe04

[Peter Parker]

Refs #1419 - Fix for non-win builds.

Just changed a templated function to no longer be generic. Templates were uneeded anyway.

Changeset: 1219ce6d273ee552a38a01d2c1c78b5c6761258f

[Peter Parker]

Closing again.

Testing as per comment:35.

[Alex Buts]

Requested functionality is very much needed and is there. Things work both from GUI and Python.

There are number of various issues around the algorithm, which highlighted by the thickets:

#5205, #5210, #5212 and #5213

[Stuart Campbell]

This ticket has been transferred to github issue ​2266