Load Algorithms, improve selection speed, particularly for Nexus files

[Nick Draper]

If we pass around an open file pointer to the FileCheck methods nexus loaders (and others) can use the open pointer to perform their tests, rather than each open the file themselves. This should be much faster.

Source email from James Lord:

If I have a script/algorithm to process a batch of runs and I write it in a generic 
way using Load(), it takes about twice as long as using the correct specific load 
routine such as LoadMuonNexus(). I suspect Load() is first opening the file to 
inspect the format then closing it and passing the filename onto the format-specific 
version, which opens it again and ignores any cached data.

I've checked with files from the local disk and those from a network drive 
(//hifi/data) and while the local files load faster there is still the same 
improvement to be had by switching to LoadMuonNexus().

In this case I am (intentionally) loading all spectra, and the logs, from muon 
Nexus files of about 750kbytes each (on disk).

Is there any improvement to be made such as caching the file or being able to 
"hint" to Load() that the next file in a sequence is almost certainly going to be 
the same file format as the last one it just loaded?

James Lord.
(Mantid 2.5.3 at present, Windows 7)

[Peter Peterson]

Just formatting the quote so it is a bit easier to read.

[Peter Peterson]

An additional bit of customization for the nexus readers:

1. We have several "nexus" readers
2. Nexus files have a hierarchical structure
3. For nexus based loaders we could pass around a bit of the hierarchy (as some sort of object?) for each one to do the deeper check against rather than the filehandle for each of them to build part of the hierarchy against
4. Most nexus readers should be looking at the value of /entry/definition (and its attributes) as their first check

[Martyn Gigg]

Avoid duplicate Algorithm registration in API tests. Refs #7263

Changeset: 1ef3b9eac35deff16ec10b9663dd727eda2d2028

[Martyn Gigg]

FileLoaderRegistry now subscribes algorithms to factory

and also defines two types of loader, NonHDF and HDF. Refs #7263

Changeset: 1571071e92df153fffebbf6e385cee40332ba18a

[Martyn Gigg]

Add HDFDescriptor class that will describe HDF files.

The checks for the headers at the start of HDF4 & HDF5 files have been put in a static isHDF member to encapsulate the details about checking for a HDF file and make it reusable. It will not retain an open file handle to the file so does not inherit from the standard FileDescriptor. Refs #7263

Changeset: 8220d8c039e66db09b6eb912fb067638eaab4eb0

[Martyn Gigg]

Add pathExists & classTypeExists checks to HDFDDescriptor. Refs #7263

Changeset: f28e98634808556c3a9f8349b8c8af86a36d00d8

[Martyn Gigg]

Implement loader searching in FileLoaderRegistry.

The FileLoaderRegistry can now be asked to search its list of loaders for the best that matches a given file. It passes along a *Descriptor object to minimize the number of times the file is opened.

Refs #7263

Changeset: b3a0dd67688dd1cfbc77f3f48e338195e012a689

[Martyn Gigg]

Add ability to restrict path search in HDF file by type.

Refs #7263

Changeset: 50a5d7fb76c0ef5de4032c8c131319a2e47db53c

[Martyn Gigg]

Add root attribute query methods to HDFDescriptor. Refs #7263

Changeset: 49fcfb3df704c1bff0f50f7fe064bb7644cf65b9

[Martyn Gigg]

First lot of updates for new loader check style. Refs #7263

Changeset: f6b6a6ce3bf2184e13c6da4dc9590fc697345e50

[Martyn Gigg]

Refs #7263 Formatting changes only

Changeset: e97ed18738e429bb81d79607c8356146380b7543

[Martyn Gigg]

LoadILL changes. Refs #7263

Changeset: 79e76cb75788bcad9601485df20500795fdef3ea

[Martyn Gigg]

Convert more loaders to new structure. Refs #7263

Changeset: ff828f6f5452c5b66e338597c9483779d9fb2676

[Martyn Gigg]

More NeXus loaders converted. Refs #7263

Changeset: e61a0109313f6d000870dab7d139d4b44134791e

[Martyn Gigg]

Add basic code for FileLoaderRegistry. Refs #7263

Currently based on string names.

Changeset: 886d0b00f831d546c4a148ced4039a0d711ccf5c

[Martyn Gigg]

Add DECLARE_FILELOADER_ALGORITHM macro. Refs #7263

Changeset: b846d131744cfed0a7c35ea29b8eeb82323247ba

[Martyn Gigg]

Add a simple FileDescriptor class that wraps a file stream

The stream is open for as long as the object is alive. Refs #7263

FileDescriptor uses streams rather than c file handles. Refs #7263

Changeset: 8681dffc29bc6a0c75b9fb931466d0857d76c23b

[Martyn Gigg]

Allow move_class.py to move untracked files. Refs #7263

Changeset: 12dfaa006ada5f6def6b4fe1b73f658741bf5185

[Martyn Gigg]

Fix AlgorithmFactory::unsubscribe method to updates version map.

It was previously not updating the version map when the algorithm was removed and then subscribing the same thing again would throw an error. Refs #7263

Changeset: f6992fc0057c1d0c27020a953126df5282ef6e7b

[Martyn Gigg]

Have AlgorithmFactory::subscribe return the class name subscribed.

Refs #7263

Changeset: 7cff7af939060c3d81fd2b47b1db92c29e5a69fc

[Martyn Gigg]

Avoid duplicate Algorithm registration in API tests. Refs #7263

Changeset: 2ba04b3ffffbd3ac9d4aef91165f4a325f314a4e

[Martyn Gigg]

FileLoaderRegistry now subscribes algorithms to factory

and also defines two types of loader, NonHDF and HDF. Refs #7263

Changeset: 79e6051d0c199175aca57ff8790994d1e49ceee0

[Martyn Gigg]

Add HDFDescriptor class that will describe HDF files.

The checks for the headers at the start of HDF4 & HDF5 files have been put in a static isHDF member to encapsulate the details about checking for a HDF file and make it reusable. It will not retain an open file handle to the file so does not inherit from the standard FileDescriptor. Refs #7263

Changeset: 6568320654211aa04f89504190f6450efa5d4db1

[Martyn Gigg]

Add pathExists & classTypeExists checks to HDFDDescriptor. Refs #7263

Changeset: 429bce6457a2d585ed1968271810e6e8f1a1469c

[Martyn Gigg]

Implement loader searching in FileLoaderRegistry.

The FileLoaderRegistry can now be asked to search its list of loaders for the best that matches a given file. It passes along a *Descriptor object to minimize the number of times the file is opened.

Refs #7263

Changeset: 9b885511a5870d50b22e1eea10176edd20fe40f2

[Martyn Gigg]

Add ability to restrict path search in HDF file by type.

Refs #7263

Changeset: eb79a4e433be3e3873635e2ed1e479068cb6b86a

[Martyn Gigg]

Add root attribute query methods to HDFDescriptor. Refs #7263

Changeset: 4293cc33b80d553a09f04a4c5e5d0bf077cf8fef

[Martyn Gigg]

First lot of updates for new loader check style. Refs #7263

Changeset: b4ba3236f57a206cd1b31c515a098ffdd555e47d

[Martyn Gigg]

Refs #7263 Formatting changes only

Changeset: d6aabf8a2dc2a120d639feadce6d8089910cf8a6

[Martyn Gigg]

LoadILL changes. Refs #7263

Changeset: bc0d6b3e8b8d052e48c6ed3acdd7b0a5604ffd83

[Martyn Gigg]

Convert more loaders to new structure. Refs #7263

Changeset: 4722d5bf1a76540bbc18180ee621d535cf427bd7

[Martyn Gigg]

More NeXus loaders converted. Refs #7263

Changeset: 29d535d4c94d86696fd7770a89115dd2e1289b90

[Martyn Gigg]

Cache first entry name & type in HDFDescriptor. Refs #7263

Changeset: e8d298baff3ff52270e8a02e3b75fc8f6af5be47

[Martyn Gigg]

Final NeXus loader converted. Refs #7263

Changeset: 04e325763073e37ed282d0501d29e298157d6621

[Martyn Gigg]

Add static isAscii checks to FileDescriptor class. Refs #7263

Changeset: c0b04b5be6d9c40d395f8b7d396dab6f3ffa6bd7

[Martyn Gigg]

Start on general file loader conversion. Refs #7263

Changeset: 3f102e2d868d4f17f2f409f275bfe578539a0945

[Martyn Gigg]

More loaders converted. Refs #7263

Changeset: 71706740104269b15e2ac7021086b1e961fda55f

[Martyn Gigg]

More loaders migrated. Refs #7263

Changeset: 7a1bd4e3e72dd409345dcd8c1dcbe863573ba59e

[Martyn Gigg]

Add basic code for FileLoaderRegistry. Refs #7263

Currently based on string names.

Changeset: c5ab6b955dfc544c1a43f00d526ebbb7829d1211

[Martyn Gigg]

Add DECLARE_FILELOADER_ALGORITHM macro. Refs #7263

Changeset: 9dc140841fdd8d29ca1d6ba2bdd949c7c41ee1af

[Martyn Gigg]

Add a simple FileDescriptor class that wraps a file stream

The stream is open for as long as the object is alive. Refs #7263

FileDescriptor uses streams rather than c file handles. Refs #7263

Changeset: 02509f87a8b33fd7973951c54abb5e7555685ec9

[Martyn Gigg]

Allow move_class.py to move untracked files. Refs #7263

Changeset: df6f2db124b0e3f6712adc42ea3f517ee6d02c2a

[Martyn Gigg]

Fix AlgorithmFactory::unsubscribe method to updates version map.

It was previously not updating the version map when the algorithm was removed and then subscribing the same thing again would throw an error. Refs #7263

Changeset: a23d639411aedc0dd1e726d369fcacbba28b73c4

[Martyn Gigg]

Have AlgorithmFactory::subscribe return the class name subscribed.

Refs #7263

Changeset: 5fb5a0ba8fe71656ee4a4468bdea23acd9e85348

[Martyn Gigg]

Avoid duplicate Algorithm registration in API tests. Refs #7263

Changeset: 688d09f690685015f61289beb0a14f4bfc766480

[Martyn Gigg]

FileLoaderRegistry now subscribes algorithms to factory

and also defines two types of loader, NonHDF and HDF. Refs #7263

Changeset: e77a0a5738a1914befe30741d366495a4953f4c3

[Martyn Gigg]

Add HDFDescriptor class that will describe HDF files.

The checks for the headers at the start of HDF4 & HDF5 files have been put in a static isHDF member to encapsulate the details about checking for a HDF file and make it reusable. It will not retain an open file handle to the file so does not inherit from the standard FileDescriptor. Refs #7263

Changeset: b33429b36beee34fcaf3d191ceb735522ce3e5ec

[Martyn Gigg]

Add pathExists & classTypeExists checks to HDFDDescriptor. Refs #7263

Changeset: 90537dd6136b16a531bd2e454e85bf32d086b5d8

[Martyn Gigg]

Implement loader searching in FileLoaderRegistry.

The FileLoaderRegistry can now be asked to search its list of loaders for the best that matches a given file. It passes along a *Descriptor object to minimize the number of times the file is opened.

Refs #7263

Changeset: ee1d2ba84a8748c70ae32afb7f980b9d849a29c9

[Martyn Gigg]

Add ability to restrict path search in HDF file by type.

Refs #7263

Changeset: 0908502cb75249ec3a7431db2e11e4f3e2a44ff5

[Martyn Gigg]

Add root attribute query methods to HDFDescriptor. Refs #7263

Changeset: 4743494db4debe9cf778e4a1610e67c89d76b89f

[Martyn Gigg]

Cache first entry name & type in HDFDescriptor. Refs #7263

Changeset: 3b88b845022c5653d8cfa6ce34cf4fa229d43d3c

[Martyn Gigg]

Refs #7263 Formatting changes only

Changeset: 574a70fe79a4720f96e73a0ef406e4679a57041b

[Martyn Gigg]

First lot of updates for new loader check style. Refs #7263

Changeset: 11988b9a591149fde1dd15e28d0e4b0968a5f8a8

[Martyn Gigg]

Add static isAscii checks to FileDescriptor class. Refs #7263

Changeset: d94c7d27c1d50d3549081a8c39608445dfa15beb

[Martyn Gigg]

General file loader conversion. Refs #7263

Changeset: 5fa890fbdf81568763df6c5d16d16341ba9a6ed7

[Martyn Gigg]

Cache isAscii flag in FileDescriptor. Refs #7263

Changeset: 9a2601fcdf1a94350f05be91554ff82d9faa11c3

[Martyn Gigg]

Port final DataHandling loaders to new structure. Refs #7263

Changeset: b3236b5c18c398bf322c23de241ba9e5384c5cdf

[Martyn Gigg]

Move FileRegistry to singleton Refs #7263

It appears that we can't start the FrameworkManager during static initialization or the destruction of the FrameworkManager instance gets instance gets confused.

Changeset: db8d95c8cdaf648b2439c416724bb59ac2617d65

[Martyn Gigg]

Move the remaining loaders to the new scheme. Refs #7263

Changeset: 2a330f57d781248e4b8ec0502726d1821c5004c5

[Martyn Gigg]

Remove file access from some NeXus checks. Refs #7263

Changeset: 146d72e7c593ab85fed6009cb6493a76633a0a40

[Martyn Gigg]

Fix bug with running correct version of loader. Refs #7263

Changeset: e1dcfe7d4145ea6c9a1d7d51435afffd3cc27c3d

[Martyn Gigg]

Flip path/type map in HDFDescriptor. Refs #7263

Looking up paths is far more common so make that faster.

Changeset: 4c4155e8e58c12d3dcac4430bc95d233fd442025

[Martyn Gigg]

FileDescriptor::resetStreamToStart needs to reset ios flags too.

Refs #7263

Changeset: c9f87457614765bdf53272037f567d0eae486c13

[Martyn Gigg]

Add base-class checks to FILELOADER* check macros. Refs #7263

Also added runtime checks to the subscribe method. This allows it to static cast at runtime when checking rather than a more expensive dynamic cast.

Changeset: 34785e5194cf9bc29e5435b4973f541ba76ed4df

[Martyn Gigg]

Fix LoadMD::confidence path checks. Refs #7263

Changeset: 961b02983a7752038c2532068499f794a8cbc60c

[Martyn Gigg]

Fix EventPreNexus & RKH file checks. Refs #7263

Changeset: 7b1bfbe989a08c9dadc96056e2e8e9cfdd48fcee

[Martyn Gigg]

Add access to open NeXus file in HDFDescriptor. Refs #7263

Changeset: 19119d59871acd545067afbca480bce35ee22941

[Martyn Gigg]

Allow stored NeXus handle to be use in ::confidence checks. Refs #7263

Changeset: d60fc3449766ac8f8a29363ca8099e65199ab7c2

[Martyn Gigg]

Ignore new LoaderVersion property in Python Load. Refs #7263

Changeset: 62314ef77d2c164f7d87892da7a333e40628a12d

[Martyn Gigg]

Add basic code for FileLoaderRegistry. Refs #7263

Currently based on string names.

Changeset: c5ab6b955dfc544c1a43f00d526ebbb7829d1211

[Martyn Gigg]

Add DECLARE_FILELOADER_ALGORITHM macro. Refs #7263

Changeset: 9dc140841fdd8d29ca1d6ba2bdd949c7c41ee1af

[Martyn Gigg]

Add a simple FileDescriptor class that wraps a file stream

The stream is open for as long as the object is alive. Refs #7263

FileDescriptor uses streams rather than c file handles. Refs #7263

Changeset: 02509f87a8b33fd7973951c54abb5e7555685ec9

[Martyn Gigg]

Allow move_class.py to move untracked files. Refs #7263

Changeset: df6f2db124b0e3f6712adc42ea3f517ee6d02c2a

[Martyn Gigg]

Fix AlgorithmFactory::unsubscribe method to updates version map.

It was previously not updating the version map when the algorithm was removed and then subscribing the same thing again would throw an error. Refs #7263

Changeset: a23d639411aedc0dd1e726d369fcacbba28b73c4

[Martyn Gigg]

Have AlgorithmFactory::subscribe return the class name subscribed.

Refs #7263

Changeset: 5fb5a0ba8fe71656ee4a4468bdea23acd9e85348

[Martyn Gigg]

Avoid duplicate Algorithm registration in API tests. Refs #7263

Changeset: 688d09f690685015f61289beb0a14f4bfc766480

[Martyn Gigg]

FileLoaderRegistry now subscribes algorithms to factory

and also defines two types of loader, NonHDF and HDF. Refs #7263

Changeset: e77a0a5738a1914befe30741d366495a4953f4c3

[Martyn Gigg]

Add HDFDescriptor class that will describe HDF files.

The checks for the headers at the start of HDF4 & HDF5 files have been put in a static isHDF member to encapsulate the details about checking for a HDF file and make it reusable. It will not retain an open file handle to the file so does not inherit from the standard FileDescriptor. Refs #7263

Changeset: b33429b36beee34fcaf3d191ceb735522ce3e5ec

[Martyn Gigg]

Add pathExists & classTypeExists checks to HDFDDescriptor. Refs #7263

Changeset: 90537dd6136b16a531bd2e454e85bf32d086b5d8

[Martyn Gigg]

Implement loader searching in FileLoaderRegistry.

The FileLoaderRegistry can now be asked to search its list of loaders for the best that matches a given file. It passes along a *Descriptor object to minimize the number of times the file is opened.

Refs #7263

Changeset: ee1d2ba84a8748c70ae32afb7f980b9d849a29c9

[Martyn Gigg]

Add ability to restrict path search in HDF file by type.

Refs #7263

Changeset: 0908502cb75249ec3a7431db2e11e4f3e2a44ff5

[Martyn Gigg]

Add root attribute query methods to HDFDescriptor. Refs #7263

Changeset: 4743494db4debe9cf778e4a1610e67c89d76b89f

[Martyn Gigg]

Cache first entry name & type in HDFDescriptor. Refs #7263

Changeset: 3b88b845022c5653d8cfa6ce34cf4fa229d43d3c

[Martyn Gigg]

Refs #7263 Formatting changes only

Changeset: 574a70fe79a4720f96e73a0ef406e4679a57041b

[Martyn Gigg]

First lot of updates for new loader check style. Refs #7263

Changeset: 11988b9a591149fde1dd15e28d0e4b0968a5f8a8

[Martyn Gigg]

Add static isAscii checks to FileDescriptor class. Refs #7263

Changeset: d94c7d27c1d50d3549081a8c39608445dfa15beb

[Martyn Gigg]

General file loader conversion. Refs #7263

Changeset: 5fa890fbdf81568763df6c5d16d16341ba9a6ed7

[Martyn Gigg]

Cache isAscii flag in FileDescriptor. Refs #7263

Changeset: 9a2601fcdf1a94350f05be91554ff82d9faa11c3

[Martyn Gigg]

Port final DataHandling loaders to new structure. Refs #7263

Changeset: b3236b5c18c398bf322c23de241ba9e5384c5cdf

[Martyn Gigg]

Move FileRegistry to singleton Refs #7263

It appears that we can't start the FrameworkManager during static initialization or the destruction of the FrameworkManager instance gets instance gets confused.

Changeset: db8d95c8cdaf648b2439c416724bb59ac2617d65

[Martyn Gigg]

Move the remaining loaders to the new scheme. Refs #7263

Changeset: 2a330f57d781248e4b8ec0502726d1821c5004c5

[Martyn Gigg]

Remove file access from some NeXus checks. Refs #7263

Changeset: 146d72e7c593ab85fed6009cb6493a76633a0a40

[Martyn Gigg]

Fix bug with running correct version of loader. Refs #7263

Changeset: e1dcfe7d4145ea6c9a1d7d51435afffd3cc27c3d

[Martyn Gigg]

Flip path/type map in HDFDescriptor. Refs #7263

Looking up paths is far more common so make that faster.

Changeset: 4c4155e8e58c12d3dcac4430bc95d233fd442025

[Martyn Gigg]

FileDescriptor::resetStreamToStart needs to reset ios flags too.

Refs #7263

Changeset: c9f87457614765bdf53272037f567d0eae486c13

[Martyn Gigg]

Add base-class checks to FILELOADER* check macros. Refs #7263

Also added runtime checks to the subscribe method. This allows it to static cast at runtime when checking rather than a more expensive dynamic cast.

Changeset: 34785e5194cf9bc29e5435b4973f541ba76ed4df

[Martyn Gigg]

Fix LoadMD::confidence path checks. Refs #7263

Changeset: 961b02983a7752038c2532068499f794a8cbc60c

[Martyn Gigg]

Fix EventPreNexus & RKH file checks. Refs #7263

Changeset: 7b1bfbe989a08c9dadc96056e2e8e9cfdd48fcee

[Martyn Gigg]

Add access to open NeXus file in HDFDescriptor. Refs #7263

Changeset: 19119d59871acd545067afbca480bce35ee22941

[Martyn Gigg]

Allow stored NeXus handle to be use in ::confidence checks. Refs #7263

Changeset: 7aad82fbecd50e5a2b784f68e954849571f06468

[Martyn Gigg]

Ignore new LoaderVersion property in Python Load. Refs #7263

Changeset: 775e5249f4dead9814565de7a26edd4afff5b6d2

[Martyn Gigg]

Merge branch 'feature/7263_improve_load_selection_performance'

into develop Refs #7263

Conflicts:

Code/Mantid/Framework/DataHandling/inc/MantidDataHandling/LoadILL.h Code/Mantid/Framework/DataHandling/src/LoadEventNexus.cpp Code/Mantid/Framework/DataHandling/src/LoadILL.cpp Code/Mantid/Framework/DataHandling/src/LoadSassena.cpp

Changeset: 49a10d7e33c34eaa40e0cde4ceaebd0b23909473

[Martyn Gigg]

Remove valgrind header left accidentally. Refs #7263

Changeset: 82d931398d792e4d9a3e58ebdead81393687b85e

[Martyn Gigg]

Fix ifstream initialization in FileDescriptor. Refs #7263

Changeset: 5ff698b5c973e64809b9a1b886215abbacc92a18

[Martyn Gigg]

Remove compile-time checks on FileLoader classes.

They don't seem to work everywhere. My guess is that the types isn't quite complete by the time we use it so some compiles don't like it. Refs #7263

Changeset: 4d64a81a01043122ca1f1ada303ec6a1570d5b4c

[Martyn Gigg]

Avoid double-algorithm registration in tests. Refs #7263

Changeset: 6e678e5ca625f1f0b25695d4f1d334ce0fa1f920

[Martyn Gigg]

Add basic code for FileLoaderRegistry. Refs #7263

Currently based on string names.

Changeset: c5ab6b955dfc544c1a43f00d526ebbb7829d1211

[Martyn Gigg]

Add DECLARE_FILELOADER_ALGORITHM macro. Refs #7263

Changeset: 9dc140841fdd8d29ca1d6ba2bdd949c7c41ee1af

[Martyn Gigg]

Add a simple FileDescriptor class that wraps a file stream

The stream is open for as long as the object is alive. Refs #7263

FileDescriptor uses streams rather than c file handles. Refs #7263

Changeset: 02509f87a8b33fd7973951c54abb5e7555685ec9

[Martyn Gigg]

Allow move_class.py to move untracked files. Refs #7263

Changeset: df6f2db124b0e3f6712adc42ea3f517ee6d02c2a

[Martyn Gigg]

Fix AlgorithmFactory::unsubscribe method to updates version map.

It was previously not updating the version map when the algorithm was removed and then subscribing the same thing again would throw an error. Refs #7263

Changeset: a23d639411aedc0dd1e726d369fcacbba28b73c4

[Martyn Gigg]

Have AlgorithmFactory::subscribe return the class name subscribed.

Refs #7263

Changeset: 483acce7607ca2aeb6bae584523fd487a80754d5

[Martyn Gigg]

Avoid duplicate Algorithm registration in API tests. Refs #7263

Changeset: 885088de599e1d0f00bcd7cca06ef888a84fa802

[Martyn Gigg]

FileLoaderRegistry now subscribes algorithms to factory

and also defines two types of loader, NonHDF and HDF. Refs #7263

Changeset: 52b5f1a5a5a1884401f76143e09290e785c49142

[Martyn Gigg]

Add HDFDescriptor class that will describe HDF files.

The checks for the headers at the start of HDF4 & HDF5 files have been put in a static isHDF member to encapsulate the details about checking for a HDF file and make it reusable. It will not retain an open file handle to the file so does not inherit from the standard FileDescriptor. Refs #7263

Changeset: da30646f22af90d25e18a0be3b2bb56cef973896

[Martyn Gigg]

Add pathExists & classTypeExists checks to HDFDDescriptor. Refs #7263

Changeset: 367b6232759944ca13e334dc859acf813fd687bc

[Martyn Gigg]

Implement loader searching in FileLoaderRegistry.

The FileLoaderRegistry can now be asked to search its list of loaders for the best that matches a given file. It passes along a *Descriptor object to minimize the number of times the file is opened.

Refs #7263

Changeset: 132675199cf21badd7d4952a78fe6b42c48c3bfb

[Martyn Gigg]

Add ability to restrict path search in HDF file by type.

Refs #7263

Changeset: 9a591118e449763fba5aefd5c9a84edac6d53e58

[Martyn Gigg]

Add root attribute query methods to HDFDescriptor. Refs #7263

Changeset: ec3e85aaf4496543aca35c186d190c2eba46e111

[Martyn Gigg]

Cache first entry name & type in HDFDescriptor. Refs #7263

Changeset: 5d723229c6be4088067f51a6c250eeadad0721da

[Martyn Gigg]

Refs #7263 Formatting changes only

Changeset: ff1f7ad4c9d894da2e96b50180809cca41273fb6

[Martyn Gigg]

First lot of updates for new loader check style. Refs #7263

Changeset: 5eec9296ab81d729b10ed22c87be140367bc9941

[Martyn Gigg]

Add static isAscii checks to FileDescriptor class. Refs #7263

Changeset: bbc2514e5a1fe7b238dd26fe5fb8baf70183a877

[Martyn Gigg]

General file loader conversion. Refs #7263

Changeset: 94047da498146c0db71a9a671ace0dacdb106b28

[Martyn Gigg]

Cache isAscii flag in FileDescriptor. Refs #7263

Changeset: 248c7047a90f0972a4abc02d308e04a8d353d85f

[Martyn Gigg]

Port final DataHandling loaders to new structure. Refs #7263

Changeset: 076350e97bd4ff2c49f7dd2ab31b3b10620b2371

[Martyn Gigg]

Move FileRegistry to singleton Refs #7263

It appears that we can't start the FrameworkManager during static initialization or the destruction of the FrameworkManager instance gets instance gets confused.

Changeset: f4dbdb03111d909f57b1094dbbdffdc2f4fd564a

[Martyn Gigg]

Move the remaining loaders to the new scheme. Refs #7263

Changeset: b008a1348f9862f22722d7d42bb3edb6c61c60be

[Martyn Gigg]

Remove file access from some NeXus checks. Refs #7263

Changeset: 9d03dfb0ad782579c16eca779ca9bf2ecae8e9bf

[Martyn Gigg]

Fix bug with running correct version of loader. Refs #7263

Changeset: 5fd18f632d4bef0eda7dbd97e8732389f38fcabf

[Martyn Gigg]

Flip path/type map in HDFDescriptor. Refs #7263

Looking up paths is far more common so make that faster.

Changeset: 14a7ab1ce7ae4b05b4f56e7596bd289591d6e679

[Martyn Gigg]

FileDescriptor::resetStreamToStart needs to reset ios flags too.

Also fix initialization that is not a pointer. Refs #7263

Changeset: 4ff024a68acbb8083a6a2a69da44fce94e9ea4d8

[Martyn Gigg]

Add base-class checks to FILELOADER* check macros. Refs #7263

Also added runtime checks to the subscribe method. This allows it to static cast at runtime when checking rather than a more expensive dynamic cast.

Changeset: e3e18db9a3750df686e41badd27fb7ec0dfcc789

[Martyn Gigg]

Fix LoadMD::confidence path checks. Refs #7263

Changeset: 7d124b9dee7bed2b6a75677b542e56d9566fe9e7

[Martyn Gigg]

Fix EventPreNexus & RKH file checks. Refs #7263

Changeset: 0e0bd35179723d2b608093b619d3eb55ea9d4a04

[Martyn Gigg]

Add access to open NeXus file in HDFDescriptor. Refs #7263

Changeset: 5a1a53c2c09e4b07f288cf35a32850492dd1e785

[Martyn Gigg]

Allow stored NeXus handle to be use in ::confidence checks. Refs #7263

Changeset: 96f42591d31ec8fddf76896e657631104b1e6622

[Martyn Gigg]

Ignore new LoaderVersion property in Python Load. Refs #7263

Changeset: aa26bb649bb8173d4b88af5c03ffa2356afc73ae

[Martyn Gigg]

Remove compile-time checks on FileLoader classes.

They don't seem to work everywhere. My guess is that the types isn't quite complete by the time we use it so some compiles don't like it. Refs #7263

Changeset: 84aa5968b2caacb2ac575ce7f3ff24cea2939b4b

[Martyn Gigg]

Fix LoadSNSspec test.

A \r left in the file after opening in binary mode confused the check. Refs #7263

Changeset: 6292160c0d040a146cd65649579a37b0f05c585a

[Martyn Gigg]

Add canLoad method to FileLoaderRegistry.

This better encapsulates the file checking so that other places don't have to worry about the desciptor stuff. Refs #7263

Changeset: a5c9c3096f25cdbbeedd23f65c320ea6f5083b76

[Martyn Gigg]

Remove IDataFileChecker & LoadAlgorithmFactory. Refs #7263

Changeset: 39bfe42ca9f987cdcd0036799c070346c6a6a26e

[Martyn Gigg]

Merge branch 'feature/7263_improve_load_selection_performance' into develop

Refs #7263 Conflicts:

Code/Mantid/Framework/DataHandling/inc/MantidDataHandling/LoadILL.h Code/Mantid/Framework/DataHandling/src/LoadEventNexus.cpp Code/Mantid/Framework/DataHandling/src/LoadILL.cpp Code/Mantid/Framework/DataHandling/src/LoadSassena.cpp

Changeset: 134bd768072544227c66568dd50764973035c92b

[Martyn Gigg]

Temp move LoadILLSANS to new loader hierachy on develop. Refs #7263

Changeset: 014f1fec3f379936607f4e72578e29b08e9c7c36

[Martyn Gigg]

Fixes for OSX 10.8 & Intel compiler.

It doesn't generate a symbol for the destructor of the *FileLoader interfaces unless explicitly defined. All other platforms, including OS X 10.6, don't show this problem. Refs #7263

Changeset: 04e169774094e6307a6f24e20cc31f0c979242f8

[Martyn Gigg]

Merge branch 'feature/7263_improve_load_selection_performance' into develop

Refs #7263 Conflicts:

Code/Mantid/Framework/API/inc/MantidAPI/IFileLoader.h Code/Mantid/Framework/API/inc/MantidAPI/IHDFFileLoader.h Code/Mantid/Framework/DataHandling/inc/MantidDataHandling/LoadILL.h Code/Mantid/Framework/DataHandling/src/LoadILL.cpp Code/Mantid/Framework/DataHandling/src/LoadSassena.cpp

Changeset: 10264aba28915ee6805abeb17bfa3f6aef967161

[Martyn Gigg]

Fix istream construction in FileDescriptorTest.

The standard is to only allow C strings but the other platforms seem to accept std::strings as well Refs #7263

Changeset: 367fe425b49a021bf1d6f71be23bde4429b8db48

[Martyn Gigg]

Branch: feature/7263_improve_load_selection_performance

Tester: There are a lot of changes here. Things got rebased as well so the comments are little bit of a mess. Best to see ​here for the changes.

The whole idea of file checking has now changed and is described here:

http://www.mantidproject.org/Hooking_In_With_Load_Algorithm

so that it is available for developers. The file being checked now only gets opened once and a wrapper object is passed to each of the loaders that gives access to meta-data & the open file stream.

There is also a special object for HDF files that caches other aspects of the file to avoid each loader having to open & parse the file.

The tests, both unit & system) should be passing, see here ​https://builds.sns.gov/job/ornl_test_rhel6_develop/, for most up to date tests results.

[Martyn Gigg]

Attached script has a loop for loading a file. Flip it between Load & LoadMuonNexus to see the difference.

[Owen Arnold]

Before changes. Script loads in 3.93 seconds. After changes, script loads in 2.33 seconds. Both running on a Debug build on the same Mac. By way of reference, I've also run the script using the LoadMuonNexus algorithm directly, and find that it takes 2.02 seconds. So the cost of finding the right loader is now quite small in comparison to what it was previously.

[Anders Markvardsen]

Merge remote-tracking branch 'origin/feature/7263_improve_load_selection_performance'

Refs #7263 Conflicts:

Code/Mantid/Framework/DataHandling/src/LoadEventNexus.cpp Code/Mantid/Framework/DataHandling/src/LoadSassena.cpp

Changeset: 28afe6d1afe3b8329978e5d8b0413f3938e08629

[Stuart Campbell]

This ticket has been transferred to github issue ​8109