Servere slowdown when loading network files through Convert to Energy interface

[Peter Parker]

Timmy is experiencing the same slow down problems as he was towards the end of last year. This is a blocker since he is unable to load any files from his main network drive without waiting a (very) long time for it to load.

[Peter Parker]

This has been narrowed down to the slowness of searching for and loading the sister log files to any of the tosca runs on the networked drive. For some runs files, the raw file has some "archive checksum information" stored within it - Mantid looks for this within the LoadLog algo, and uses it to load the relevant files. For the many files that dont have it however, it has to glob the rest of the (considerably large) directory to find the logs.

[Peter Parker]

Email from Timmy:

The TOSCA files have been generated over 10 years, starting with VMS and later under different flavours of Windows, I am not surprised that they have funny behaviour. They are obviously not quite consistent either, that's why you have some with log files and some without The log files are completely irrelevant to the data analysis in TOSCA so do not load them

In the best case, not loading the log files appears to shorten loading times from 20+ minutes, to less than a second.

We will implement this by:

• Adding an optional "load log files for this instrument?" property in the tosca parameters file. Other instruments could use the same property in their own files, but it would not be required.
• Ensuring that this option is passed to load in the Convert to Energy interface.

[Peter Parker]

It turns out that the Convert to Energy interface is already set to load files with LoadLogFiles set to false. This exact problem only occurs for the case where you load a file via other methods, with it set to true.

The recent problem with the interface has to do with a change in the order that the file finder deals with extensions. It used to search for .raw files before many others, but now there are some extensions searched for before .raw, that have "*" wild cards. Globbing in such a fashion across the network, with very large files, is what is causing the slow down.

[Peter Parker]

Refs #4520 - Reordering of .ext vector before findRun is called.

Change made to FileFinder, to ensure that non-wildcard extensions are searched for, before extensions like .N*, etc. This has resulted in a VERY large improvement in responsiveness of the Convert to Interface tab when loading files from a large directory across a network.

Also made a (perhaps temporary) change to WorkspaceGroup to deal with a case where m_wsNames.end() was being called in .contains(), and a fatal error occuring as a result. I have yet to find why such an error could be thrown in the first place.

Changeset: 23de600d74636572d8755a64aa5294ace24a2d0b

[Peter Parker]

To test:

1. You will need to be at ISIS, with access to the
   ndw615\DATA network folder.
2. Run Mantid, and open up the Convert to Energy tab.
3. Click the "Manage User Directories" button, and make sure the folder is added.
4. Go back to the interface, and enter 12963 into the Run Files field. Click "Run".
5. It wont be instantaneous, but all files should load without a problem.

Repeat this process again, but this time change the user directory to a *mounted* version of the drive (and give it a name like "N:/"). I supposed this is probably best tested in Windows where you can use both methods, but this should also be tested on Mac as well.

[Peter Parker]

Refs #4520 - Reordering of .ext vector before findRun is called.

Change made to FileFinder, to ensure that non-wildcard extensions are searched for, before extensions like .N*, etc. This has resulted in a VERY large improvement in responsiveness of the Convert to Interface tab when loading files from a large directory across a network.

Also made a (perhaps temporary) change to WorkspaceGroup to deal with a case where m_wsNames.end() was being called in .contains(), and a fatal error occuring as a result. I have yet to find why such an error could be thrown in the first place.

Changeset: 23de600d74636572d8755a64aa5294ace24a2d0b

[Robert Whitley]

No slowdown seen when loading runs

[Stuart Campbell]

This ticket has been transferred to github issue ​5367