Ticket #4520 (closed: fixed)

Opened 9 years ago

Last modified 5 years ago

Servere slowdown when loading network files through Convert to Energy interface

Reported by: Peter Parker Owned by: Peter Parker
Priority: blocker Milestone: Release 2.0
Component: Mantid Keywords:
Cc: Blocked By:
Blocking: Tester: Robert Whitley

Description

Timmy is experiencing the same slow down problems as he was towards the end of last year. This is a blocker since he is unable to load any files from his main network drive without waiting a (very) long time for it to load.

Change History

comment:1 Changed 9 years ago by Peter Parker

This has been narrowed down to the slowness of searching for and loading the sister log files to any of the tosca runs on the networked drive. For some runs files, the raw file has some "archive checksum information" stored within it - Mantid looks for this within the LoadLog algo, and uses it to load the relevant files. For the many files that dont have it however, it has to glob the rest of the (considerably large) directory to find the logs.

comment:2 Changed 9 years ago by Peter Parker

Email from Timmy:

The TOSCA files have been generated over 10 years, starting with VMS and later under different flavours of Windows, I am not surprised that they have funny behaviour. They are obviously not quite consistent either, that's why you have some with log files and some without The log files are completely irrelevant to the data analysis in TOSCA so do not load them

In the best case, not loading the log files appears to shorten loading times from 20+ minutes, to less than a second.

We will implement this by:

  • Adding an optional "load log files for this instrument?" property in the tosca parameters file. Other instruments could use the same property in their own files, but it would not be required.
  • Ensuring that this option is passed to load in the Convert to Energy interface.

comment:3 Changed 9 years ago by Peter Parker

It turns out that the Convert to Energy interface is already set to load files with LoadLogFiles set to false. This exact problem only occurs for the case where you load a file via other methods, with it set to true.

The recent problem with the interface has to do with a change in the order that the file finder deals with extensions. It used to search for .raw files before many others, but now there are some extensions searched for before .raw, that have "*" wild cards. Globbing in such a fashion across the network, with very large files, is what is causing the slow down.

comment:4 Changed 9 years ago by Peter Parker

Refs #4520 - Reordering of .ext vector before findRun is called.

Change made to FileFinder, to ensure that non-wildcard extensions are searched for, before extensions like .N*, etc. This has resulted in a VERY large improvement in responsiveness of the Convert to Interface tab when loading files from a large directory across a network.

Also made a (perhaps temporary) change to WorkspaceGroup to deal with a case where m_wsNames.end() was being called in .contains(), and a fatal error occuring as a result. I have yet to find why such an error could be thrown in the first place.

Changeset: 23de600d74636572d8755a64aa5294ace24a2d0b

comment:5 Changed 9 years ago by Peter Parker

To test:

  1. You will need to be at ISIS, with access to the
    ndw615\DATA network folder.
  2. Run Mantid, and open up the Convert to Energy tab.
  3. Click the "Manage User Directories" button, and make sure the folder is added.
  4. Go back to the interface, and enter 12963 into the Run Files field. Click "Run".
  5. It wont be instantaneous, but all files should load without a problem.

Repeat this process again, but this time change the user directory to a *mounted* version of the drive (and give it a name like "N:/"). I supposed this is probably best tested in Windows where you can use both methods, but this should also be tested on Mac as well.

Last edited 9 years ago by Peter Parker (previous) (diff)

comment:6 Changed 9 years ago by Peter Parker

  • Status changed from new to accepted

comment:7 Changed 9 years ago by Peter Parker

  • Status changed from accepted to verify
  • Resolution set to fixed

comment:8 Changed 9 years ago by Peter Parker

Refs #4520 - Reordering of .ext vector before findRun is called.

Change made to FileFinder, to ensure that non-wildcard extensions are searched for, before extensions like .N*, etc. This has resulted in a VERY large improvement in responsiveness of the Convert to Interface tab when loading files from a large directory across a network.

Also made a (perhaps temporary) change to WorkspaceGroup to deal with a case where m_wsNames.end() was being called in .contains(), and a fatal error occuring as a result. I have yet to find why such an error could be thrown in the first place.

Changeset: 23de600d74636572d8755a64aa5294ace24a2d0b

comment:9 Changed 9 years ago by Robert Whitley

  • Status changed from verify to verifying
  • Tester set to Robert Whitley

comment:10 Changed 9 years ago by Robert Whitley

  • Status changed from verifying to closed

comment:11 Changed 9 years ago by Robert Whitley

No slowdown seen when loading runs

comment:12 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 5367

Note: See TracTickets for help on using tickets.