File-based back-end for MDEventWorkspace

[Janik Zikovsky]

Refs #3316. Load lists of MDEvent's from a source .NXS file on demand during processing.

[Janik Zikovsky]

In [13298]:

Refs #3392: First go at a file-backed MDEventWorkspace

[Janik Zikovsky]

In [13299]:

Refs #3392: Fix test.

[Janik Zikovsky]

In [13301]:

Refs #3392: Got rid of old MDBox NXS saving code that was too slow. More tests for the file back-end.

[Janik Zikovsky]

In [13302]:

Refs #3392: Method to release the event list when done with it. Used only in the binning routine so far.

[Janik Zikovsky]

Looks like some commits didn't show up. [13460] Refs #3392: Mutexes in file-back-end of MDBox

[13461] The parallel test might be unstable/not working on Windows, so disabling it for now. Additional mutex against simultaneous file access.

Also, here are some preliminary speed results from accessing a fake data file 8GB in size (200 million events in 3D).

Loading with FileBackEnd 0  took  103.245970964  seconds 

Binning everything. FileBackEnd 0 . IterateEvents 0
Took:  0.904938936234  seconds   and Signal max is  201889.0
Binning a slice, 1/10th. FileBackEnd 0 . IterateEvents 0
Took:  0.314957857132  seconds   and Signal max is  22575.0
Binning everything. FileBackEnd 0 . IterateEvents 1
Took:  16.9913930893  seconds   and Signal max is  201889.0
Binning a slice, 1/10th. FileBackEnd 0 . IterateEvents 1
Took:  2.13666176796  seconds   and Signal max is  22575.0

Loading with FileBackEnd 1  took  4.30377316475  seconds 

Binning everything. FileBackEnd 1 . IterateEvents 0
Took:  122.313542128  seconds   and Signal max is  201889.0
Binning a slice, 1/10th. FileBackEnd 1 . IterateEvents 0
Took:  17.1517679691  seconds   and Signal max is  22575.0
Binning everything. FileBackEnd 1 . IterateEvents 1
Took:  49.948884964  seconds   and Signal max is  201889.0
Binning a slice, 1/10th. FileBackEnd 1 . IterateEvents 1
Took:  5.71084284782  seconds   and Signal max is  22575.0

The 122 second case looks like a case of file mutex deadlocking. Using a MRU list and keeping SOME events in memory will likely help a ton on speed, especially upon the 2nd go.

[Janik Zikovsky]

Next steps:

• Keep a list of Most-Recently-Used boxes in memory.
• User-specified memory in MB when loading (with a reasonable default).
• Keep MDBoxes with very few events in memory. Allocate e.g. 10% of the total memory to this.

[Janik Zikovsky]

In [13732]:

Refs #3392: Fix incorrect depth of boxes when loading MDEW nxs file.

[Alex Buts]

In [13888]:

refs #3392 suggested format for MD workspace

[Janik Zikovsky]

In [13905]:

Refs #3316, #3392: Moved some of the nexus fields around to separate the box structure from the event data.

[Janik Zikovsky]

In [14017]:

Refs #3392: Option to make an in-memory MDEW into a file-backed one in SaveMDEW algo.

[Janik Zikovsky]

In [14022]:

Refs #3392: Some additional performance tests to demonstrate DiskMRU usage.

[Russell Taylor]

In [14042]:

Clear Intel compiler warning. Re #3392.

[Owen Arnold]

Looks like it's doing exactly what it's supposed to. Loading Algorithm exposes option. Selecting file back-end reports:

• Smaller Workspace in-memory than otherwise
• File backed MRU information

[Stuart Campbell]

This ticket has been transferred to github issue ​4239