File storage for MD event workspaces

[Nick Draper]

Work with Alex Buts to create a file based storage backend for the MDEventWorkspace.

Performance and efficiency is key here, particulary for calculating 1D, 2D and 3D cuts.

Also consider the need to have a saved file format for the entire workspace in designing the file format.

A potential design is in the document ​http://svn.mantidproject.org/mantid/trunk/Documents/Design/Nexus/MantidImermediateDataFormat.doc

Although this may need further work, or complete scrapping

[Janik Zikovsky]

(In [13057]) Refs #3316: Nexus saving/loading of IMDBox.

[Janik Zikovsky]

(In [13058]) Refs #3316: Fix a namespace confusion. NEXUS!!!

[Janik Zikovsky]

(In [13059]) Refs #3316: Some kind of write permission problem on windows?

[Janik Zikovsky]

(In [13060]) Refs #3316: Another windows try

[Janik Zikovsky]

(In [13061]) Refs #3316: MacOS try to fix. NEXUS (again)!!!

[Janik Zikovsky]

(In [13062]) Refs #3316: More attempts for windows.

[Janik Zikovsky]

(In [13063]) Refs #3316: More attempts for windows.

[Janik Zikovsky]

(In [13064]) Refs #3316: Attempted mac fix.

[Janik Zikovsky]

(In [13066]) Refs #3316: Debugging on windows is hard.

[Janik Zikovsky]

(In [13070]) Refs #3316: Disabling test for windows for now.

[Janik Zikovsky]

(In [13071]) Refs #3316: Must close the file before deleting it

[Alex Buts]

In [13086]:

refs #2383 minor working modifications to test
refs #3316 committed against this thicket to be updated on progress on it.

[Janik Zikovsky]

In [13089]:

Refs #3316: MDBox saving to Nexus

[Janik Zikovsky]

In [13092]:

Refs #3316: Loading MDBox from NXS.

[Janik Zikovsky]

In [13093]:

Refs #3316: Saving and loading of a MDGridBox to NXS.

[Janik Zikovsky]

In [13094]:

Refs #3316: MacOS build fix

[Janik Zikovsky]

In [13096]:

Refs #3316: Try to get windows to build even though it really should be.

[Janik Zikovsky]

In [13098]:

Refs #3316: Stub for SaveMDEW.

[Janik Zikovsky]

In [13121]:

Refs #3316: Alternative way of saving MDEW to nxs.

[Janik Zikovsky]

In [13128]:

Refs #3316: Loading MDEW started. Extra methods in Nexus C++ API.

[Mathieu Doucet]

WARNING alert: LoadMDEW.cpp adds a warning to the pile...

[Mathieu Doucet]

In [13133]:

Fixing warnings. Re #3316

[Mathieu Doucet]

In [13134]:

Fixing warning in LoadMountainDew. Re #3316

[Mathieu Doucet]

In [13135]:

Fixing warnings. Re #3316

[Mathieu Doucet]

In [13136]:

Fixing warnings. Re #3316

[Janik Zikovsky]

In [13150]:

Refs #3316: Saving event data as a single data block looks to be about 3x faster.

[Janik Zikovsky]

In [13156]:

Refs #3316: MountainDew loader with test.

[Janik Zikovsky]

In [13157]:

Refs #3316: Cleaned up a couple of warnings, better test.

[Janik Zikovsky]

In [13158]:

Refs #3316: More details in performance test. In a trial on my machine, a 8GB file with 200 million events was saved in 118 seconds, and loaded back in 38 seconds.

[Janik Zikovsky]

From above tests:

• Write speed = 68 MB/sec
• Read speed = 210 MB/sec.

To compare with the limits of HD performance on my systems:

Write speed:
 dd if=/dev/zero of=./testwrite bs=10k count=100k
 1048576000 bytes (1.0 GB) copied, 14.6511 s, 71.6 MB/s

Read speed:
 sudo hdparm -t /dev/sda
 Timing buffered disk reads: 992 MB in  3.01 seconds = 330.11 MB/sec

So it seems that the writing speed is very close to being hardware limited (good), whereas the hardware limit for reading would be ~24 seconds for a 8GB file; not too far off.

[Janik Zikovsky]

In [13164]:

Fixes #3349: XML box controller. Fix test on some platforms. Refs #3316: Loading/saving box controller to NXS.

[Janik Zikovsky]

In [13165]:

Refs #3316: Fix build for windows.

[Janik Zikovsky]

In [13167]:

Refs #3316: Made MDBox'es not track signal while adding. Improved LoadMDEW and its test.

[Janik Zikovsky]

In [13168]:

Refs #3316: Make sure boxes have sequential IDs for saving. Fix Mac build.

[Russell Taylor]

In [13169]:

MDBoxIterator needs to be excluded from unity build. Re #3316.

[Janik Zikovsky]

In [13170]:

Refs #3316: Better test for LoadMDEW.

[Janik Zikovsky]

In [13171]:

Refs #3316: Alternative way to lock threads to see if it works on Windows.

[Janik Zikovsky]

In [13187]:

Refs #3314: Calls to split the boxes in the MDEventWorkspace made by loading SQW. Made the MDHistoDimension builder trim spaces from dimensions names. Also refs #3316: Speed up in saving to .NXS.

[Janik Zikovsky]

In [13194]:

Refs #3316: Even more thorough test in LoadMDEW. Put algo back in CMakeLists.

[Janik Zikovsky]

In [13225]:

Refs #3316: Minor NXS file changes.

[Janik Zikovsky]

In [13458]:

Refs #3316: IMDBox constructor with extents allows better initialisation of the boxes when loading from disk.

[Janik Zikovsky]

In [13459]:

Refs #3316: More thorough test.

[Janik Zikovsky]

In [13461]:

Refs #3316: The parallel test might be unstable/not working on Windows, so disabling it for now. Additional mutex against simultaneous file access.

[Janik Zikovsky]

In [13483]:

Refs #3316 and refs #3420: Made the MDEW .nxs file format closer to the Design Document specification. Lots of bits (sample/run/instrument) are still missing. Adjusted the PV plugins to find the right kind of loader and fail (gracefully) if given a processedNexus or a SNS event Nexus.

Refs #3271: Sped up vtkThresholdingQuadFactory and vtkThresholdingHexahedronFactory furthere.
Time to create voxels sped up by 40x (19 seconds to 0.45 seconds for 100^{3 voxels).
Time to check the signal for thresholding of 500}3 voxels dropped from 19 seconds to 3.4 seconds.
Made vtkThresholdingUnstructuredGridFactory subclass vtkThresholdingHexahedronFactory so that they could share their common code.

[Janik Zikovsky]

In [13848]:

Refs #2265: Fix to NXS writeExtendibleData(). Refs #3532: Due to bad chunk size when making extendible data, this option is temporarily disabled, because it makes the files super slow. Am waiting on help from the NXS supercommittee. Refs #3316: Changed a field in the data file to conform to nexus_underscore_lowercase style.

[Janik Zikovsky]

In [13905]:

Refs #3316, #3392: Moved some of the nexus fields around to separate the box structure from the event data.

[Janik Zikovsky]

See #3687 for continuation

[Owen Arnold]

Works! I've been using this feature for some time.

[Stuart Campbell]

This ticket has been transferred to github issue ​4163