Ticket #3316 (closed: fixed)

Opened 9 years ago

Last modified 5 years ago

File storage for MD event workspaces

Reported by: Nick Draper Owned by: Janik Zikovsky
Priority: blocker Milestone: Iteration 30
Component: VATES Keywords:
Cc: Blocked By: #3337
Blocking: Tester: Owen Arnold

Description

Work with Alex Buts to create a file based storage backend for the MDEventWorkspace.

Performance and efficiency is key here, particulary for calculating 1D, 2D and 3D cuts.

Also consider the need to have a saved file format for the entire workspace in designing the file format.

A potential design is in the document http://svn.mantidproject.org/mantid/trunk/Documents/Design/Nexus/MantidImermediateDataFormat.doc

Although this may need further work, or complete scrapping

Change History

comment:1 Changed 9 years ago by Janik Zikovsky

  • Status changed from new to accepted

comment:2 Changed 9 years ago by Janik Zikovsky

(In [13057]) Refs #3316: Nexus saving/loading of IMDBox.

comment:3 Changed 9 years ago by Janik Zikovsky

(In [13058]) Refs #3316: Fix a namespace confusion. NEXUS!!!

comment:4 Changed 9 years ago by Janik Zikovsky

(In [13059]) Refs #3316: Some kind of write permission problem on windows?

comment:5 Changed 9 years ago by Janik Zikovsky

(In [13060]) Refs #3316: Another windows try

comment:6 Changed 9 years ago by Janik Zikovsky

(In [13061]) Refs #3316: MacOS try to fix. NEXUS (again)!!!

comment:7 Changed 9 years ago by Janik Zikovsky

(In [13062]) Refs #3316: More attempts for windows.

comment:8 Changed 9 years ago by Janik Zikovsky

(In [13063]) Refs #3316: More attempts for windows.

comment:9 Changed 9 years ago by Janik Zikovsky

(In [13064]) Refs #3316: Attempted mac fix.

comment:10 Changed 9 years ago by Janik Zikovsky

(In [13066]) Refs #3316: Debugging on windows is hard.

comment:11 Changed 9 years ago by Janik Zikovsky

(In [13070]) Refs #3316: Disabling test for windows for now.

comment:12 Changed 9 years ago by Janik Zikovsky

(In [13071]) Refs #3316: Must close the file before deleting it

comment:13 Changed 9 years ago by Alex Buts

In [13086]:

refs #2383 minor working modifications to test
refs #3316 committed against this thicket to be updated on progress on it.

comment:14 Changed 9 years ago by Janik Zikovsky

In [13089]:

Refs #3316: MDBox saving to Nexus

comment:15 Changed 9 years ago by Janik Zikovsky

In [13092]:

Refs #3316: Loading MDBox from NXS.

comment:16 Changed 9 years ago by Janik Zikovsky

In [13093]:

Refs #3316: Saving and loading of a MDGridBox to NXS.

comment:17 Changed 9 years ago by Janik Zikovsky

In [13094]:

Refs #3316: MacOS build fix

comment:18 Changed 9 years ago by Janik Zikovsky

In [13096]:

Refs #3316: Try to get windows to build even though it really should be.

comment:19 Changed 9 years ago by Janik Zikovsky

In [13098]:

Refs #3316: Stub for SaveMDEW.

comment:20 Changed 9 years ago by Janik Zikovsky

  • Blocked By 3337 added

comment:21 Changed 9 years ago by Janik Zikovsky

In [13121]:

Refs #3316: Alternative way of saving MDEW to nxs.

comment:22 Changed 9 years ago by Janik Zikovsky

In [13128]:

Refs #3316: Loading MDEW started. Extra methods in Nexus C++ API.

comment:23 Changed 9 years ago by Mathieu Doucet

WARNING alert: LoadMDEW.cpp adds a warning to the pile...

comment:24 Changed 9 years ago by Mathieu Doucet

In [13133]:

Fixing warnings. Re #3316

comment:25 Changed 9 years ago by Mathieu Doucet

In [13134]:

Fixing warning in LoadMountainDew. Re #3316

comment:26 Changed 9 years ago by Mathieu Doucet

In [13135]:

Fixing warnings. Re #3316

comment:27 Changed 9 years ago by Mathieu Doucet

In [13136]:

Fixing warnings. Re #3316

comment:28 Changed 9 years ago by Janik Zikovsky

In [13150]:

Refs #3316: Saving event data as a single data block looks to be about 3x faster.

comment:29 Changed 9 years ago by Janik Zikovsky

In [13156]:

Refs #3316: MountainDew loader with test.

comment:30 Changed 9 years ago by Janik Zikovsky

In [13157]:

Refs #3316: Cleaned up a couple of warnings, better test.

comment:31 Changed 9 years ago by Janik Zikovsky

In [13158]:

Refs #3316: More details in performance test. In a trial on my machine, a 8GB file with 200 million events was saved in 118 seconds, and loaded back in 38 seconds.

comment:32 Changed 9 years ago by Janik Zikovsky

From above tests:

  • Write speed = 68 MB/sec
  • Read speed = 210 MB/sec.

To compare with the limits of HD performance on my systems:

Write speed:
 dd if=/dev/zero of=./testwrite bs=10k count=100k
 1048576000 bytes (1.0 GB) copied, 14.6511 s, 71.6 MB/s

Read speed:
 sudo hdparm -t /dev/sda
 Timing buffered disk reads: 992 MB in  3.01 seconds = 330.11 MB/sec

So it seems that the writing speed is very close to being hardware limited (good), whereas the hardware limit for reading would be ~24 seconds for a 8GB file; not too far off.

comment:33 Changed 9 years ago by Janik Zikovsky

In [13164]:

Fixes #3349: XML box controller. Fix test on some platforms. Refs #3316: Loading/saving box controller to NXS.

comment:4 Changed 9 years ago by Janik Zikovsky

In [13165]:

Refs #3316: Fix build for windows.

comment:5 Changed 9 years ago by Janik Zikovsky

In [13167]:

Refs #3316: Made MDBox'es not track signal while adding. Improved LoadMDEW and its test.

comment:6 Changed 9 years ago by Janik Zikovsky

In [13168]:

Refs #3316: Make sure boxes have sequential IDs for saving. Fix Mac build.

comment:7 Changed 9 years ago by Russell Taylor

In [13169]:

MDBoxIterator needs to be excluded from unity build. Re #3316.

comment:8 Changed 9 years ago by Janik Zikovsky

In [13170]:

Refs #3316: Better test for LoadMDEW.

comment:9 Changed 9 years ago by Janik Zikovsky

In [13171]:

Refs #3316: Alternative way to lock threads to see if it works on Windows.

comment:10 Changed 9 years ago by Janik Zikovsky

In [13187]:

Refs #3314: Calls to split the boxes in the MDEventWorkspace made by loading SQW. Made the MDHistoDimension builder trim spaces from dimensions names. Also refs #3316: Speed up in saving to .NXS.

comment:11 Changed 9 years ago by Janik Zikovsky

In [13194]:

Refs #3316: Even more thorough test in LoadMDEW. Put algo back in CMakeLists.

comment:12 Changed 9 years ago by Janik Zikovsky

In [13225]:

Refs #3316: Minor NXS file changes.

comment:13 Changed 9 years ago by Janik Zikovsky

In [13458]:

Refs #3316: IMDBox constructor with extents allows better initialisation of the boxes when loading from disk.

comment:14 Changed 9 years ago by Janik Zikovsky

In [13459]:

Refs #3316: More thorough test.

comment:15 Changed 9 years ago by Janik Zikovsky

In [13461]:

Refs #3316: The parallel test might be unstable/not working on Windows, so disabling it for now. Additional mutex against simultaneous file access.

comment:16 Changed 9 years ago by Janik Zikovsky

In [13483]:

Refs #3316 and refs #3420: Made the MDEW .nxs file format closer to the Design Document specification. Lots of bits (sample/run/instrument) are still missing. Adjusted the PV plugins to find the right kind of loader and fail (gracefully) if given a processedNexus or a SNS event Nexus.

Refs #3271: Sped up vtkThresholdingQuadFactory and vtkThresholdingHexahedronFactory furthere.
Time to create voxels sped up by 40x (19 seconds to 0.45 seconds for 1003 voxels).
Time to check the signal for thresholding of 500
3 voxels dropped from 19 seconds to 3.4 seconds.
Made vtkThresholdingUnstructuredGridFactory subclass vtkThresholdingHexahedronFactory so that they could share their common code.

comment:17 Changed 9 years ago by Janik Zikovsky

In [13848]:

Refs #2265: Fix to NXS writeExtendibleData(). Refs #3532: Due to bad chunk size when making extendible data, this option is temporarily disabled, because it makes the files super slow. Am waiting on help from the NXS supercommittee. Refs #3316: Changed a field in the data file to conform to nexus_underscore_lowercase style.

comment:11 Changed 9 years ago by Janik Zikovsky

In [13905]:

Refs #3316, #3392: Moved some of the nexus fields around to separate the box structure from the event data.

comment:12 Changed 9 years ago by Janik Zikovsky

  • Status changed from accepted to verify
  • Resolution set to fixed

See #3687 for continuation

comment:13 Changed 9 years ago by Owen Arnold

  • Status changed from verify to verifying
  • Tester set to Owen Arnold

comment:14 Changed 9 years ago by Owen Arnold

  • Status changed from verifying to closed

Works! I've been using this feature for some time.

comment:15 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 4163

Note: See TracTickets for help on using tickets.