LoadSQW: optimize for very large files

[Janik Zikovsky]

Need to be able to load the large SQW files from toby into a MDEventWorkspace (presumably a file-backed one).

[Owen Arnold]

20Gb and 40Gb Fe files contain visible spin-waves when rebinned. We should also ensure that the MDEW rebinning algorithm can work effectively on this volume of data. Janik may want to split this into a separate ticket.

[Nick Draper]

Bulk move of tickets to iteration 31 at the iteration 30 code freeze

[Janik Zikovsky]

In [14851]:

Refs #3586: LoadSQW works a lot better for SQW files that are large but still fit in memory. 6 GB SQW file loads in about 2 minutes.

[Janik Zikovsky]

In [14855]:

Refs #3586: Version of LoadSQW that can load directly to a file-backed MDworkspace

[Janik Zikovsky]

In [14856]:

Refs #3586: Refresh cache

[Janik Zikovsky]

In [14860]:

Refs #3586: Parallelized part of the SQW loading, tweak to make loading to a file back-end significantly faster (~ twice the speed)

[Janik Zikovsky]

In [14862]:

Refs #3586

[Janik Zikovsky]

My latest test converted a 60 GB SQW file (fe_E1400_8K.sqw) into a 81 GB .nxs file with 1.76 billion events in 66 minutes. This is a good improvement over the last time of ~190 minutes! Output workspace had 11 million boxes. Memory use was around 4 GB during this process.

This performance is in line with (a bit slower) the performance of the MergeMD algorithm which treated 1 billion events in 25 minutes. Keep in mind that you have 140 GB of disk IO which would take ~25 minutes alone, so while the algorithm is not quite IO limited it is within a factor of 2-3x of IO limits.

[Janik Zikovsky]

In [14889]:

Refs #3586: Make Loading large files faster in paraview

[Owen Arnold]

Works as expected. Test run Loaded a 30 GB Iron dataset in 3135 seconds. Acceptable given that SQW is soon to become a redundant format anyway.

[Stuart Campbell]

This ticket has been transferred to github issue ​4433