Paraview performance on large files

[Janik Zikovsky]

Take some of the SQW files converted to .NXS and test binning performance of them.

[Janik Zikovsky]

In [14893]:

Refs #3848: Binning performance of reduced4.sqw data set improved from 32 sec to 24 sec by pre-sorting the MDBoxes by file position to reduce seek time.

[Janik Zikovsky]

In [14895]:

Refs #3848: Better progress reporting/stopping of algo

[Janik Zikovsky]

Owen's email:

Here are my steps:

1)      Convert the 29 Gb Iron data file to a MDEW event nexus file .

2)      Load resulting *.nxs file into PV. Takes 623 seconds (reporting given by LoadMD code)

3)      Perform axis-aligned rebin in PV using 50 * 50 * 50 * 50 bins. Takes 1365.5 seconds!

For comparison using Alex's code.

1)      Rename the 29 Gb Iron File to *.sqw_old (so that it is recognised by the older loading/rebinning code)

2)      Load SQW file into PV. Works practically instantaneously.

3)      Perform axis-aligned rebin in PV using 50 * 50 * 50 * 50 bins. Takes 310.1 seconds!

[Janik Zikovsky]

After the latest commits:

I ran a binning operation 50x50x50x50 on fe_E1400_8K (the originally 60 GB SQW file) and it took 625 seconds on my system. This is a debug build so I will build in release and see if there is a difference.

[Janik Zikovsky]

Same test, run in release mode: 440 seconds. Using Thread-Time profiling with the Zoom profiler reports that 89% of the run time is spent in IO.

[Owen Arnold]

Tested again. 29 Gb file now Loads in 8.35 seconds and rebins in 381 seconds.

[Janik Zikovsky]

In [14898]:

Refs #3848: Make rebinning MDEW parallel by default, but ignore that for file-backed workspaces.

[Janik Zikovsky]

Binning operation to 51x51x51x51 on fe_E1400_8K:

• Alex's code: 611 seconds.
• My code: 676 seconds.

This is consistent with Owen's results (381 vs 311 sec) which means that my algorithm is 10-20% slower than Alex's for file-backed workspaces. I will take this to be sufficiently close to not cause too much trouble. There may be yet be a way to squeeze extra performance and narrow the gap.

Note to testers: Maybe check out a version from revision 14892 and compare with the current performance?

[Michael Reuter]

This ticket can quite be checked for the sqw_old file since that reader has been removed. All of the other timings look good.

[Stuart Campbell]

This ticket has been transferred to github issue ​4695