Ticket #3848 (closed: fixed)
Paraview performance on large files
Reported by: | Janik Zikovsky | Owned by: | Janik Zikovsky |
---|---|---|---|
Priority: | major | Milestone: | Release 2.0 |
Component: | Mantid | Keywords: | |
Cc: | owen.arnold@… | Blocked By: | |
Blocking: | Tester: | Michael Reuter |
Description
Take some of the SQW files converted to .NXS and test binning performance of them.
Change History
comment:4 Changed 9 years ago by Janik Zikovsky
Owen's email:
Here are my steps: 1) Convert the 29 Gb Iron data file to a MDEW event nexus file . 2) Load resulting *.nxs file into PV. Takes 623 seconds (reporting given by LoadMD code) 3) Perform axis-aligned rebin in PV using 50 * 50 * 50 * 50 bins. Takes 1365.5 seconds! For comparison using Alex's code. 1) Rename the 29 Gb Iron File to *.sqw_old (so that it is recognised by the older loading/rebinning code) 2) Load SQW file into PV. Works practically instantaneously. 3) Perform axis-aligned rebin in PV using 50 * 50 * 50 * 50 bins. Takes 310.1 seconds!
comment:5 Changed 9 years ago by Janik Zikovsky
After the latest commits:
I ran a binning operation 50x50x50x50 on fe_E1400_8K (the originally 60 GB SQW file) and it took 625 seconds on my system. This is a debug build so I will build in release and see if there is a difference.
comment:6 Changed 9 years ago by Janik Zikovsky
Same test, run in release mode: 440 seconds. Using Thread-Time profiling with the Zoom profiler reports that 89% of the run time is spent in IO.
comment:7 Changed 9 years ago by Owen Arnold
Tested again. 29 Gb file now Loads in 8.35 seconds and rebins in 381 seconds.
comment:9 Changed 9 years ago by Janik Zikovsky
- Status changed from accepted to verify
- Resolution set to fixed
Binning operation to 51x51x51x51 on fe_E1400_8K:
- Alex's code: 611 seconds.
- My code: 676 seconds.
This is consistent with Owen's results (381 vs 311 sec) which means that my algorithm is 10-20% slower than Alex's for file-backed workspaces. I will take this to be sufficiently close to not cause too much trouble. There may be yet be a way to squeeze extra performance and narrow the gap.
Note to testers: Maybe check out a version from revision 14892 and compare with the current performance?
comment:10 Changed 9 years ago by Michael Reuter
- Status changed from verify to verifying
- Tester set to Michael Reuter
comment:11 Changed 9 years ago by Michael Reuter
- Status changed from verifying to closed
This ticket can quite be checked for the sqw_old file since that reader has been removed. All of the other timings look good.
comment:12 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 4695