Speed up loading of VIS_1200_event.nxs

[Peter Peterson]

LoadEventNexus of /SNS/VIS/IPTS-6966/0/1200/NeXus/VIS_1200_event.nxs is very slow.

[Peter Peterson]

Two comments on this already:

• On my machine this only takes ~2 minutes. This is not really enough to warrant a ton of work optimizing. (workspace is 19GB)
• The bottle-neck isn't seeing what banks are empty. That is fast (i.e. doesn't show up in the profiler)
• The actual bottle-neck is creating events and adding them to the event lists. To make loading faster this has to be split up. This would improve performance for all files that have many events in a (relatively) small number of banks.

[Peter Peterson]

Re #7415. Added debug print for how long a worker thread runs.

This is for processing banks to help determine what is going on.

Changeset: 0d95ca6049b1a880c338787925ec066e3ed9b19f

[Peter Peterson]

Re #7415. Moved many bare pointers to shared_arrays.

Changeset: b21bbe19ba17b36d38990371e26ce511188594a4

[Peter Peterson]

Re #7415. Fixed up precount option so it actually makes things faster.

This includes passing the pixel ranges down into the ProcessBankData.

Changeset: a186eb7c47159aa5156fe1cd24450e973cea5e77

[Peter Peterson]

Re #7415. Preparing for the final change to have two procs per bank.

Changeset: 1a2e2c65beae88ac9c894e217fbf8303377ee093

[Peter Peterson]

Re #7415. Launching two jobs to process each bank.

Changeset: 2c5bb0eb0272601b2556e7d22ca8e99af26e610c

[Peter Peterson]

Re #7415. Check before launching two processing threads.

The performance gets better for the case when the number of banks is smaller than the number of cores (i.e. MANDI). Also added a better check for the number of events in a bank. This prevents launching jobs on empty banks.

Changeset: 36540217efca01bd7a04eb40651e6e87455d2a69

[Peter Peterson]

Re #7415. Fixed name of hidden LoadEventPreNexus call.

Changeset: a893170c0a46f087cdaf252c8e66e115c30e04fa

[Peter Peterson]

Re #7415. Corrected progress bar reporting.

Changeset: 53e111d0bd081aaa43dcea34535f50ccd860d56f

[Peter Peterson]

This is on ​feature/7415_vision_performance.

To test: try lots of different things that require loading event nexus files. At worst the performance should be unchanged as there is a check for not using the extra thread.

[Peter Peterson]

This broke some builds servers so I'm reopening it.

[Peter Peterson]

Re #7415. Don't print out the timer info on windoze.

Changeset: 5068b75cd3813a1e7ebedf27ca3fb2bdef9c70e1

[Vickie Lynch]

{{ seconds comparing loading before/after changes: ARCS_12924 25.49 to 13.14 MANDI_306 109.05 to 31.91 NOM_8330 2.87 to 2.78 PG3_13428 79.12 to 35.56 SNAP_9526 5.62 to 9.51 TOPAZ_7424 30.81 to 26.13 VIS_311 100.77 to 63.83 VULCAN_20002 2.04 to 1.98 }}}

[Peter Peterson]

Re #7415. Fixing pulse id assignment issue.

Changeset: 2f55080a175a8710ac0c1218937bdc7e3997397f

[Peter Peterson]

Re #7415. Inverting not-and into and or.

Changeset: 679e17b57ab8279c2fa223f943ad9ea0f00c5ac7

[Peter Peterson]

Merge branch 'feature/7415_vision_performance' into develop

Re #7415

Conflicts:

Code/Mantid/Framework/DataHandling/src/LoadEventNexus.cpp

Changeset: 263db7a04ccbe84d2084f3eb02a0141db89b9af7

[Peter Peterson]

Revert "Revert "Merge branch 'feature/7415_vision_performance' into develop""

This reverts commit f8d4cb464a18c17b03a7f84e5d2b6ebd6e2a827f.

Re #7415

Conflicts:

Code/Mantid/Framework/DataHandling/src/LoadEventNexus.cpp

Changeset: ba369c2765ff402b42aab820235ac158a5351cef

[Peter Peterson]

Same information for branch and testing as before in comment:13

[Vickie Lynch]

Reduction with chunking works for PG3 data

seconds comparing loading before/after changes: 
ARCS_12924 25.49 to 17.81
MANDI_306 109.05 to 36.61 
NOM_8330 2.87 to 3.02
PG3_13428 79.12 to 41.03 
SNAP_9526 5.62 to 9.29 
SNAP_8653 353.53 to 64.89
TOPAZ_7424 30.81 to 26.80
VIS_311 100.77 to 50.93
VULCAN_20002 2.04 to 1.84

[Stuart Campbell]

This ticket has been transferred to github issue ​8261