Ticket #7415 (closed: fixed)

Opened 7 years ago

Last modified 5 years ago

Speed up loading of VIS_1200_event.nxs

Reported by: Peter Peterson Owned by: Peter Peterson
Priority: major Milestone: Release 2.6
Component: Framework Keywords:
Cc: campbellsi@… Blocked By:
Blocking: Tester: Vickie Lynch

Description

LoadEventNexus of /SNS/VIS/IPTS-6966/0/1200/NeXus/VIS_1200_event.nxs is very slow.

Change History

comment:1 Changed 7 years ago by Peter Peterson

  • Status changed from new to accepted

comment:2 Changed 7 years ago by Peter Peterson

Two comments on this already:

  • On my machine this only takes ~2 minutes. This is not really enough to warrant a ton of work optimizing. (workspace is 19GB)
  • The bottle-neck isn't seeing what banks are empty. That is fast (i.e. doesn't show up in the profiler)
  • The actual bottle-neck is creating events and adding them to the event lists. To make loading faster this has to be split up. This would improve performance for all files that have many events in a (relatively) small number of banks.

comment:3 Changed 7 years ago by Nick Draper

  • Status changed from accepted to assigned

comment:4 Changed 7 years ago by Nick Draper

  • Status changed from assigned to new

comment:5 Changed 7 years ago by Peter Peterson

Re #7415. Added debug print for how long a worker thread runs.

This is for processing banks to help determine what is going on.

Changeset: 0d95ca6049b1a880c338787925ec066e3ed9b19f

comment:6 Changed 7 years ago by Peter Peterson

Re #7415. Moved many bare pointers to shared_arrays.

Changeset: b21bbe19ba17b36d38990371e26ce511188594a4

comment:7 Changed 7 years ago by Peter Peterson

Re #7415. Fixed up precount option so it actually makes things faster.

This includes passing the pixel ranges down into the ProcessBankData.

Changeset: a186eb7c47159aa5156fe1cd24450e973cea5e77

comment:8 Changed 7 years ago by Peter Peterson

Re #7415. Preparing for the final change to have two procs per bank.

Changeset: 1a2e2c65beae88ac9c894e217fbf8303377ee093

comment:9 Changed 7 years ago by Peter Peterson

Re #7415. Launching two jobs to process each bank.

Changeset: 2c5bb0eb0272601b2556e7d22ca8e99af26e610c

comment:10 Changed 7 years ago by Peter Peterson

Re #7415. Check before launching two processing threads.

The performance gets better for the case when the number of banks is smaller than the number of cores (i.e. MANDI). Also added a better check for the number of events in a bank. This prevents launching jobs on empty banks.

Changeset: 36540217efca01bd7a04eb40651e6e87455d2a69

comment:11 Changed 7 years ago by Peter Peterson

Re #7415. Fixed name of hidden LoadEventPreNexus call.

Changeset: a893170c0a46f087cdaf252c8e66e115c30e04fa

comment:12 Changed 7 years ago by Peter Peterson

Re #7415. Corrected progress bar reporting.

Changeset: 53e111d0bd081aaa43dcea34535f50ccd860d56f

comment:13 Changed 7 years ago by Peter Peterson

  • Status changed from new to verify
  • Resolution set to fixed

This is on feature/7415_vision_performance.

To test: try lots of different things that require loading event nexus files. At worst the performance should be unchanged as there is a check for not using the extra thread.

comment:14 Changed 7 years ago by Vickie Lynch

  • Status changed from verify to verifying
  • Tester set to Vickie Lynch

comment:15 Changed 7 years ago by Peter Peterson

  • Status changed from verifying to reopened
  • Resolution fixed deleted

This broke some builds servers so I'm reopening it.

comment:16 Changed 7 years ago by Peter Peterson

Re #7415. Don't print out the timer info on windoze.

Changeset: 5068b75cd3813a1e7ebedf27ca3fb2bdef9c70e1

comment:17 Changed 7 years ago by Vickie Lynch

{{ seconds comparing loading before/after changes: ARCS_12924 25.49 to 13.14 MANDI_306 109.05 to 31.91 NOM_8330 2.87 to 2.78 PG3_13428 79.12 to 35.56 SNAP_9526 5.62 to 9.51 TOPAZ_7424 30.81 to 26.13 VIS_311 100.77 to 63.83 VULCAN_20002 2.04 to 1.98 }}}

comment:18 Changed 7 years ago by Peter Peterson

Re #7415. Fixing pulse id assignment issue.

Changeset: 2f55080a175a8710ac0c1218937bdc7e3997397f

comment:19 Changed 7 years ago by Peter Peterson

Re #7415. Inverting not-and into and or.

Changeset: 679e17b57ab8279c2fa223f943ad9ea0f00c5ac7

comment:20 Changed 7 years ago by Peter Peterson

Merge branch 'feature/7415_vision_performance' into develop

Re #7415

Conflicts:

Code/Mantid/Framework/DataHandling/src/LoadEventNexus.cpp

Changeset: 263db7a04ccbe84d2084f3eb02a0141db89b9af7

comment:21 Changed 7 years ago by Peter Peterson

Revert "Revert "Merge branch 'feature/7415_vision_performance' into develop""

This reverts commit f8d4cb464a18c17b03a7f84e5d2b6ebd6e2a827f.

Re #7415

Conflicts:

Code/Mantid/Framework/DataHandling/src/LoadEventNexus.cpp

Changeset: ba369c2765ff402b42aab820235ac158a5351cef

comment:22 Changed 7 years ago by Peter Peterson

  • Status changed from reopened to verify
  • Resolution set to fixed

Same information for branch and testing as before in comment:13

comment:23 Changed 7 years ago by Vickie Lynch

  • Status changed from verify to verifying

comment:24 Changed 7 years ago by Vickie Lynch

Reduction with chunking works for PG3 data

seconds comparing loading before/after changes: 
ARCS_12924 25.49 to 17.81
MANDI_306 109.05 to 36.61 
NOM_8330 2.87 to 3.02
PG3_13428 79.12 to 41.03 
SNAP_9526 5.62 to 9.29 
SNAP_8653 353.53 to 64.89
TOPAZ_7424 30.81 to 26.80
VIS_311 100.77 to 50.93
VULCAN_20002 2.04 to 1.84

comment:25 Changed 7 years ago by Vickie Lynch

  • Status changed from verifying to closed

comment:26 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 8261

Note: See TracTickets for help on using tickets.