Ticket #3244 (closed: fixed)

Opened 9 years ago

Last modified 5 years ago

Make LoadEventPreNeXus faster

Reported by: Peter Peterson Owned by: Janik Zikovsky
Priority: major Milestone: Iteration 30
Component: Mantid Keywords:
Cc: Blocked By:
Blocking: Tester: Peter Peterson

Description

This is a continuation of #3224.

Profiling still says that the majority of the time spent is in processing the list rather than reading from disk by at least a factor of 100. The next big target is finding a better way of allocating memory in the individual event lists (via reserve) rather than just using the defaults and letting vector re-allocate memory as necessary.

Change History

comment:1 Changed 9 years ago by Janik Zikovsky

  • Owner set to Janik Zikovsky
  • Status changed from new to accepted

I think the theory on how to do it is to split up the file in N small (but not too small) chunks and load them into separate EventWorkspaces which we will then merge into one big one.

Will probably require moving quite a bit of code around in the algorithm.

comment:2 Changed 9 years ago by Janik Zikovsky

In [13642]:

Refs #3244: LoadEventPreNexus parallelized. Loading NOM_640 took 143 seconds (release mode) before, now takes about 46 seconds (for a 4 GB file).

comment:3 Changed 9 years ago by Janik Zikovsky

In [13655]:

Refs #3244: Cached a few values to try to get the most speed possible in LoadEventPreNexus.

comment:4 Changed 9 years ago by Janik Zikovsky

In [13658]:

Refs #3244: Fix test.

comment:5 Changed 9 years ago by Janik Zikovsky

In [13659]:

Refs #3244: Fix segfault.

comment:6 Changed 9 years ago by Janik Zikovsky

In [13662]:

Refs #3244: Fix segfaults for incorrect pixel IDs.

comment:7 Changed 9 years ago by Mathieu Doucet

In [13670]:

Fix warnings. Re #3244

comment:8 Changed 9 years ago by Janik Zikovsky

  • Status changed from accepted to verify
  • Resolution set to fixed

In [13700]:

Fixes #3244: Property for LoadEventPreNexus to automatically choose between serial/parallel loading for whichever case is fastest, based on the number of events to load.

comment:9 Changed 9 years ago by Peter Peterson

  • Status changed from verify to verifying
  • Tester set to Peter Peterson

comment:10 Changed 9 years ago by Peter Peterson

  • Status changed from verifying to closed

It is faster.

comment:11 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 4091

Note: See TracTickets for help on using tickets.