Ticket #5118 (closed: fixed)

Opened 8 years ago

Last modified 5 years ago

MergeMDFiles: read several boxes at once to speed things up

Reported by: Janik Zikovsky Owned by: Janik Zikovsky
Priority: major Milestone: Release 2.1
Component: Mantid Keywords:
Cc: Blocked By:
Blocking: Tester: Michael Reuter

Description

Toby Perring said that in HORACE he would load N contiguous boxes (to the extent of available memory) so as to limit disk access. Might speed things up.

Change History

comment:1 Changed 8 years ago by Janik Zikovsky

  • Status changed from new to accepted

comment:2 Changed 8 years ago by Janik Zikovsky

Refs #5118: MergeMDFiles loads several boxes at once

speed up in my test was 330 sec -> 265 sec for 263 million events

Changeset: a54225469d3cc6d66167e1684745f160058adb19

comment:3 Changed 8 years ago by Janik Zikovsky

Refs #5118: use emplace_back to (maybe) speed up LoadMD.

Only gave a 1% speed up :(

Changeset: baa2f782b779c87802dec5d203beb715b14472e7

comment:4 Changed 8 years ago by Janik Zikovsky

Refs #5118 fix windows build I hope

Changeset: 4daf6e3b93df05da08632e40029a8374be10f833

comment:5 Changed 8 years ago by Janik Zikovsky

Refs #5118 doesn't work on mac either. Reverting

Changeset: ed28d897eefdd5401906d475bce824cc71648865

comment:6 Changed 8 years ago by Janik Zikovsky

Refs #5118: MergeMDFiles loads several boxes at once

speed up in my test was 330 sec -> 265 sec for 263 million events

Changeset: a54225469d3cc6d66167e1684745f160058adb19

comment:7 Changed 8 years ago by Janik Zikovsky

Refs #5118: use emplace_back to (maybe) speed up LoadMD.

Only gave a 1% speed up :(

Changeset: baa2f782b779c87802dec5d203beb715b14472e7

comment:8 Changed 8 years ago by Janik Zikovsky

Refs #5118 fix windows build I hope

Changeset: 4daf6e3b93df05da08632e40029a8374be10f833

comment:9 Changed 8 years ago by Janik Zikovsky

Refs #5118 doesn't work on mac either. Reverting

Changeset: ed28d897eefdd5401906d475bce824cc71648865

comment:10 Changed 8 years ago by Janik Zikovsky

  • Status changed from accepted to verify
  • Resolution set to fixed

comment:11 Changed 8 years ago by Michael Reuter

  • Status changed from verify to verifying
  • Tester set to Michael Reuter

comment:12 Changed 8 years ago by Michael Reuter

  • Status changed from verifying to closed

I can merge 60 SEQ MD event files with 35.2 million events in 28.7 seconds. This yields a rate of about 1.2 million events per second. Awesome!

comment:13 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue ​5964

Note: See TracTickets for help on using tickets.