Ticket #7908 (inprogress)
Parallelize the LoadLotsOfInstruments system test
| Reported by: | Russell Taylor | Owned by: | Martyn Gigg |
|---|---|---|---|
| Priority: | minor | Milestone: | Backlog |
| Component: | Tools | Keywords: | |
| Cc: | Blocked By: | #6261 | |
| Blocking: | Tester: |
Description
It's taking 30 minutes or more to run in places, and should be trivial to parallelize using the multiprocessing module. In general, we ought to prefer parallelizing the Mantid code itself rather than the tests, but the LoadInstrument itself is not and probably can't be parallelized.
Change History
comment:2 Changed 7 years ago by Russell Taylor
Re #7908. Parallelize LoadLotsOfInstruments test.
Uses the multiprocessing module to run LoadEmptyInstrument on the different IDF files in parallel.
Changeset: 3032a3c7e30d80aad447f7a433ee175947f5f372
comment:3 Changed 7 years ago by Russell Taylor
- Blocked By 6261 added
Something deep within OpenCascade concerning the writing of the vtp files is scuppering me here - it's locking up for some reason.
Re-visit this after #6261 is done. In fact, that will likely remove most of the benefit of this ticket anyway as most of the time is in generating these files. Here are the timings on my machine:
- Parallel, not writing vtp files: 21s
- Not parallel, vtp files there: 95s
- Not parallel, writing vtp files: 20 minutes
comment:4 Changed 7 years ago by Russell Taylor
The parallel code works fine if all the vtp files are already there.
comment:5 Changed 7 years ago by Russell Taylor
- Milestone changed from Release 3.0 to Backlog
The blocking ticket has been moved out of the 3.0 milestone so this one has to be as well.
comment:9 Changed 6 years ago by Russell Taylor
Re #7908. Move exception handling into loading function.
Changeset: af9b0367d08f68334b5d1ff500b98c6b4552ee2b
comment:10 Changed 6 years ago by Russell Taylor
Re #7908. Parallelize LoadLotsOfInstruments test.
Uses the multiprocessing module to run LoadEmptyInstrument on the different IDF files in parallel.
Changeset: 208e262536bae205475a55c07284938ae875c9d5
comment:11 Changed 6 years ago by Russell Taylor
The above 2 commits are a rebasing of the previous 2 onto a more recent snapshot of master.
comment:12 Changed 6 years ago by Russell Taylor
- Owner changed from Russell Taylor to Martyn Gigg
The work is done and the branch is at https://github.com/mantidproject/systemtests/tree/feature/7908_parallelize_loadlotsofinstruments
This just needs the blocking ticket to be done before it can be merged and run without hanging.
comment:14 Changed 6 years ago by Nick Draper
Moved to the backlog at the code freeze of R3.3
comment:16 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 8753
Re #7908. Move exception handling into loading function.
Changeset: 1625cecd6ef533a41b52ab3ba9ef41f12835fcff