Ticket #1330 (closed: fixed)

Opened 10 years ago

Last modified 5 years ago

Improve Homer analysis performance

Reported by: Nick Draper Owned by: Martyn Gigg
Priority: critical Milestone: Iteration 24
Component: Keywords:
Cc: Blocked By:
Blocking: Tester: Steve Williams

Description

Look at where the bottlenecks are and improve.

Specifically think about multi threading as this is often run on a 16+ core machine.

Change History

comment:1 Changed 10 years ago by Martyn Gigg

(In [5237]) Implemented multi threading for the excitations detector efficiency calculation along with code tidying and reworking some parts ot give a speed up in single-threaded modes. Re #1330

comment:2 Changed 10 years ago by Martyn Gigg

(In [5240]) Update the detector effic test to include a small test workspace that includes grouped detectors as the original MARI test doesn't test this. Re #1330

comment:3 Changed 10 years ago by Martyn Gigg

(In [5242]) Fix the tests build. Re #1330

comment:4 Changed 10 years ago by Martyn Gigg

(In [5243]) Fix an array out-of-bounds test bug that Linux let me get away with. Also cleared a warning and cleared up some formatting. Re #1330

comment:5 Changed 10 years ago by Martyn Gigg

(In [5244]) Disable a test until I understand why the Mac is failing. Re #1330

comment:6 Changed 10 years ago by Russell Taylor

(In [5255]) Fixed Mac test for Detector efficiency algorithm. Re #1330

comment:7 Changed 10 years ago by Martyn Gigg

(In [5256]) Hopefully found the real problem. Re #1330

comment:8 Changed 10 years ago by Martyn Gigg

(In [5257]) Reworked the detector shape caching as it wasn't thread safe, which I originally thought it was. Re #1330

comment:9 Changed 10 years ago by Martyn Gigg

(In [5258]) Add parallelisation to the detector diagnostic algorithms. Re #1330

comment:10 Changed 10 years ago by Martyn Gigg

(In [5263]) The order a list in MedianDetectorTest matters so use a set instead of a vector. Also refactored the file writing code. Re #1330

comment:11 Changed 10 years ago by Martyn Gigg

(In [5268]) Use GroupDetectors algorithm in efficiency test rather than doing it by hand. Re #1330

comment:12 Changed 10 years ago by Martyn Gigg

  • Status changed from new to accepted

DetectorEfficiencyCor algorithm has dropped from 14 seconds to 1 second on an 8 core machine. Detector diagnosis algorithms have also been multi threaded so a big increase there.

comment:13 Changed 10 years ago by Martyn Gigg

  • Status changed from accepted to testing
  • Resolution set to fixed

comment:14 Changed 10 years ago by Nick Draper

  • Status changed from testing to verify

Moved to verify state

comment:15 Changed 10 years ago by Steve Williams

  • Status changed from verify to verifying
  • Tester set to Steve Williams

comment:16 Changed 10 years ago by Steve Williams

  • Status changed from verifying to reopened
  • Resolution fixed deleted

There's a problem just affects running under Redhat

comment:17 Changed 10 years ago by Martyn Gigg

(In [5954]) Fix some Python 2.4 compatability problems. Re #1330

comment:18 Changed 10 years ago by Martyn Gigg

  • Status changed from reopened to accepted

comment:19 Changed 10 years ago by Martyn Gigg

  • Status changed from accepted to verify
  • Resolution set to fixed

Issues were with Python 2.4 and missing functions

comment:20 Changed 10 years ago by Martyn Gigg

(In [5968]) Add some multi-threading protection that was missing. Re #1330

comment:21 Changed 10 years ago by Steve Williams

  • Status changed from verify to verifying

comment:22 Changed 10 years ago by Steve Williams

  • Status changed from verifying to closed

Converting a MAPS run on a multiple processor Redhat machine the CPU usage was consistently >100%, except the save algorithms that ran at ~100%. The whole reduction detector efficiency, convert units and absolute units took 1 minute on mantidlx1.

comment:23 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue ​2177

Note: See TracTickets for help on using tickets.