IntegratePeaksMD seg faults sporadically

[Dennis Mikkelson]

When used repeatedly in a script, IntegratePeaksMD will seg fault, typically after processing something between 2 and 6 TOPAZ runs. This is related to OpenMP, since if the line: PRAGMA_OMP( parallel for schedule(dynamic, 10) ) is commented out, IntegratePeaksMD runs reliably.

[Dennis Mikkelson]

Re 5533: Comment out OMP Pragma

If the OMP pragma is included in IntegratePeaksMD, the algorithm seg faults sporadically when processing multiple TOPAZ runs in a script, on Scientific Linux 6.2. Typically, it seg faults after 2 to 6 runs are processed, though occasionally it will process all 8 requested in the script without crashing. Since the lower level codes already use OpenMP, parallelizing at this level is only marginally useful, giving about a 5-10% speedup. Perhaps it should just be removed permanantly, but for now it is commented out to avoid the seg faults. Refs #5533

Changeset: a56c8967e4d65bd6c0502742ddc429c0c4a32b6e

[Russell Taylor]

Dennis, you say the use of OpenMP provides a small benefit, yet the IntegratePeaksMD performances tests have shown a marked slowdown - 0.9 -> 2.7s for one, 1.1 -> 4.0s for the other. How many cores were you running on? (presumably the fewer cores, the less change is seen).

I'll have a look to see if I can spot where the race condition is.

[Dennis Mikkelson]

Re 5533: Comment out OMP Pragma

If the OMP pragma is included in IntegratePeaksMD, the algorithm seg faults sporadically when processing multiple TOPAZ runs in a script, on Scientific Linux 6.2. Typically, it seg faults after 2 to 6 runs are processed, though occasionally it will process all 8 requested in the script without crashing. Since the lower level codes already use OpenMP, parallelizing at this level is only marginally useful, giving about a 5-10% speedup. Perhaps it should just be removed permanantly, but for now it is commented out to avoid the seg faults. Refs #5533

Changeset: a56c8967e4d65bd6c0502742ddc429c0c4a32b6e

[Nick Draper]

Moved at the end of release 2.2

[Nick Draper]

Moved to milestone 2.4

[Nick Draper]

Moved to r2.6 at the end of r2.5

[Nick Draper]

Moved to backlog at the code freeze for R2.6

[Nick Draper]

Bulk move to assigned at the introduction of the triage step

[Russell Taylor]

I took the last commit out of the develop branch because it was still causing the Mac build to get stuck. Moreover, it caused crashes in performance and system tests so it's clearly not threadsafe.

[Vickie Lynch]

Refs #5533 limit number of threads

Changeset: 3d536d35a9bcab324aac9b32d9debda9da90725b

[Vickie Lynch]

Revert "Refs #5533 limit number of threads"

This reverts commit 3d536d35a9bcab324aac9b32d9debda9da90725b.

Changeset: 908e0beee54161bffe1d6582cba7a1b6e8396e7e

[Vickie Lynch]

Refs #5533 try threads with fitting moved inside function

Changeset: 787a9cba192bf1e8c386ebfdd73ade3c8102ff04

[Vickie Lynch]

Refs #5533 fix merge conflict

Changeset: 1f776b496bf10088d41133e58a45ec14d9cf349c

[Vickie Lynch]

Revert "Refs #5533 try threads with fitting moved inside function"

This reverts commit 787a9cba192bf1e8c386ebfdd73ade3c8102ff04.

Changeset: da7ae980f13a5454373c2e4b5edb088b7cf6f79e

[Vickie Lynch]

There is nothing to test in this ticket. All the changes have been reverted primarily because they hang on the Mac build. In ticket 7651 a parallel SCD reduction workflow will be written which will replace the current parallel python scripts and in ticket 9228 IntegratePeaksMD2 will be refactored which may fix the problem with the Mac builds.

[Michael Reuter]

Verified changes have been backed out.

[Stuart Campbell]

This ticket has been transferred to github issue ​6379