Fit the SCARF job scheduler remote algorithm in the 'RemoteJobManager' design

[Federico M Pouzols]

Ticket #10591 develops an algorithm to control jobs on SCARF remotely. This is being done for IMAT tomography but can be used in many other contexts.

A so-called-by-me 'RemoteJobManager' design was started some time ago for this very type of algorithms, and the 'AbortRemoteJob', 'Authenticate', 'DownloadRemoteFile', etc. algorithms were implemented and have been used for the Fermi cluster at ORNL.

This 'RemoteJobManager' design is explained/specified on:

• slide 19 of this document: ​https://github.com/mantidproject/documents/blob/master/Presentations/SOS18/Mantid%20HPC%20Challenges.pptx
• this wiki page: http://www.mantidproject.org/Remote_Job_Submission_API

The objective here is to do the SCARF remote job control following this design.

The implementation of the algorithms 'AbortRemoteJob' etc. seems to be a bit specific to Fermi at the moment, so this ticket will most likely require another ticket to move specifics to a IRemoteJobManager subclass, while ensuring that the use of Fermi at ORNL is not affected by our changes.

So this ticket will add a remote job manager that implements the IRemoteJobManager interface, something like SCARFLSFJobManager.

The unit test RemoteJobManagerFactoryTest (introduced in #11124) should also be updated to test creation of this SCARFLSFJobManager type of job manager.

[Federico M Pouzols]

Now that the IRemoteJobManager interface and the factory are in master, this ticket can get in before #11126, and #11126 would be the very last step, needing verification at SNS.

[Federico Montesino Pouzols]

added new LSF and SCARFLSF job managers, re #11064

Changeset: 85a97c6b49a26fd8eb6a69804c93b34398ce3f14

[Federico Montesino Pouzols]

add new RemoteJobManagers subdir, re #11064

Changeset: d49d7f17cd40c497eaea9471a53771c01c68b25e

[Federico Montesino Pouzols]

use the dynamic factory create, more verbose msgs, re #11064

Changeset: 2e062376ab04f7ac5d4b99ba69708d3b01c33fff

[Federico Montesino Pouzols]

extent tests, and move out bits specific to SCARFLSF, re #11064

Changeset: eb3c515afa417938daffb11f2ff87f231955c861

[Federico Montesino Pouzols]

update clean-up of LSFJobManager, new minimal test, re #11064

Changeset: 76084245c5717fcb92f11c3dae9f5c4c3b250199

[Federico Montesino Pouzols]

clean-up SCARFLSF, add ping() and logout(), add tests, re #11064

Changeset: e531821c414b7c412a3a81eed0ff1516417d35ad

[Federico Montesino Pouzols]

This is being verified as ​pull request #503.

[Federico Montesino Pouzols]

Merge remote-tracking branch 'origin/master' into 11064_add_LSFJobManger_and_SCARFLSFJobManager

Conflicts:

Code/Mantid/Framework/CMakeLists.txt

Sort out conflict with removal of MDEvents, re #11064

Changeset: 64d55c05546ff767eedc0b0bce9168c68ca5fcf2

[Federico Montesino Pouzols]

Jenkins, be nice and retest this please

[Federico Montesino Pouzols]

add namespace for rhel7, re #11064

Changeset: e07d8f8796ff824127e0b19cb1bae9bba25c1967

[Federico Montesino Pouzols]

fill in the requested name when empty, cppcheck caught it, re #11064

Changeset: e88bc0d01e753b2046e53ac3f45f078f86a1debe

[Federico Montesino Pouzols]

be more verbose and use exception object.what(), re #11064

Changeset: ec68a4c17ff706ef7de147cb1762eba2b894d7f5

[Federico Montesino Pouzols]

It seems that this is more or less fine, but we got a PoldiCreatePeaksFromFileTest failure on osx and a checkout issue on rhel7. Jenkins, retest this please

[Federico Montesino Pouzols]

Jenkins, you can do it, retest this please

[Federico Montesino Pouzols]

many code improvements suggested by Roman in the PR re #11064, re

Changeset: e3719361e8dac45be0a90290a1b1c2c6195fe4f1

[Federico Montesino Pouzols]

avoid UNUSED_ARG, and prefer TSM_ over TS_ASSERT, re #11064

Changeset: 8626f6f885699678004cfb51346728584ce71917

[Federico Montesino Pouzols]

adjust new tests with proper exception, re #11064

Changeset: 4e1e937ff7c2de8109d84bf250841a4176bcafa7

[Federico Montesino Pouzols]

I've pushed changes for many of your comments and some other improvements to the unit tests that Owen suggested in another PR. A few changes, including the Poco URI and Path points, are still missing. I'll let you know and remove the "in progress" label when this is finished.

[Federico Montesino Pouzols]

avoid unnecessary string/c_str casts, re #11064

Changeset: eee927d5761eb892fc3f7f239111b7617fb636f6

[Federico Montesino Pouzols]

more tests on methods inputs, re #11064

Changeset: db8a64b9ce31c953320ee0ebeae1e5c5cd84dacb

[Federico Montesino Pouzols]

more code improvements, makeHeaders helper, use Poco::Path, re #11064

Changeset: bc3cdb1e2026ac8fdfb2334b7ee8c6a6854d258d

[Federico Montesino Pouzols]

makeHeaders() and use Poco::URI rather than raw strings, re #11064

Changeset: df9855f45d4308b79cf29f55a7c47851d22af8fc

[Federico Montesino Pouzols]

I think this is ready again and the builds seem to be going well, so I've removed the "in progress" label.

[Roman Tolchenov]

Merge pull request #503 from mantidproject/11064_add_LSFJobManger_and_SCARFLSFJobManager

Add classes LSFJobManager and SCARFLSFJobManager

Full changeset: ​280e7d03431239b969ec126377e48b1612cf1024

[Nick Draper]

Somehow these slipped through without a resolution. Set to Fixed.

[Stuart Campbell]

This ticket has been transferred to github issue ​11903