Ticket #4884 (closed: fixed)

Opened 9 years ago

Last modified 5 years ago

Implement Job Submission to Remote Clusters

Reported by: Ross Miller Owned by: Ross Miller
Priority: major Milestone: Release 2.6
Component: MantidPlot Keywords: Mantid-MPI ADARA
Cc: Blocked By:
Blocking: #6924 Tester: Mathieu Doucet

Description

Implement functionality to submit and track jobs running on remote compute clusters.

Initial work will be via Moab Web Services, but there needs to be an abstraction layer so that other job submission schemes can be implemented.

See Chapter 6 of the ADARA design document.

Change History

comment:1 Changed 8 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

refs #4884

Changeset: 9d78735367a204d1058e8963f5c2ddc9ace76a0e

comment:2 Changed 8 years ago by Ross Miller

Further work on the remote algorithms project.

Still not anywhere near complete, but some more functionality has been added: cluster info is save to / restored from the properties file and the code to fetch the specified configuration file from a URL works.

Refs #4884

Changeset: 79ec21f58dba0371f5d68b1cdbe8350b025363cc

comment:3 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: b3176dfcf9397eb041b93a4ede4190819574b552

comment:4 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: d4663554859ff1d4d45b968e3bc88e72c8403d77

comment:5 Changed 8 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: f9320e86d64abd86a2f70d22856fb97df0e2e466

comment:6 Changed 8 years ago by Ross Miller

Minor comment change

refs #4884

Changeset: b55fd2a94579d8727a96b1b4be13bd679ed15597

comment:7 Changed 8 years ago by Ross Miller

Small tweaks to the remote job submission code.

Setting the job name (instead of leaving it as the default of STDIN) and picking up the user id from the RemoteJobManager class. Still need to figure out how to handle the password though!! refs #4884

Changeset: 0f50d82543c47ec1f5534fbcd28e16053249a0f7

comment:8 Changed 8 years ago by Ross Miller

Fixed a signed/unsigned comparison warning

refs #4884

Changeset: d43bc4467cafe7b3d9ce0a1729884b6e64e1d28c

comment:9 Changed 8 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Changeset: 419ddcfd443ac976e10a12fb0611e4169d7d54fe

comment:10 Changed 8 years ago by Ross Miller

Minor cleanups in preparation for a code review

Declared default constructor for RemoteJob private (and left unimplemented), removed unnecessary #include's from MantidRemoteAlg.h and commented out a currently unused paramter in RemoteAlgorithmDockWidget::xmlParseServerAttributes()

Refs #4884

Changeset: 673ec2a23adfc942519bbcf2583482dd8a90c21c

comment:11 Changed 8 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: e75089be537f10dba7615fc8f6d764828dd869d4

comment:12 Changed 8 years ago by Ross Miller

Replaced a few more instances of 'algorithm' with 'task'

Refs #4884

Changeset: 6afcfc862b0b048a62cd2fbff9d660d07839131a

comment:13 Changed 8 years ago by Ross Miller

Changes to the remote job submission code

Added code to automatically pick a name for the remote job's output file and pass that as a parameter to the job submission URL. This is needed to allow the user to download the results after the remote job has run.

Refs #4884

Changeset: 3cd2be453e12d5bd0b2ad6c08443697872f78f4c

comment:14 Changed 8 years ago by Ross Miller

Added ability to download output files from cluster

'Download' buttons in the Job Status dialog will now actually attempt to download the output file and open it in a new workspace. Also added code to verify that there actually is an output file available before adding the download button.

Refs #4884

Changeset: 3364c2bcb44897aebf8364b21aaec3786577e67c

comment:15 Changed 8 years ago by Ross Miller

Only show jobs submitted from MantidPlot in the job status dialog

Uses the 'variables' entry in the submitted JSON to set an entry called "SUBMITTING_APP" when the job is submitted. The jobStatusAll() function checks for this entry and only includes jobs that have it set in its result list.

Refs #4884

Changeset: 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21

comment:16 Changed 8 years ago by Ross Miller

Disable previous change to job status dialog

Commit 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21 changed the job status dialog box to only show jobs that were actually submitted from MantidPlot. Unfortunately, there's a bug in Moab Web Services (support case #15864) that renders this useless. Once the folks at Adaptive Computing fix their bug, I'll uncomment one line of code to re-enable this feature.

Refs #4884

Changeset: ec10df543ee43832c4954e96e07771a1f9ce5888

comment:17 Changed 8 years ago by Ross Miller

Several changes to job status dialog (and job submission in general)

1) HTTPS is now enabled and enforced. No unencrypted connections are allowed. (Note: We're accepting the certificate from the server as-is with no validation. That's not really correct, but it's the best we can do for now.) 2) Jobs in the job status dialog are now sorted by their ID's 3) Had to disable the use of QTemporaryFile objects for downloading the output files because the Load algorithm runs in a separate thread and the object was cleaning up its file before the algorithm had a chance to run.

Refs #4884

Changeset: ad09fd3970607afa7e36e4bbcbd8a1c3a3c20db9

comment:18 Changed 8 years ago by Ross Miller

Better error messages if there's a problem parsing the XML config file

Refs #4884

Changeset: 3e5d53c7c115c53d1b895382c2058da1b89de223

comment:19 Changed 8 years ago by Ross Miller

Jub submission GUI now supports checkboxes for boolean values

Refs #4884

Changeset: 17c7d011f41cebf8d2002ed3aa2d984e53b011da

comment:20 Changed 8 years ago by Ross Miller

Enable session cookies in MWSRemoteJobManager

Refs #4884

Changeset: 8fa9d837bf460a01574326a034e21d2484a08230

comment:21 Changed 8 years ago by Ross Miller

Starting to make GUI changes for the new job submission design

Refs #4884

Changeset: 93089021385e64a7a56c06e37fbc4d6b4860b4ab

comment:22 Changed 8 years ago by Ross Miller

More minor GUI changes plus changing a class name

Refs #4884

Changeset: 3574d978ec4778c23e29c98db91f33ac589d0f73

comment:23 Changed 8 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: e883109062cb549377be6d3e79803a214872809b

comment:24 Changed 8 years ago by Ross Miller

Make the remote cluster list a singleton object

This is in preparation for the new design for remote job submission. In the new design, individual algorithms will need a way to access the cluster list, so it can't be stored as part of the dock widget.

Refs #4884

Changeset: 20dce3e941acd477793f236702c867a25d536668

comment:25 Changed 8 years ago by Ross Miller

Starting to add functions necessary for the new design of the...

...remote job submission code. Specifically, adding functions for starting & stopping transactions.

Refs #4884

Changeset: 3d2eb6359cd92e950588c9b95381172b9305ad15

comment:26 Changed 8 years ago by Ross Miller

Comment & whitespace cleanups only - no code changes

Refs #4884

Changeset: 47f6f0ee8ac5cd34d15c050c4be62a6e34285103

comment:27 Changed 8 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: cb9c8e5c7d80868bd0d6011b068daa8ae8e84ea8

comment:28 Changed 8 years ago by Ross Miller

Add Fermi cluster to Facilities.xml

Adding a description of the Fermi cluster to Facilities.xml

Refs #4884

Changeset: 48bbc9e7347563746f42e4c744e47d361c0d5d58

comment:29 Changed 8 years ago by Ross Miller

Fix typo in a comment

refs #4884

Changeset: 566d22c23d9990df7f3b46647a04b6139b3b08c3

comment:30 Changed 8 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

refs #4884

Changeset: ae82af26c846f0ba01eb845d9ba6ce3f88942833

comment:31 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: 5d4672bb61fa97dbb97d571127a03533533a74a7

comment:32 Changed 8 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: 48b7d3018198c2709e1ea310e3f62e394e5b5ed4

comment:33 Changed 8 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Changeset: 3e87fdd3d34cfd1c33d53794687544edf098dbae

comment:34 Changed 8 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: cc19d7c17eca8c7e22b440b285e0b6fbfbb9c456

comment:35 Changed 8 years ago by Ross Miller

Update CMakeLists to compile the new JSON parser code

Refs #4884

Changeset: b8e273960b7473bb4ae9d3165b2b48defebf9813

comment:36 Changed 8 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: cf76adc6b9fdf417bcf45ec38e98cda54bc658dc

comment:37 Changed 8 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: e75d4b20bdef1aaa08ee365d866d117cc0f66aab

comment:38 Changed 8 years ago by Ross Miller

Cleaning up the mess caused by renaming the remote branch...

...and then re-basing off master. (Not sure why, but all kinds of manual conflict resolution was required.) These changes are probably because I did the conflict resolution incorrectly...

Refs #4884

Changeset: 3923b63695659e26b5b493037e09eedf21068871

comment:39 Changed 8 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

Cherry-picked from original commit: 9d78735367a204d1058e8963f5c2ddc9ace76a0e

refs #4884

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 13f20a76ab5ef5e23b423e60b38a11137491cd82

comment:40 Changed 8 years ago by Ross Miller

Further work on the remote algorithms project.

Still not anywhere near complete, but some more functionality has been added: cluster info is save to / restored from the properties file and the code to fetch the specified configuration file from a URL works.

Refs #4884

Changeset: e968d027d25390abf7e40d8570f22d6f290db2b2

comment:41 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: 26bdc03dda83076df230859653c70e18d8dddd13

comment:42 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Cherry-picked from original commit: d4663554859ff1d4d45b968e3bc88e72c8403d77

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 9345cc4bcdc279c8e0cd9d61c29aaa544b7ee84f

comment:43 Changed 8 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: 11053ee634cf50b643d560a545eff926fe8badb4

comment:44 Changed 8 years ago by Ross Miller

Minor comment change

refs #4884

Changeset: f2cdf6f8383545ed91767542ba6f4b2c334f52a5

comment:45 Changed 8 years ago by Ross Miller

Small tweaks to the remote job submission code.

Setting the job name (instead of leaving it as the default of STDIN) and picking up the user id from the RemoteJobManager class. Still need to figure out how to handle the password though!! refs #4884

Changeset: cad8e6dde07380ecc0164d89cc0d56af69ad9054

comment:46 Changed 8 years ago by Ross Miller

Fixed a signed/unsigned comparison warning

refs #4884

Changeset: 7208f5db8b53fadb3edd1e8a35ee3a7775d40d1a

comment:47 Changed 8 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Cherry-picked from original commit: 419ddcfd443ac976e10a12fb0611e4169d7d54fe

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: c3a6c2d5e81ada966eb645c7fb15df67b9c325c6

comment:48 Changed 8 years ago by Ross Miller

Minor cleanups in preparation for a code review

Declared default constructor for RemoteJob private (and left unimplemented), removed unnecessary #include's from MantidRemoteAlg.h and commented out a currently unused paramter in RemoteAlgorithmDockWidget::xmlParseServerAttributes()

Refs #4884

Changeset: 0ecc4c4f2c76e7681328d5807c233ec786b0aed2

comment:49 Changed 8 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: c51f1d5617c1794697e82d2bc3ee356290f35e71

comment:50 Changed 8 years ago by Ross Miller

Replaced a few more instances of 'algorithm' with 'task'

Refs #4884

Changeset: 652231400e7361120f02409d0e52a6bff13b0210

comment:51 Changed 8 years ago by Ross Miller

Filter the job status dialog

Added a slider widget to the job status dialog so that users can filter out jobs older than a specified number of days. NOTE: This is just the GUI bits; the actual filtering code isn't ready yet.

Refs #4884

Changeset: c081756f62c00817f233c2f9b92c8f68ef102b68

comment:52 Changed 8 years ago by Ross Miller

Adding a JSON parser for Moab Web Services output

Refs #4884

Changeset: 0eb6462c0eada5ed56469fc4c3a809f63a45df47

comment:53 Changed 8 years ago by Ross Miller

Update CMakeLists to compile the new JSON parser code

Refs #4884

Changeset: 8316eaca0dd80b0d06032ad7ce59500bf6dfdf65

comment:54 Changed 8 years ago by Ross Miller

Change Job Status Dialog Box

Changed the job status dialog box to query the MWS server for all the jobs the user has submitted. Uses the new JSON parser code to parse the (rather complicated) output. Also includes a spinbox to let the user set the maximum age of jobs to be displayed.

Refs #4884

Changeset: 3084b6a879dfe70adde846ef68a6ee7dab783069

comment:55 Changed 8 years ago by Ross Miller

Changes to the remote job submission code

Added code to automatically pick a name for the remote job's output file and pass that as a parameter to the job submission URL. This is needed to allow the user to download the results after the remote job has run.

Refs #4884

Changeset: 48e45f219b1c89d6597d7440c3c3087545640396

comment:56 Changed 8 years ago by Ross Miller

Added ability to download output files from cluster

'Download' buttons in the Job Status dialog will now actually attempt to download the output file and open it in a new workspace. Also added code to verify that there actually is an output file available before adding the download button.

Refs #4884

Changeset: 9cfb785bc423a47fdb386c25391b51d2b40695ef

comment:57 Changed 8 years ago by Ross Miller

Only show jobs submitted from MantidPlot in the job status dialog

Uses the 'variables' entry in the submitted JSON to set an entry called "SUBMITTING_APP" when the job is submitted. The jobStatusAll() function checks for this entry and only includes jobs that have it set in its result list.

Refs #4884

Changeset: 99f408268ec0b825f11fce334eeadacd49038979

comment:58 Changed 8 years ago by Ross Miller

Disable previous change to job status dialog

Commit 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21 changed the job status dialog box to only show jobs that were actually submitted from MantidPlot. Unfortunately, there's a bug in Moab Web Services (support case #15864) that renders this useless. Once the folks at Adaptive Computing fix their bug, I'll uncomment one line of code to re-enable this feature.

Refs #4884

Changeset: 9f72ea64f82a4053c8d9244993b7c7d09e157114

comment:59 Changed 8 years ago by Ross Miller

Several changes to job status dialog (and job submission in general)

1) HTTPS is now enabled and enforced. No unencrypted connections are allowed. (Note: We're accepting the certificate from the server as-is with no validation. That's not really correct, but it's the best we can do for now.) 2) Jobs in the job status dialog are now sorted by their ID's 3) Had to disable the use of QTemporaryFile objects for downloading the output files because the Load algorithm runs in a separate thread and the object was cleaning up its file before the algorithm had a chance to run.

Refs #4884

Changeset: 448df4f0fd7f053d33b707edab3a4431b948e549

comment:60 Changed 8 years ago by Ross Miller

Preliminary support for Combo Boxes in the job submission dialog

NOTE: NOT FULLY TESTED YET! Refs #4884

Changeset: b4a95c037ba4bbeed2c81e6cd59b8b3a3e3e3862

comment:61 Changed 8 years ago by Ross Miller

Better error messages if there's a problem parsing the XML config file

Refs #4884

Changeset: 2f7416bcbaba571f919756ed7c1ea8f92727bdbf

comment:62 Changed 8 years ago by Ross Miller

Jub submission GUI now supports checkboxes for boolean values

Refs #4884

Changeset: 97a3fa50851d69e67860136d873abbfba6c58630

comment:63 Changed 8 years ago by Ross Miller

Enable session cookies in MWSRemoteJobManager

Refs #4884

Changeset: 15a9098ecabad7350f12e73a3a02763d4984a770

comment:64 Changed 8 years ago by Ross Miller

Starting to make GUI changes for the new job submission design

Refs #4884

Changeset: 95df1703c0ec2365f000f8cbf861b1a22e7b6da9

comment:65 Changed 8 years ago by Ross Miller

More minor GUI changes plus changing a class name

Refs #4884

Changeset: 21d8ff219245bb7b164d92e28e2d10699026dcc2

comment:66 Changed 8 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: f4e10f00bbed1e86d64492f427cd7957ab0a09da

comment:67 Changed 8 years ago by Ross Miller

Make the remote cluster list a singleton object

This is in preparation for the new design for remote job submission. In the new design, individual algorithms will need a way to access the cluster list, so it can't be stored as part of the dock widget.

Refs #4884

Changeset: 557bbe552cdbad6742716e0d3fb622d1b9361409

comment:68 Changed 8 years ago by Ross Miller

Starting to add functions necessary for the new design of the...

...remote job submission code. Specifically, adding functions for starting & stopping transactions.

Refs #4884

Changeset: eee01d60d7630213cc3ae6bcc8df646b9e62f2bf

comment:69 Changed 8 years ago by Ross Miller

Comment & whitespace cleanups only - no code changes

Refs #4884

Changeset: c67b4a2904b11aae4d0e460ad3373e1c5f336162

comment:70 Changed 8 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: f6b283c06d007156fa40447cdd16251fdb543cfc

comment:71 Changed 8 years ago by Ross Miller

Add Fermi cluster to Facilities.xml

Adding a description of the Fermi cluster to Facilities.xml

Refs #4884

Changeset: 13357b1ad898a1eb056fe9049bd3246a16646cb4

comment:72 Changed 8 years ago by Ross Miller

Fix typo in a comment

refs #4884

Changeset: fde97ad9b5b71de4ca49a81a1d5887864a7f75f5

comment:73 Changed 8 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

Cherry-picked from original commit: 9d78735367a204d1058e8963f5c2ddc9ace76a0e

refs #4884

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 930d7005ba9f768e877c4978a7a5889fc076660b

comment:74 Changed 8 years ago by Ross Miller

Further work on the remote algorithms project.

Still not anywhere near complete, but some more functionality has been added: cluster info is save to / restored from the properties file and the code to fetch the specified configuration file from a URL works.

Refs #4884

Changeset: c75c2c193e286f2310ade7a4b2fb7da4b8daa9a6

comment:75 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: 43bc8606fc2787fe06bb9355347e38e36862ea24

comment:76 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Cherry-picked from original commit: d4663554859ff1d4d45b968e3bc88e72c8403d77

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 3ac6f19462799752fb1a96ac5e197eedca4f17d1

comment:77 Changed 8 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: 11e92f2261f477c63c2ad8bd82b81bff248086b1

comment:78 Changed 8 years ago by Ross Miller

Minor comment change

refs #4884

Changeset: 949cfd5d329e096435ef8a9a1ddc67981a06dc84

comment:79 Changed 8 years ago by Ross Miller

Small tweaks to the remote job submission code.

Setting the job name (instead of leaving it as the default of STDIN) and picking up the user id from the RemoteJobManager class. Still need to figure out how to handle the password though!! refs #4884

Changeset: 9f28438b2f2826baa05eab6fd8d11d4c31fe0fbf

comment:80 Changed 8 years ago by Ross Miller

Fixed a signed/unsigned comparison warning

refs #4884

Changeset: e10c2ee5e14f7ab483e4dabe17869156cbf4de07

comment:81 Changed 8 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Cherry-picked from original commit: 419ddcfd443ac976e10a12fb0611e4169d7d54fe

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 9bdad0d3d1fbefbc5de2d276a15ad31806d21f54

comment:82 Changed 8 years ago by Ross Miller

Minor cleanups in preparation for a code review

Declared default constructor for RemoteJob private (and left unimplemented), removed unnecessary #include's from MantidRemoteAlg.h and commented out a currently unused paramter in RemoteAlgorithmDockWidget::xmlParseServerAttributes()

Refs #4884

Changeset: 66dc9004125bab10a8e352d43934fcc51259d9d6

comment:83 Changed 8 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: b0398533bc3bbf95a656833d5c83277eb3748330

comment:84 Changed 8 years ago by Ross Miller

Replaced a few more instances of 'algorithm' with 'task'

Refs #4884

Changeset: 0d275e5cf72badd8d1cb3e40dd995a65c91144de

comment:85 Changed 8 years ago by Ross Miller

Filter the job status dialog

Added a slider widget to the job status dialog so that users can filter out jobs older than a specified number of days. NOTE: This is just the GUI bits; the actual filtering code isn't ready yet.

Refs #4884

Changeset: 6bcef6c076f9ff48b1eaedb8f13b001c98a4a64a

comment:86 Changed 8 years ago by Ross Miller

Adding a JSON parser for Moab Web Services output

Refs #4884

Changeset: cf6d781198551cb1f8b14b6c5425058fbb4f02f5

comment:87 Changed 8 years ago by Ross Miller

Update CMakeLists to compile the new JSON parser code

Refs #4884

Changeset: 8a37c6fbc91ae7f5427a6f6aac2a443280c62502

comment:88 Changed 8 years ago by Ross Miller

Change Job Status Dialog Box

Changed the job status dialog box to query the MWS server for all the jobs the user has submitted. Uses the new JSON parser code to parse the (rather complicated) output. Also includes a spinbox to let the user set the maximum age of jobs to be displayed.

Refs #4884

Changeset: 840acc960f738cdc8b1fbb2b63e72d27dc62c30c

comment:89 Changed 8 years ago by Ross Miller

Changes to the remote job submission code

Added code to automatically pick a name for the remote job's output file and pass that as a parameter to the job submission URL. This is needed to allow the user to download the results after the remote job has run.

Refs #4884

Changeset: 085ff8c57b65f5d946daa42eacd87971e9935758

comment:90 Changed 8 years ago by Ross Miller

Added ability to download output files from cluster

'Download' buttons in the Job Status dialog will now actually attempt to download the output file and open it in a new workspace. Also added code to verify that there actually is an output file available before adding the download button.

Refs #4884

Changeset: b2678a7ea93efd265c5c8e1faa6f16b159fc7b99

comment:91 Changed 8 years ago by Ross Miller

Only show jobs submitted from MantidPlot in the job status dialog

Uses the 'variables' entry in the submitted JSON to set an entry called "SUBMITTING_APP" when the job is submitted. The jobStatusAll() function checks for this entry and only includes jobs that have it set in its result list.

Refs #4884

Changeset: d6927a27ed0517cb9bf3c7b1ec5802811ccbdd14

comment:92 Changed 8 years ago by Ross Miller

Disable previous change to job status dialog

Commit 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21 changed the job status dialog box to only show jobs that were actually submitted from MantidPlot. Unfortunately, there's a bug in Moab Web Services (support case #15864) that renders this useless. Once the folks at Adaptive Computing fix their bug, I'll uncomment one line of code to re-enable this feature.

Refs #4884

Changeset: ee0ac80c469afed56339a7efc751706fb3d96c0d

comment:93 Changed 8 years ago by Ross Miller

Several changes to job status dialog (and job submission in general)

1) HTTPS is now enabled and enforced. No unencrypted connections are allowed. (Note: We're accepting the certificate from the server as-is with no validation. That's not really correct, but it's the best we can do for now.) 2) Jobs in the job status dialog are now sorted by their ID's 3) Had to disable the use of QTemporaryFile objects for downloading the output files because the Load algorithm runs in a separate thread and the object was cleaning up its file before the algorithm had a chance to run.

Refs #4884

Changeset: 92492c1a94fd593f3fa6ede2488053185d8e3294

comment:94 Changed 8 years ago by Ross Miller

Preliminary support for Combo Boxes in the job submission dialog

NOTE: NOT FULLY TESTED YET! Refs #4884

Changeset: 4e84edf560b9f7c4b0ca0ad1f3863103e5b48d68

comment:95 Changed 8 years ago by Ross Miller

Better error messages if there's a problem parsing the XML config file

Refs #4884

Changeset: c973a993b55e0cae2cabd2a634f069f1dbe8d218

comment:96 Changed 8 years ago by Ross Miller

Jub submission GUI now supports checkboxes for boolean values

Refs #4884

Changeset: c4909066f8608bd10a69c0c483531d1c02028854

comment:97 Changed 8 years ago by Ross Miller

Enable session cookies in MWSRemoteJobManager

Refs #4884

Changeset: dfbeaad22c67f492b5d150177e1d927408d3019d

comment:98 Changed 8 years ago by Ross Miller

Starting to make GUI changes for the new job submission design

Refs #4884

Changeset: 745792a3e06ed5de703d4894cfa2f4b2370d52ec

comment:99 Changed 8 years ago by Ross Miller

More minor GUI changes plus changing a class name

Refs #4884

Changeset: bfdb792ed4cb96a0f73b9de7ffcf347ca006f7b1

comment:100 Changed 8 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: 270ac746a886e607dcb51c98bffb293e4e9bfc82

comment:101 Changed 8 years ago by Ross Miller

Make the remote cluster list a singleton object

This is in preparation for the new design for remote job submission. In the new design, individual algorithms will need a way to access the cluster list, so it can't be stored as part of the dock widget.

Refs #4884

Changeset: 5ad2d4f50bf15f76c686a7d260437f609cc7f4a9

comment:102 Changed 8 years ago by Ross Miller

Starting to add functions necessary for the new design of the...

...remote job submission code. Specifically, adding functions for starting & stopping transactions.

Refs #4884

Changeset: 814763c4156f9c0b4f399a3d9f1f3f9134594376

comment:103 Changed 8 years ago by Ross Miller

Comment & whitespace cleanups only - no code changes

Refs #4884

Changeset: ccacbe7e5960dd6624176439b354b029e1b6b28e

comment:104 Changed 8 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: 84fd907f394ee9b2538e6514e734c54c17a8f3ea

comment:105 Changed 8 years ago by Ross Miller

Add Fermi cluster to Facilities.xml

Adding a description of the Fermi cluster to Facilities.xml

Refs #4884

Changeset: 386481b2cf85ebebc0c52b759dacbc9a3da6f080

comment:106 Changed 8 years ago by Ross Miller

Fix typo in a comment

refs #4884

Changeset: 9bcfbe7e8a188e33a7220acae5ef5fc6c46811c3

comment:107 Changed 8 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

Cherry-picked from original commit: 9d78735367a204d1058e8963f5c2ddc9ace76a0e

refs #4884

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 135b917f070497cc1c9dda77f155091ed9da95dd

comment:108 Changed 8 years ago by Ross Miller

Further work on the remote algorithms project.

Still not anywhere near complete, but some more functionality has been added: cluster info is save to / restored from the properties file and the code to fetch the specified configuration file from a URL works.

Refs #4884

Changeset: a1cea83ff4571479f71e6220aa330962aa23f43e

comment:109 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: 6852ff377c0496fa692421c78ebc2ffa39cb5d48

comment:110 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Cherry-picked from original commit: d4663554859ff1d4d45b968e3bc88e72c8403d77

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 9b2c8e25d20647277c702e67746d421237b30761

comment:111 Changed 8 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: eba0bdf30dee51f09b859d9f4bc8adfb6918ea11

comment:112 Changed 8 years ago by Ross Miller

Minor comment change

refs #4884

Changeset: de1d780e3e8c7eee9d2bbce582ba14d02c7e6d78

comment:113 Changed 8 years ago by Ross Miller

Small tweaks to the remote job submission code.

Setting the job name (instead of leaving it as the default of STDIN) and picking up the user id from the RemoteJobManager class. Still need to figure out how to handle the password though!! refs #4884

Changeset: cfffff87fcf05c822cc903aeb1f8fddae8ee0aa9

comment:114 Changed 8 years ago by Ross Miller

Fixed a signed/unsigned comparison warning

refs #4884

Changeset: 3d3247a24fbfd4f1e4261309afcd7860f1201fb3

comment:115 Changed 8 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Cherry-picked from original commit: 419ddcfd443ac976e10a12fb0611e4169d7d54fe

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 05c762c25e462ad9bbac0d69cbebba1b71a27e76

comment:116 Changed 8 years ago by Ross Miller

Minor cleanups in preparation for a code review

Declared default constructor for RemoteJob private (and left unimplemented), removed unnecessary #include's from MantidRemoteAlg.h and commented out a currently unused paramter in RemoteAlgorithmDockWidget::xmlParseServerAttributes()

Refs #4884

Changeset: 1d5935066c6c9687ff2f1425d0c1fcc509dcbcfc

comment:117 Changed 8 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: 635de30b949e01f96542e4037740f23f2c1c644d

comment:118 Changed 8 years ago by Ross Miller

Replaced a few more instances of 'algorithm' with 'task'

Refs #4884

Changeset: 60f0aeb2292f28384141d293f646a002ab3aa3f7

comment:119 Changed 8 years ago by Ross Miller

Filter the job status dialog

Added a slider widget to the job status dialog so that users can filter out jobs older than a specified number of days. NOTE: This is just the GUI bits; the actual filtering code isn't ready yet.

Refs #4884

Changeset: e969b48f171bbda055a4003d7114801faffe297e

comment:120 Changed 8 years ago by Ross Miller

Adding a JSON parser for Moab Web Services output

Refs #4884

Changeset: 745ce2bd79e9eeccaa3685a7f3968e7dcb6d651e

comment:121 Changed 8 years ago by Ross Miller

Update CMakeLists to compile the new JSON parser code

Refs #4884

Changeset: a8e3146699c6b529147c5383514e65c399cc3bd6

comment:122 Changed 8 years ago by Ross Miller

Change Job Status Dialog Box

Changed the job status dialog box to query the MWS server for all the jobs the user has submitted. Uses the new JSON parser code to parse the (rather complicated) output. Also includes a spinbox to let the user set the maximum age of jobs to be displayed.

Refs #4884

Changeset: 94ebca1ca9db02e2dac3292aa6171ec517f17f8a

comment:123 Changed 8 years ago by Ross Miller

Changes to the remote job submission code

Added code to automatically pick a name for the remote job's output file and pass that as a parameter to the job submission URL. This is needed to allow the user to download the results after the remote job has run.

Refs #4884

Changeset: 8aa7c6c8b1959a007cf5092d13b3302afc2fc64f

comment:124 Changed 8 years ago by Ross Miller

Added ability to download output files from cluster

'Download' buttons in the Job Status dialog will now actually attempt to download the output file and open it in a new workspace. Also added code to verify that there actually is an output file available before adding the download button.

Refs #4884

Changeset: 8af7f7138026e07d60dcf57789440921cd863b58

comment:125 Changed 8 years ago by Ross Miller

Only show jobs submitted from MantidPlot in the job status dialog

Uses the 'variables' entry in the submitted JSON to set an entry called "SUBMITTING_APP" when the job is submitted. The jobStatusAll() function checks for this entry and only includes jobs that have it set in its result list.

Refs #4884

Changeset: c4cf4aab1072427fcf9ad637a482804952d95c07

comment:126 Changed 8 years ago by Ross Miller

Disable previous change to job status dialog

Commit 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21 changed the job status dialog box to only show jobs that were actually submitted from MantidPlot. Unfortunately, there's a bug in Moab Web Services (support case #15864) that renders this useless. Once the folks at Adaptive Computing fix their bug, I'll uncomment one line of code to re-enable this feature.

Refs #4884

Changeset: 9b6cfc37c0a41539f77698de1389b5e6dff94fd4

comment:127 Changed 8 years ago by Ross Miller

Several changes to job status dialog (and job submission in general)

1) HTTPS is now enabled and enforced. No unencrypted connections are allowed. (Note: We're accepting the certificate from the server as-is with no validation. That's not really correct, but it's the best we can do for now.) 2) Jobs in the job status dialog are now sorted by their ID's 3) Had to disable the use of QTemporaryFile objects for downloading the output files because the Load algorithm runs in a separate thread and the object was cleaning up its file before the algorithm had a chance to run.

Refs #4884

Changeset: 466c4eef9014de8478615e516adace4efbff2c4b

comment:128 Changed 8 years ago by Ross Miller

Preliminary support for Combo Boxes in the job submission dialog

NOTE: NOT FULLY TESTED YET! Refs #4884

Changeset: 71e5c8a7ddaa8b907cd6f0c67a8a21c8da05e3df

comment:129 Changed 8 years ago by Ross Miller

Better error messages if there's a problem parsing the XML config file

Refs #4884

Changeset: fa1fad2863642f84ed17023236359f1a929f8565

comment:130 Changed 8 years ago by Ross Miller

Jub submission GUI now supports checkboxes for boolean values

Refs #4884

Changeset: bfcc456a03315baaae6abedaa2432212b9802a6b

comment:131 Changed 8 years ago by Ross Miller

Enable session cookies in MWSRemoteJobManager

Refs #4884

Changeset: 21a8d648167b8245ecdc73f27c3368d4dd3d0f3a

comment:132 Changed 8 years ago by Ross Miller

Starting to make GUI changes for the new job submission design

Refs #4884

Changeset: e74d96e93f755c53b47d0f9bf8cb31d487ed58f3

comment:133 Changed 8 years ago by Ross Miller

More minor GUI changes plus changing a class name

Refs #4884

Changeset: 3f6c36513cdc541f28b503276221032737941b17

comment:134 Changed 8 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: 52467d3b3287d1d419ba8aa7e5ec1d272a986c6f

comment:135 Changed 8 years ago by Ross Miller

Make the remote cluster list a singleton object

This is in preparation for the new design for remote job submission. In the new design, individual algorithms will need a way to access the cluster list, so it can't be stored as part of the dock widget.

Refs #4884

Changeset: d1e0afedfc492a73408980c4b9742abfb37903bf

comment:136 Changed 8 years ago by Ross Miller

Starting to add functions necessary for the new design of the...

...remote job submission code. Specifically, adding functions for starting & stopping transactions.

Refs #4884

Changeset: 2564cca89039018f3cfddeb02fa2ce00edcb630e

comment:137 Changed 8 years ago by Ross Miller

Comment & whitespace cleanups only - no code changes

Refs #4884

Changeset: bcdca96f966ab68c502f0e84b3f6876e3f268928

comment:138 Changed 8 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: 84b81913530ef85b21190870cd158ef2df16d597

comment:139 Changed 8 years ago by Ross Miller

Add Fermi cluster to Facilities.xml

Adding a description of the Fermi cluster to Facilities.xml

Refs #4884

Changeset: 519b60b8a75fe7754dc1f4fbfd9918ddfd3feefb

comment:140 Changed 8 years ago by Ross Miller

Fix typo in a comment

refs #4884

Changeset: 4991be4549c7946ad49c0501dfaa6c82a3b18f78

comment:141 Changed 8 years ago by Ross Miller

First cut at using Algorithms for job submission

Not anywhere near complete, but wanted to get it checked in.

Refs #4884

Changeset: b4d37c31970f1701930971f4be7371fb0b3bf475

comment:142 Changed 8 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

Cherry-picked from original commit: 9d78735367a204d1058e8963f5c2ddc9ace76a0e

refs #4884

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: ec00ec09792a5f7f85522d850045c707e40ab468

comment:143 Changed 8 years ago by Ross Miller

Further work on the remote algorithms project.

Still not anywhere near complete, but some more functionality has been added: cluster info is save to / restored from the properties file and the code to fetch the specified configuration file from a URL works.

Refs #4884

Changeset: 6dfe0b0b829a0437df381a904ed0727a9c3bd873

comment:144 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: 82db1d4d3b4717e6130f0969684992de1cc33417

comment:145 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Cherry-picked from original commit: d4663554859ff1d4d45b968e3bc88e72c8403d77

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: d67ef5a631acf5ad1e465e301562ee1659811bcc

comment:146 Changed 8 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: eb67cdaf1219f972c8a15423b43d4e166132cfab

comment:147 Changed 8 years ago by Ross Miller

Minor comment change

refs #4884

Changeset: 2b55887814e070d6d5c5a783be3f94e9c5684b9a

comment:148 Changed 8 years ago by Ross Miller

Small tweaks to the remote job submission code.

Setting the job name (instead of leaving it as the default of STDIN) and picking up the user id from the RemoteJobManager class. Still need to figure out how to handle the password though!! refs #4884

Changeset: bd85ec7c3628c37968480a7c958226bf8391d5b4

comment:149 Changed 8 years ago by Ross Miller

Fixed a signed/unsigned comparison warning

refs #4884

Changeset: 108c39bfac83f9d0e4bea098087c33f793c2b2e7

comment:150 Changed 8 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Cherry-picked from original commit: 419ddcfd443ac976e10a12fb0611e4169d7d54fe

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: cf4bd5b8ab2e6da6c5616faa06a16c683f8c6ce8

comment:151 Changed 8 years ago by Ross Miller

Minor cleanups in preparation for a code review

Declared default constructor for RemoteJob private (and left unimplemented), removed unnecessary #include's from MantidRemoteAlg.h and commented out a currently unused paramter in RemoteAlgorithmDockWidget::xmlParseServerAttributes()

Refs #4884

Changeset: a3870cd899ea237ec097180fc6ef75b640626e87

comment:152 Changed 8 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: 88ce6ce3b325ada226a9385e437ba107c26e2791

comment:153 Changed 8 years ago by Ross Miller

Replaced a few more instances of 'algorithm' with 'task'

Refs #4884

Changeset: 72868707b829ca359799677b5ad550da8cc792cb

comment:154 Changed 8 years ago by Ross Miller

Filter the job status dialog

Added a slider widget to the job status dialog so that users can filter out jobs older than a specified number of days. NOTE: This is just the GUI bits; the actual filtering code isn't ready yet.

Refs #4884

Changeset: 6254c92f80440716bd232891d525b682b33c0c3a

comment:155 Changed 8 years ago by Ross Miller

Adding a JSON parser for Moab Web Services output

Refs #4884

Changeset: 218272c004b88677eda89a01b91489ba69656da6

comment:156 Changed 8 years ago by Ross Miller

Update CMakeLists to compile the new JSON parser code

Refs #4884

Changeset: fe13347d54d00fc3c382d9dd74cdc1f4ee6db339

comment:157 Changed 8 years ago by Ross Miller

Change Job Status Dialog Box

Changed the job status dialog box to query the MWS server for all the jobs the user has submitted. Uses the new JSON parser code to parse the (rather complicated) output. Also includes a spinbox to let the user set the maximum age of jobs to be displayed.

Refs #4884

Changeset: e7e320e051539f9bc9ced4cee35dd2ac6cb3cc76

comment:158 Changed 8 years ago by Ross Miller

Changes to the remote job submission code

Added code to automatically pick a name for the remote job's output file and pass that as a parameter to the job submission URL. This is needed to allow the user to download the results after the remote job has run.

Refs #4884

Changeset: 9ca71aced6df572af26a31fe89d10c45703e537f

comment:159 Changed 8 years ago by Ross Miller

Added ability to download output files from cluster

'Download' buttons in the Job Status dialog will now actually attempt to download the output file and open it in a new workspace. Also added code to verify that there actually is an output file available before adding the download button.

Refs #4884

Changeset: 7fffa8e241707a11225f18eab0e4c38fdbfc9564

comment:160 Changed 8 years ago by Ross Miller

Only show jobs submitted from MantidPlot in the job status dialog

Uses the 'variables' entry in the submitted JSON to set an entry called "SUBMITTING_APP" when the job is submitted. The jobStatusAll() function checks for this entry and only includes jobs that have it set in its result list.

Refs #4884

Changeset: 411d74401fd2d2f25b162e9810ad12b542cc6c14

comment:161 Changed 8 years ago by Ross Miller

Disable previous change to job status dialog

Commit 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21 changed the job status dialog box to only show jobs that were actually submitted from MantidPlot. Unfortunately, there's a bug in Moab Web Services (support case #15864) that renders this useless. Once the folks at Adaptive Computing fix their bug, I'll uncomment one line of code to re-enable this feature.

Refs #4884

Changeset: 22ac676a9e8e287ee934170d8595bf2d96bb6374

comment:162 Changed 8 years ago by Ross Miller

Several changes to job status dialog (and job submission in general)

1) HTTPS is now enabled and enforced. No unencrypted connections are allowed. (Note: We're accepting the certificate from the server as-is with no validation. That's not really correct, but it's the best we can do for now.) 2) Jobs in the job status dialog are now sorted by their ID's 3) Had to disable the use of QTemporaryFile objects for downloading the output files because the Load algorithm runs in a separate thread and the object was cleaning up its file before the algorithm had a chance to run.

Refs #4884

Changeset: ac7dc5dcd4d21e83c8e2b11961799be7e4e002dd

comment:163 Changed 8 years ago by Ross Miller

Preliminary support for Combo Boxes in the job submission dialog

NOTE: NOT FULLY TESTED YET! Refs #4884

Changeset: 490d49be25cd09c828702c0ebcab89b2f48d37b2

comment:164 Changed 8 years ago by Ross Miller

Better error messages if there's a problem parsing the XML config file

Refs #4884

Changeset: 2ed5e81e5c8a7374a62b03efe2b1424a9875be59

comment:165 Changed 8 years ago by Ross Miller

Jub submission GUI now supports checkboxes for boolean values

Refs #4884

Changeset: 3c7a62280188f8e11c480b10f8c623bf9bb2eec4

comment:166 Changed 8 years ago by Ross Miller

Enable session cookies in MWSRemoteJobManager

Refs #4884

Changeset: ee44832ec5c4531754a5006508a0c877521acef1

comment:167 Changed 8 years ago by Ross Miller

Starting to make GUI changes for the new job submission design

Refs #4884

Changeset: 32d654c9c639a86ae7af4ec88b4506448fb05b57

comment:168 Changed 8 years ago by Ross Miller

More minor GUI changes plus changing a class name

Refs #4884

Changeset: 4144ba148206e904e2fff14171e591a16fd81cb7

comment:169 Changed 8 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: 889227e70e516f301963dad33d2b0e805676a868

comment:170 Changed 8 years ago by Ross Miller

Make the remote cluster list a singleton object

This is in preparation for the new design for remote job submission. In the new design, individual algorithms will need a way to access the cluster list, so it can't be stored as part of the dock widget.

Refs #4884

Changeset: fd221f5476f8bac6c966080d8897227b7ea8e9c4

comment:171 Changed 8 years ago by Ross Miller

Starting to add functions necessary for the new design of the...

...remote job submission code. Specifically, adding functions for starting & stopping transactions.

Refs #4884

Changeset: fee8ff5141295e878bff6cfb8a5e2f59b483fe06

comment:172 Changed 8 years ago by Ross Miller

Comment & whitespace cleanups only - no code changes

Refs #4884

Changeset: 03072663564a41c6d81b6c8c8edd7164f191fbe6

comment:173 Changed 8 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: 9779134e86d57048c4d97a78b5e5ca257238afdf

comment:174 Changed 8 years ago by Ross Miller

Add Fermi cluster to Facilities.xml

Adding a description of the Fermi cluster to Facilities.xml

Refs #4884

Changeset: bd4135b06d238fbae8fbcb274b6aa13a139cf64e

comment:175 Changed 8 years ago by Ross Miller

Fix typo in a comment

refs #4884

Changeset: fbd00ab37ac824210a47d68402604e5937237f41

comment:176 Changed 8 years ago by Ross Miller

First cut at using Algorithms for job submission

Not anywhere near complete, but wanted to get it checked in.

Refs #4884

Changeset: 7ca75da08732369dc98ca9657e6e8451a93f8c23

comment:177 Changed 8 years ago by Ross Miller

Remove QtMwsRemoteJobManager

Got rid of the QtMwsRemoteJobManager class and made MWSRemoteJobManager the concrete class. Had to add functions to set the username and password to the RemoteJobManager API. (Need to do this so python scripts can use the remote job submission stuff.)

Refs #4884

Changeset: f6d474f12d152093508eccec79e85ff9c4a7b008

comment:178 Changed 8 years ago by Ross Miller

Remove a reference to QtMwsRemoteJobManager.hpp

MantidRemoteCluster.cpp (which itself is going to be deleted soon) still had a reference to QtMwsRemoteJobManager.hpp (which is now gone).

Refs #4884

Changeset: 1d3e6dbe59c049b037db9f6d3b835bf66d0bf944

comment:179 Changed 8 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

Cherry-picked from original commit: 9d78735367a204d1058e8963f5c2ddc9ace76a0e

refs #4884

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: acd23ed5b85af3b4211b71693a078c56e22af6e1

comment:180 Changed 8 years ago by Ross Miller

Further work on the remote algorithms project.

Still not anywhere near complete, but some more functionality has been added: cluster info is save to / restored from the properties file and the code to fetch the specified configuration file from a URL works.

Refs #4884

Changeset: 1002ef63845964b16f08e83f48c6d1d94f84c138

comment:181 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: 97c8d46c0733100a395c5785ffa7b70d43efae80

comment:182 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Cherry-picked from original commit: d4663554859ff1d4d45b968e3bc88e72c8403d77

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 2c4b9dcd38f65861d60d823ac60125d6d4becfe5

comment:183 Changed 8 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: 5149f32b930c3c12cd16e1c2a82b8d00cb80bd26

comment:184 Changed 8 years ago by Ross Miller

Minor comment change

refs #4884

Changeset: c5d9202391c610c0718f756c7d6575153abdc479

comment:185 Changed 8 years ago by Ross Miller

Small tweaks to the remote job submission code.

Setting the job name (instead of leaving it as the default of STDIN) and picking up the user id from the RemoteJobManager class. Still need to figure out how to handle the password though!! refs #4884

Changeset: be7836448d79f2ef7941bb8790819424fb755a73

comment:186 Changed 8 years ago by Ross Miller

Fixed a signed/unsigned comparison warning

refs #4884

Changeset: 61d983013b9ee565e4027f9c7efe05e2e4750efe

comment:187 Changed 8 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Cherry-picked from original commit: 419ddcfd443ac976e10a12fb0611e4169d7d54fe

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 7260ebc6384b6b41fcf1a57f52a1b9dd22520546

comment:188 Changed 8 years ago by Ross Miller

Minor cleanups in preparation for a code review

Declared default constructor for RemoteJob private (and left unimplemented), removed unnecessary #include's from MantidRemoteAlg.h and commented out a currently unused paramter in RemoteAlgorithmDockWidget::xmlParseServerAttributes()

Refs #4884

Changeset: 6bc5e779210a0859112bf058219020b018eaa302

comment:189 Changed 8 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: 3ee685c2e1cd170c278683d682c5551555f81c77

comment:190 Changed 8 years ago by Ross Miller

Replaced a few more instances of 'algorithm' with 'task'

Refs #4884

Changeset: 331e5fcbdd027baea57f5ca545be04e6ef1ad7e1

comment:191 Changed 8 years ago by Ross Miller

Filter the job status dialog

Added a slider widget to the job status dialog so that users can filter out jobs older than a specified number of days. NOTE: This is just the GUI bits; the actual filtering code isn't ready yet.

Refs #4884

Changeset: f4bd88afc4ba220db7f89924e116c01c78d84806

comment:192 Changed 8 years ago by Ross Miller

Adding a JSON parser for Moab Web Services output

Refs #4884

Changeset: 7fd8d66a3e55e9de4ef1005a9092974e04c6ecc2

comment:193 Changed 8 years ago by Ross Miller

Update CMakeLists to compile the new JSON parser code

Refs #4884

Changeset: 75026296d0780d4c46863ff90efa6c26d0fdb2b9

comment:194 Changed 8 years ago by Ross Miller

Change Job Status Dialog Box

Changed the job status dialog box to query the MWS server for all the jobs the user has submitted. Uses the new JSON parser code to parse the (rather complicated) output. Also includes a spinbox to let the user set the maximum age of jobs to be displayed.

Refs #4884

Changeset: 5e14b90530852332c8b65e6854149b7e7978e873

comment:195 Changed 8 years ago by Ross Miller

Changes to the remote job submission code

Added code to automatically pick a name for the remote job's output file and pass that as a parameter to the job submission URL. This is needed to allow the user to download the results after the remote job has run.

Refs #4884

Changeset: 0b279b4185f9e0e29427958fd8b4c5be675e9d9f

comment:196 Changed 8 years ago by Ross Miller

Added ability to download output files from cluster

'Download' buttons in the Job Status dialog will now actually attempt to download the output file and open it in a new workspace. Also added code to verify that there actually is an output file available before adding the download button.

Refs #4884

Changeset: 9d9cc258cbaf05e0780f827e5ad8250b358e1c8c

comment:197 Changed 8 years ago by Ross Miller

Only show jobs submitted from MantidPlot in the job status dialog

Uses the 'variables' entry in the submitted JSON to set an entry called "SUBMITTING_APP" when the job is submitted. The jobStatusAll() function checks for this entry and only includes jobs that have it set in its result list.

Refs #4884

Changeset: 411a8ace56ae04e1370c511dc864479c0e2ee6e5

comment:198 Changed 8 years ago by Ross Miller

Disable previous change to job status dialog

Commit 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21 changed the job status dialog box to only show jobs that were actually submitted from MantidPlot. Unfortunately, there's a bug in Moab Web Services (support case #15864) that renders this useless. Once the folks at Adaptive Computing fix their bug, I'll uncomment one line of code to re-enable this feature.

Refs #4884

Changeset: dbe9e1178b1b242d1399a0e58a8c85d551302eff

comment:199 Changed 8 years ago by Ross Miller

Several changes to job status dialog (and job submission in general)

1) HTTPS is now enabled and enforced. No unencrypted connections are allowed. (Note: We're accepting the certificate from the server as-is with no validation. That's not really correct, but it's the best we can do for now.) 2) Jobs in the job status dialog are now sorted by their ID's 3) Had to disable the use of QTemporaryFile objects for downloading the output files because the Load algorithm runs in a separate thread and the object was cleaning up its file before the algorithm had a chance to run.

Refs #4884

Changeset: 7eca24da2fa29ed801976a47670282e4b2a2dc1c

comment:200 Changed 8 years ago by Ross Miller

Preliminary support for Combo Boxes in the job submission dialog

NOTE: NOT FULLY TESTED YET! Refs #4884

Changeset: e6fc785b94107486e39438c2fd9dac1ad04f9f6a

comment:201 Changed 8 years ago by Ross Miller

Better error messages if there's a problem parsing the XML config file

Refs #4884

Changeset: da20090faf9431f2f3d5cc06c20a986053afaf17

comment:202 Changed 8 years ago by Ross Miller

Jub submission GUI now supports checkboxes for boolean values

Refs #4884

Changeset: 04d6769d4bb012976a64224ea0ca99a7861a2670

comment:203 Changed 8 years ago by Ross Miller

Enable session cookies in MWSRemoteJobManager

Refs #4884

Changeset: 022abaf80a0588f71fd67d2481bd44be90dcf727

comment:204 Changed 8 years ago by Ross Miller

Starting to make GUI changes for the new job submission design

Refs #4884

Changeset: 84dcbafe3112d65ece99bda1427e229c879a444c

comment:205 Changed 8 years ago by Ross Miller

More minor GUI changes plus changing a class name

Refs #4884

Changeset: be731260287f6d10b004a4574f38c8ca1c76c7b0

comment:206 Changed 8 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: 0cbb6239c1bbb7c9cfeea5dc9b2c743da69404ba

comment:207 Changed 8 years ago by Ross Miller

Make the remote cluster list a singleton object

This is in preparation for the new design for remote job submission. In the new design, individual algorithms will need a way to access the cluster list, so it can't be stored as part of the dock widget.

Refs #4884

Changeset: e38537331aac9e57b5bbf7858e5faae17ba79f3f

comment:208 Changed 8 years ago by Ross Miller

Starting to add functions necessary for the new design of the...

...remote job submission code. Specifically, adding functions for starting & stopping transactions.

Refs #4884

Changeset: ba18afd13a67f9310ea5afb117a13297c582ab84

comment:209 Changed 8 years ago by Ross Miller

Comment & whitespace cleanups only - no code changes

Refs #4884

Changeset: 964e3cc8d5348db2f15ceff1c37eaaf256dde775

comment:210 Changed 8 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: c16e0d22bf81fb7c04d0921b25e9268813436e96

comment:211 Changed 8 years ago by Ross Miller

Add Fermi cluster to Facilities.xml

Adding a description of the Fermi cluster to Facilities.xml

Refs #4884

Changeset: d4690ce60d189d9e11742797b6757b2d04f37a85

comment:212 Changed 8 years ago by Ross Miller

Fix typo in a comment

refs #4884

Changeset: 6440d9dd838864b5dbb17ebed4a8f50fdc87401a

comment:213 Changed 8 years ago by Ross Miller

First cut at using Algorithms for job submission

Not anywhere near complete, but wanted to get it checked in.

Refs #4884

Changeset: 76f1112d8df595e4ddbcb60cea4ff763724a16f8

comment:214 Changed 8 years ago by Ross Miller

Remove QtMwsRemoteJobManager

Got rid of the QtMwsRemoteJobManager class and made MWSRemoteJobManager the concrete class. Had to add functions to set the username and password to the RemoteJobManager API. (Need to do this so python scripts can use the remote job submission stuff.)

Refs #4884

Changeset: ffb69af2a2d38692b7c5611f340caf1a12486a5e

comment:215 Changed 8 years ago by Ross Miller

Remove a reference to QtMwsRemoteJobManager.hpp

MantidRemoteCluster.cpp (which itself is going to be deleted soon) still had a reference to QtMwsRemoteJobManager.hpp (which is now gone).

Refs #4884

Changeset: 3caa45a9926fe03b68e923dfc240e36aaaa95b6c

comment:216 Changed 8 years ago by Ross Miller

More work on the Remote Job Submission stuff

1) Added location of mpirun and python executables to Facilities.xml 2) Removed property for executable from submit algorithm (because of #1) 3) Changed the password property to a masked property 4) Added validators to the other properties

Refs #4884

Changeset: 8e571013b7bc8d6ce48e0df584cdff883fd95174

comment:217 Changed 8 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

Cherry-picked from original commit: 9d78735367a204d1058e8963f5c2ddc9ace76a0e

refs #4884

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: c32b68055c9e3d2741ffb094b8359bc4992abb88

comment:218 Changed 8 years ago by Ross Miller

Further work on the remote algorithms project.

Still not anywhere near complete, but some more functionality has been added: cluster info is save to / restored from the properties file and the code to fetch the specified configuration file from a URL works.

Refs #4884

Changeset: 92f9118a5b82805705a8be1a77c91f2bab9a93e2

comment:219 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: b4a1f0a91d19af0de9daba7ce9904a3d2bf418a0

comment:220 Changed 8 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Cherry-picked from original commit: d4663554859ff1d4d45b968e3bc88e72c8403d77

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: c898d2cb236c7b1b3213f14ccc668c97f069f2b9

comment:221 Changed 8 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: a57827a817862c6dc7208e96cb828e029e8f9d92

comment:222 Changed 8 years ago by Ross Miller

Minor comment change

refs #4884

Changeset: c371cde431f70385861dfe1c94fd22ef8f629c5c

comment:223 Changed 8 years ago by Ross Miller

Small tweaks to the remote job submission code.

Setting the job name (instead of leaving it as the default of STDIN) and picking up the user id from the RemoteJobManager class. Still need to figure out how to handle the password though!! refs #4884

Changeset: 5e9a192c0b8b91a6c5b786ced581250567876985

comment:224 Changed 8 years ago by Ross Miller

Fixed a signed/unsigned comparison warning

refs #4884

Changeset: 3e77ddba9b2cb75795b67db0033567449e78759b

comment:225 Changed 8 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Cherry-picked from original commit: 419ddcfd443ac976e10a12fb0611e4169d7d54fe

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 56cf4830d456568d6bbd28f4fe96ce531cf4ef1d

comment:226 Changed 8 years ago by Ross Miller

Minor cleanups in preparation for a code review

Declared default constructor for RemoteJob private (and left unimplemented), removed unnecessary #include's from MantidRemoteAlg.h and commented out a currently unused paramter in RemoteAlgorithmDockWidget::xmlParseServerAttributes()

Refs #4884

Changeset: 29ac75aea9f4361a6bc7f01ac76af015927ca1d5

comment:227 Changed 8 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: 8630681cdbda317f09ba057fe3512bee4e034d0b

comment:228 Changed 8 years ago by Ross Miller

Replaced a few more instances of 'algorithm' with 'task'

Refs #4884

Changeset: da2184e8b3e2a55a06c6baf52d435d3773ce9fbd

comment:229 Changed 8 years ago by Ross Miller

Filter the job status dialog

Added a slider widget to the job status dialog so that users can filter out jobs older than a specified number of days. NOTE: This is just the GUI bits; the actual filtering code isn't ready yet.

Refs #4884

Changeset: 636cd1dc41e5cf9826d4b1a4e35010386c7005c5

comment:230 Changed 8 years ago by Ross Miller

Adding a JSON parser for Moab Web Services output

Refs #4884

Changeset: ff5354b92ac4fcb3df08fc9a2ed726d6aba74522

comment:231 Changed 8 years ago by Ross Miller

Update CMakeLists to compile the new JSON parser code

Refs #4884

Changeset: 8550af43325ef4a065e4bd10f447882facb5e2be

comment:232 Changed 8 years ago by Ross Miller

Change Job Status Dialog Box

Changed the job status dialog box to query the MWS server for all the jobs the user has submitted. Uses the new JSON parser code to parse the (rather complicated) output. Also includes a spinbox to let the user set the maximum age of jobs to be displayed.

Refs #4884

Changeset: 400e162e54fd9e9fb7fc9332812a53b3619a7e18

comment:233 Changed 8 years ago by Ross Miller

Changes to the remote job submission code

Added code to automatically pick a name for the remote job's output file and pass that as a parameter to the job submission URL. This is needed to allow the user to download the results after the remote job has run.

Refs #4884

Changeset: b41f4e531d2a402801daa9d7d576c11f90003fb6

comment:234 Changed 8 years ago by Ross Miller

Added ability to download output files from cluster

'Download' buttons in the Job Status dialog will now actually attempt to download the output file and open it in a new workspace. Also added code to verify that there actually is an output file available before adding the download button.

Refs #4884

Changeset: dcf41df12a99c80fe797b4b06821d52cb6b45c06

comment:235 Changed 8 years ago by Ross Miller

Only show jobs submitted from MantidPlot in the job status dialog

Uses the 'variables' entry in the submitted JSON to set an entry called "SUBMITTING_APP" when the job is submitted. The jobStatusAll() function checks for this entry and only includes jobs that have it set in its result list.

Refs #4884

Changeset: 5f95f9f044563c8c0ae5977108639b4711d9b853

comment:236 Changed 8 years ago by Ross Miller

Disable previous change to job status dialog

Commit 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21 changed the job status dialog box to only show jobs that were actually submitted from MantidPlot. Unfortunately, there's a bug in Moab Web Services (support case #15864) that renders this useless. Once the folks at Adaptive Computing fix their bug, I'll uncomment one line of code to re-enable this feature.

Refs #4884

Changeset: c7c9593afb45337a5b98e2680f31b007346bf2b2

comment:237 Changed 8 years ago by Ross Miller

Several changes to job status dialog (and job submission in general)

1) HTTPS is now enabled and enforced. No unencrypted connections are allowed. (Note: We're accepting the certificate from the server as-is with no validation. That's not really correct, but it's the best we can do for now.) 2) Jobs in the job status dialog are now sorted by their ID's 3) Had to disable the use of QTemporaryFile objects for downloading the output files because the Load algorithm runs in a separate thread and the object was cleaning up its file before the algorithm had a chance to run.

Refs #4884

Changeset: d5e9d3be96ce8e532a1dc1e3962c067cc092700e

comment:238 Changed 8 years ago by Ross Miller

Preliminary support for Combo Boxes in the job submission dialog

NOTE: NOT FULLY TESTED YET! Refs #4884

Changeset: b80419d13e20311398fac6edda6eb81282d45302

comment:239 Changed 8 years ago by Ross Miller

Better error messages if there's a problem parsing the XML config file

Refs #4884

Changeset: 31d1ad36ef05b23aa093216079e6ecfd3ad11c47

comment:240 Changed 8 years ago by Ross Miller

Jub submission GUI now supports checkboxes for boolean values

Refs #4884

Changeset: a170e05653fae853a81f6cbceff1f703a73d9db4

comment:241 Changed 8 years ago by Ross Miller

Enable session cookies in MWSRemoteJobManager

Refs #4884

Changeset: bc7689d2f63d0a7be900ef31989ddaddcdf4f921

comment:242 Changed 8 years ago by Ross Miller

Starting to make GUI changes for the new job submission design

Refs #4884

Changeset: 08ab1a58a4b969b289a31390971530e637f0dfb2

comment:243 Changed 8 years ago by Ross Miller

More minor GUI changes plus changing a class name

Refs #4884

Changeset: 429b0fd6fcb0270bf60167a7b186a4292f7b90c9

comment:244 Changed 8 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: b5547d1f9ed829f1e846e712b8bb284e8bca4a92

comment:245 Changed 8 years ago by Ross Miller

Make the remote cluster list a singleton object

This is in preparation for the new design for remote job submission. In the new design, individual algorithms will need a way to access the cluster list, so it can't be stored as part of the dock widget.

Refs #4884

Changeset: ac8388e8ae366d01a89b761d4b10e196fdb1f4cc

comment:246 Changed 8 years ago by Ross Miller

Starting to add functions necessary for the new design of the...

...remote job submission code. Specifically, adding functions for starting & stopping transactions.

Refs #4884

Changeset: 35e270b7a76240aa4f7e39f03074571f6c024b1b

comment:247 Changed 8 years ago by Ross Miller

Comment & whitespace cleanups only - no code changes

Refs #4884

Changeset: fff4d5223745072b7368ade218ffcc60bd02436d

comment:248 Changed 8 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: 39f1afb320bb4e4d317b47ee06e44fb18d81eaba

comment:249 Changed 8 years ago by Ross Miller

Add Fermi cluster to Facilities.xml

Adding a description of the Fermi cluster to Facilities.xml

Refs #4884

Changeset: f8e66f035f41bd53d281de43c3b67d9d21d6b3b4

comment:250 Changed 8 years ago by Ross Miller

Fix typo in a comment

refs #4884

Changeset: b60e4838f8002fd2157e2182b7f1742da12e0455

comment:251 Changed 8 years ago by Ross Miller

First cut at using Algorithms for job submission

Not anywhere near complete, but wanted to get it checked in.

Refs #4884

Changeset: b5205b3f9c1d4269ca68238bd88a38eaad173901

comment:252 Changed 8 years ago by Ross Miller

Remove QtMwsRemoteJobManager

Got rid of the QtMwsRemoteJobManager class and made MWSRemoteJobManager the concrete class. Had to add functions to set the username and password to the RemoteJobManager API. (Need to do this so python scripts can use the remote job submission stuff.)

Refs #4884

Changeset: 34c61e072b3c3ab9360cbf25ca95f0d08a013f37

comment:253 Changed 8 years ago by Ross Miller

Remove a reference to QtMwsRemoteJobManager.hpp

MantidRemoteCluster.cpp (which itself is going to be deleted soon) still had a reference to QtMwsRemoteJobManager.hpp (which is now gone).

Refs #4884

Changeset: 746ca5ab7be635353183fbb9e89802f44624350c

comment:254 Changed 8 years ago by Ross Miller

More work on the Remote Job Submission stuff

1) Added location of mpirun and python executables to Facilities.xml 2) Removed property for executable from submit algorithm (because of #1) 3) Changed the password property to a masked property 4) Added validators to the other properties

Refs #4884

Changeset: 234a3d73a8de717fffc93ddd8f096704c852c129

comment:255 Changed 8 years ago by Ross Miller

Added algorithms for starting & stopping remote transactions

Added algorithms to start and stop remote transactions. Also fixed the exception throwing in the SubmitRemoteJob algorithm.

Refs #4884

Changeset: 9db4c771a2a7031ff919a14b4fa5d7a1071ae335

comment:256 Changed 7 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

Cherry-picked from original commit: 9d78735367a204d1058e8963f5c2ddc9ace76a0e

refs #4884

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 06512bc7d1e4311de088e2c6b8129c9ba96e5004

comment:257 Changed 7 years ago by Ross Miller

Further work on the remote algorithms project.

Still not anywhere near complete, but some more functionality has been added: cluster info is save to / restored from the properties file and the code to fetch the specified configuration file from a URL works.

Refs #4884

Changeset: 8c2e125a979e453790d89479089d5d230fb68267

comment:258 Changed 7 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: 55ce5563168f569e9db7781e28714cc55b7ce6e0

comment:259 Changed 7 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Cherry-picked from original commit: d4663554859ff1d4d45b968e3bc88e72c8403d77

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 234221ef6676e8b0d42a3493682b9f57549546ad

comment:260 Changed 7 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: add18c1ca4a8ed2ca13e9e299c7a93269eac3a7e

comment:261 Changed 7 years ago by Ross Miller

Minor comment change

refs #4884

Changeset: 605983ea02de1db1a353a7da122a9bc21a1011e1

comment:262 Changed 7 years ago by Ross Miller

Small tweaks to the remote job submission code.

Setting the job name (instead of leaving it as the default of STDIN) and picking up the user id from the RemoteJobManager class. Still need to figure out how to handle the password though!! refs #4884

Changeset: 2be11953f85ef3e040c7aec53e42bf6e6154a0f1

comment:263 Changed 7 years ago by Ross Miller

Fixed a signed/unsigned comparison warning

refs #4884

Changeset: 68ee3aea296397b501215ec668f3bf14c479df17

comment:264 Changed 7 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Cherry-picked from original commit: 419ddcfd443ac976e10a12fb0611e4169d7d54fe

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: dae804e9d4b13ba25acec63fa4f8ad2a3b0d8399

comment:265 Changed 7 years ago by Ross Miller

Minor cleanups in preparation for a code review

Declared default constructor for RemoteJob private (and left unimplemented), removed unnecessary #include's from MantidRemoteAlg.h and commented out a currently unused paramter in RemoteAlgorithmDockWidget::xmlParseServerAttributes()

Refs #4884

Changeset: da11bccbda7dd759d97bb4f3550fa92b97ce6a60

comment:266 Changed 7 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: ffa99154278c036396feb6f04e5a3bfdb6fccc19

comment:267 Changed 7 years ago by Ross Miller

Replaced a few more instances of 'algorithm' with 'task'

Refs #4884

Changeset: 5ed43c6f75411ea3453645296874cb9f51911b0b

comment:268 Changed 7 years ago by Ross Miller

Filter the job status dialog

Added a slider widget to the job status dialog so that users can filter out jobs older than a specified number of days. NOTE: This is just the GUI bits; the actual filtering code isn't ready yet.

Refs #4884

Changeset: 00bbf214c0a9c1670c1bcacf35a8ee61a565044c

comment:269 Changed 7 years ago by Ross Miller

Adding a JSON parser for Moab Web Services output

Refs #4884

Changeset: 5a2d369b95730a20f9df4ff8705072152b758456

comment:270 Changed 7 years ago by Ross Miller

Update CMakeLists to compile the new JSON parser code

Refs #4884

Changeset: 83a401bcb4cadb792231cbd663a111b9d00ff301

comment:271 Changed 7 years ago by Ross Miller

Change Job Status Dialog Box

Changed the job status dialog box to query the MWS server for all the jobs the user has submitted. Uses the new JSON parser code to parse the (rather complicated) output. Also includes a spinbox to let the user set the maximum age of jobs to be displayed.

Refs #4884

Changeset: c8d690edb7394648f018a6df09d982e6c255e39a

comment:272 Changed 7 years ago by Ross Miller

Changes to the remote job submission code

Added code to automatically pick a name for the remote job's output file and pass that as a parameter to the job submission URL. This is needed to allow the user to download the results after the remote job has run.

Refs #4884

Changeset: cb7d4525570370b41cca1bdf4c2263f4a6771ec1

comment:273 Changed 7 years ago by Ross Miller

Added ability to download output files from cluster

'Download' buttons in the Job Status dialog will now actually attempt to download the output file and open it in a new workspace. Also added code to verify that there actually is an output file available before adding the download button.

Refs #4884

Changeset: b4fe43f6f94034fcd8bee366e627851b1b71f35d

comment:274 Changed 7 years ago by Ross Miller

Only show jobs submitted from MantidPlot in the job status dialog

Uses the 'variables' entry in the submitted JSON to set an entry called "SUBMITTING_APP" when the job is submitted. The jobStatusAll() function checks for this entry and only includes jobs that have it set in its result list.

Refs #4884

Changeset: 17bbb39f71268b1e3dda961258b4a8ac5f1a986f

comment:275 Changed 7 years ago by Ross Miller

Disable previous change to job status dialog

Commit 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21 changed the job status dialog box to only show jobs that were actually submitted from MantidPlot. Unfortunately, there's a bug in Moab Web Services (support case #15864) that renders this useless. Once the folks at Adaptive Computing fix their bug, I'll uncomment one line of code to re-enable this feature.

Refs #4884

Changeset: fedc90fe9497e6d2ba25d889e0da322370a26a75

comment:276 Changed 7 years ago by Ross Miller

Several changes to job status dialog (and job submission in general)

1) HTTPS is now enabled and enforced. No unencrypted connections are allowed. (Note: We're accepting the certificate from the server as-is with no validation. That's not really correct, but it's the best we can do for now.) 2) Jobs in the job status dialog are now sorted by their ID's 3) Had to disable the use of QTemporaryFile objects for downloading the output files because the Load algorithm runs in a separate thread and the object was cleaning up its file before the algorithm had a chance to run.

Refs #4884

Changeset: 9d576b104981698d11c30cc5d477b7b5fb614afa

comment:277 Changed 7 years ago by Ross Miller

Preliminary support for Combo Boxes in the job submission dialog

NOTE: NOT FULLY TESTED YET! Refs #4884

Changeset: a497aea763620053259f8d605a8696428e47e640

comment:278 Changed 7 years ago by Ross Miller

Better error messages if there's a problem parsing the XML config file

Refs #4884

Changeset: 6950ad4cabef46a41af06e064b804e45a81074d7

comment:279 Changed 7 years ago by Ross Miller

Jub submission GUI now supports checkboxes for boolean values

Refs #4884

Changeset: d1018e969ab40c5a14e2366e840405668b78e7f4

comment:280 Changed 7 years ago by Ross Miller

Enable session cookies in MWSRemoteJobManager

Refs #4884

Changeset: ce0b1f34732d066f2db46fd6a97464d78be4b2ac

comment:281 Changed 7 years ago by Ross Miller

Starting to make GUI changes for the new job submission design

Refs #4884

Changeset: d3c66c48bdbf6353965c992f5264101c8b65bdff

comment:282 Changed 7 years ago by Ross Miller

More minor GUI changes plus changing a class name

Refs #4884

Changeset: 68085e56e97a6238e9b0d07f55f98e119634d16b

comment:283 Changed 7 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: 62dd60509854ba19830a718b28a25b5d65ca8066

comment:284 Changed 7 years ago by Ross Miller

Make the remote cluster list a singleton object

This is in preparation for the new design for remote job submission. In the new design, individual algorithms will need a way to access the cluster list, so it can't be stored as part of the dock widget.

Refs #4884

Changeset: 7baed1d0fe21f60661fdbbae80f68c56483c8ec0

comment:285 Changed 7 years ago by Ross Miller

Starting to add functions necessary for the new design of the...

...remote job submission code. Specifically, adding functions for starting & stopping transactions.

Refs #4884

Changeset: e8d7983852e498e03d3a442d8329bd06dc42d6b7

comment:286 Changed 7 years ago by Ross Miller

Comment & whitespace cleanups only - no code changes

Refs #4884

Changeset: dac682bf3c46fb0d2e137a2bd9b470cd8a39170f

comment:287 Changed 7 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: 77a040c1bffda739cf120a37f6284621ff86f519

comment:288 Changed 7 years ago by Ross Miller

Add Fermi cluster to Facilities.xml

Adding a description of the Fermi cluster to Facilities.xml

Refs #4884

Changeset: d3e6954396f2d4df146331857f5f3339b7bda04f

comment:289 Changed 7 years ago by Ross Miller

Fix typo in a comment

refs #4884

Changeset: 07a3485e29de520e20168114074fcb422d7f5012

comment:290 Changed 7 years ago by Ross Miller

First cut at using Algorithms for job submission

Not anywhere near complete, but wanted to get it checked in.

Refs #4884

Changeset: f5227140e23df999e5836b69cc1c003cb88d7118

comment:291 Changed 7 years ago by Ross Miller

Remove QtMwsRemoteJobManager

Got rid of the QtMwsRemoteJobManager class and made MWSRemoteJobManager the concrete class. Had to add functions to set the username and password to the RemoteJobManager API. (Need to do this so python scripts can use the remote job submission stuff.)

Refs #4884

Changeset: 56ac8ad4ea649a071e8a4544e717c963cacee0cf

comment:292 Changed 7 years ago by Ross Miller

Remove a reference to QtMwsRemoteJobManager.hpp

MantidRemoteCluster.cpp (which itself is going to be deleted soon) still had a reference to QtMwsRemoteJobManager.hpp (which is now gone).

Refs #4884

Changeset: 3bdd2b579a0b0f86595cfbf8d7acef7676d7dc3d

comment:293 Changed 7 years ago by Ross Miller

More work on the Remote Job Submission stuff

1) Added location of mpirun and python executables to Facilities.xml 2) Removed property for executable from submit algorithm (because of #1) 3) Changed the password property to a masked property 4) Added validators to the other properties

Refs #4884

Changeset: 82ceec34249ff3ca0213b98c0d00cd41025fbbe6

comment:294 Changed 7 years ago by Ross Miller

Added algorithms for starting & stopping remote transactions

Added algorithms to start and stop remote transactions. Also fixed the exception throwing in the SubmitRemoteJob algorithm.

Refs #4884

Changeset: a8d469a10d231441bfe8f7d62731d686634e79f6

comment:295 Changed 7 years ago by Ross Miller

API changes to RemoteJobManager class

Replaced jobOutputRead() and getJobOutput() with more generic functions for directory listing and file download.

Refs #4884

Changeset: dd30d9ca77547b89c3489624f33b910398043ac4

comment:296 Changed 7 years ago by Ross Miller

New algorithms for querying and downloading remote files

Algorithms make use of the changes to RemoteJobManager that were just checked in.

Refs #4884

Changeset: 860f785eb42a8e3b2508fea873388a2780512681

comment:297 Changed 7 years ago by Ross Miller

Better validator on ComputeResource properties

The various algorithms that deal with remote job submission all have a property to select the compute cluster to be used. This property's validator now checks against the list of compute resource names found in Facilities.xml. (Previously, the property was just a free-form string.)

Refs #4884

Changeset: 68761b3a032d091cc176f9ee0e78ff40e2ef8740

comment:298 Changed 7 years ago by Ross Miller

Change error message in download algorithm

Had the wrong error message when throwing an exception.

Refs #4884

Changeset: ab6e19ee90aa86e5f3b153909c3fb4e3ea2833b0

comment:299 Changed 7 years ago by Ross Miller

Change RemoteJobManager API

Add upload function to the RemoteJobManager API.

Also changed the HTTP connection initialization code to allow us to make POST requests and remove the 'using namespace std' line.

Refs #4884

Changeset: 7227aaf7e575d2867cf5d990015c5eb3c6aba663

comment:300 Changed 7 years ago by Ross Miller

Add file upload algorithm

Refs #4884

Changeset: 8307e34ad52c309475c4c0a0196fd877fcc9a307

comment:301 Changed 7 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

Cherry-picked from original commit: 9d78735367a204d1058e8963f5c2ddc9ace76a0e

refs #4884

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: ada0de4e6dda82e629f49f80313f689ae77cfe81

comment:302 Changed 7 years ago by Ross Miller

Further work on the remote algorithms project.

Still not anywhere near complete, but some more functionality has been added: cluster info is save to / restored from the properties file and the code to fetch the specified configuration file from a URL works.

Refs #4884

Changeset: 7abd33dd02345dfc0f199db08ebd2f3a9fec044f

comment:303 Changed 7 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: acd8981031e3874f2b23e83df954873dbf4bc570

comment:304 Changed 7 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Cherry-picked from original commit: d4663554859ff1d4d45b968e3bc88e72c8403d77

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: ffe4cd968340a080f57e19535a02d5f9d070977f

comment:305 Changed 7 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: 5afc572121faf22e870538522a148252d4dbff8f

comment:306 Changed 7 years ago by Ross Miller

Minor comment change

refs #4884

Changeset: c251e1d3c250e23bebb9d07d8f044b10ad2959bf

comment:307 Changed 7 years ago by Ross Miller

Small tweaks to the remote job submission code.

Setting the job name (instead of leaving it as the default of STDIN) and picking up the user id from the RemoteJobManager class. Still need to figure out how to handle the password though!! refs #4884

Changeset: 0b57669fe743fab9b26fc94a01e3e1955e92eb19

comment:308 Changed 7 years ago by Ross Miller

Fixed a signed/unsigned comparison warning

refs #4884

Changeset: f6afd875d4bdf12a2d25d6498a07066e92b48a85

comment:309 Changed 7 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Cherry-picked from original commit: 419ddcfd443ac976e10a12fb0611e4169d7d54fe

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 7a7f62981a6b6d636ead8b3b13780473bbf84270

comment:310 Changed 7 years ago by Ross Miller

Minor cleanups in preparation for a code review

Declared default constructor for RemoteJob private (and left unimplemented), removed unnecessary #include's from MantidRemoteAlg.h and commented out a currently unused paramter in RemoteAlgorithmDockWidget::xmlParseServerAttributes()

Refs #4884

Changeset: b90cd95b4e03b1bef3c89439a117ead61d347590

comment:311 Changed 7 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: a236e94deed57146092889d6644bc4ff1fa8dfc1

comment:312 Changed 7 years ago by Ross Miller

Replaced a few more instances of 'algorithm' with 'task'

Refs #4884

Changeset: 27774438815e584f61847096c8ddb4827b60854e

comment:313 Changed 7 years ago by Ross Miller

Filter the job status dialog

Added a slider widget to the job status dialog so that users can filter out jobs older than a specified number of days. NOTE: This is just the GUI bits; the actual filtering code isn't ready yet.

Refs #4884

Changeset: f84fd7c9da8e06ca80b01284d6c84e50667b3a1d

comment:314 Changed 7 years ago by Ross Miller

Adding a JSON parser for Moab Web Services output

Refs #4884

Changeset: 88b6e25ce596e73de214592462ce83d6218596c3

comment:315 Changed 7 years ago by Ross Miller

Update CMakeLists to compile the new JSON parser code

Refs #4884

Changeset: e5bb89d1bd118c46a95985c8cd45e6496ac205f5

comment:316 Changed 7 years ago by Ross Miller

Change Job Status Dialog Box

Changed the job status dialog box to query the MWS server for all the jobs the user has submitted. Uses the new JSON parser code to parse the (rather complicated) output. Also includes a spinbox to let the user set the maximum age of jobs to be displayed.

Refs #4884

Changeset: 8d463724a58044679f9165ab22fda55c1468a853

comment:317 Changed 7 years ago by Ross Miller

Changes to the remote job submission code

Added code to automatically pick a name for the remote job's output file and pass that as a parameter to the job submission URL. This is needed to allow the user to download the results after the remote job has run.

Refs #4884

Changeset: 78b45eb309d59d3fd18c5256465a708ffff08280

comment:318 Changed 7 years ago by Ross Miller

Added ability to download output files from cluster

'Download' buttons in the Job Status dialog will now actually attempt to download the output file and open it in a new workspace. Also added code to verify that there actually is an output file available before adding the download button.

Refs #4884

Changeset: 2d2701d981831292f5880c222883f9fcc5ea6d15

comment:319 Changed 7 years ago by Ross Miller

Only show jobs submitted from MantidPlot in the job status dialog

Uses the 'variables' entry in the submitted JSON to set an entry called "SUBMITTING_APP" when the job is submitted. The jobStatusAll() function checks for this entry and only includes jobs that have it set in its result list.

Refs #4884

Changeset: afa38f736b721b6319437f79224f118bc121aa61

comment:320 Changed 7 years ago by Ross Miller

Disable previous change to job status dialog

Commit 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21 changed the job status dialog box to only show jobs that were actually submitted from MantidPlot. Unfortunately, there's a bug in Moab Web Services (support case #15864) that renders this useless. Once the folks at Adaptive Computing fix their bug, I'll uncomment one line of code to re-enable this feature.

Refs #4884

Changeset: 290edda728eb8628ac31dc2509cee9b5ecc85402

comment:321 Changed 7 years ago by Ross Miller

Several changes to job status dialog (and job submission in general)

1) HTTPS is now enabled and enforced. No unencrypted connections are allowed. (Note: We're accepting the certificate from the server as-is with no validation. That's not really correct, but it's the best we can do for now.) 2) Jobs in the job status dialog are now sorted by their ID's 3) Had to disable the use of QTemporaryFile objects for downloading the output files because the Load algorithm runs in a separate thread and the object was cleaning up its file before the algorithm had a chance to run.

Refs #4884

Changeset: 61eebe0df4dfa902b7feb9ce28d2f62a9199950d

comment:322 Changed 7 years ago by Ross Miller

Preliminary support for Combo Boxes in the job submission dialog

NOTE: NOT FULLY TESTED YET! Refs #4884

Changeset: 3a5347a08d8a056848e0c4b4bc77f819cef04538

comment:323 Changed 7 years ago by Ross Miller

Better error messages if there's a problem parsing the XML config file

Refs #4884

Changeset: 7c62a6c23c2319f60358405d6ce5eea01fd0db41

comment:324 Changed 7 years ago by Ross Miller

Jub submission GUI now supports checkboxes for boolean values

Refs #4884

Changeset: 6b948d3f2dbb078b3ae4c8a6cf1ef0e021e2a66a

comment:325 Changed 7 years ago by Ross Miller

Enable session cookies in MWSRemoteJobManager

Refs #4884

Changeset: 60984d551fb8d0e155a40453a80bda0c7e19f103

comment:326 Changed 7 years ago by Ross Miller

Starting to make GUI changes for the new job submission design

Refs #4884

Changeset: 49bbf93517bb3b5f4756c7db178c3885b393c95e

comment:327 Changed 7 years ago by Ross Miller

More minor GUI changes plus changing a class name

Refs #4884

Changeset: b698eb7af2f09f8b97399065dc451ac69f774d94

comment:328 Changed 7 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: 372f4e0f770a9f1dda6521ad2ba6a3c02fa63220

comment:329 Changed 7 years ago by Ross Miller

Make the remote cluster list a singleton object

This is in preparation for the new design for remote job submission. In the new design, individual algorithms will need a way to access the cluster list, so it can't be stored as part of the dock widget.

Refs #4884

Changeset: f39dd36c9c028700564f1b2ff6710cc71ed9eab2

comment:330 Changed 7 years ago by Ross Miller

Starting to add functions necessary for the new design of the...

...remote job submission code. Specifically, adding functions for starting & stopping transactions.

Refs #4884

Changeset: f64c4c5926c0c4dfaa17cb8e0d864fd4b0300135

comment:331 Changed 7 years ago by Ross Miller

Comment & whitespace cleanups only - no code changes

Refs #4884

Changeset: 3cbd7aacbeef2882c4ee610993715624ad9eba6f

comment:332 Changed 7 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: dcf645ed809e0cccf8cac589b3691c86871b248b

comment:333 Changed 7 years ago by Ross Miller

Add Fermi cluster to Facilities.xml

Adding a description of the Fermi cluster to Facilities.xml

Refs #4884

Changeset: f973f6d3ffd233e61e9b6ca03e52506ed4187965

comment:334 Changed 7 years ago by Ross Miller

Fix typo in a comment

refs #4884

Changeset: d12e0ac874fa3bccc80fa4681f050581b6825d1a

comment:335 Changed 7 years ago by Ross Miller

First cut at using Algorithms for job submission

Not anywhere near complete, but wanted to get it checked in.

Refs #4884

Changeset: 4f5b07b1ae2f98c374a1aa572f3cf2c72331e5d7

comment:336 Changed 7 years ago by Ross Miller

Remove QtMwsRemoteJobManager

Got rid of the QtMwsRemoteJobManager class and made MWSRemoteJobManager the concrete class. Had to add functions to set the username and password to the RemoteJobManager API. (Need to do this so python scripts can use the remote job submission stuff.)

Refs #4884

Changeset: 468d4be1302991d4061efdfc3f6491bdb9bef8d5

comment:337 Changed 7 years ago by Ross Miller

Remove a reference to QtMwsRemoteJobManager.hpp

MantidRemoteCluster.cpp (which itself is going to be deleted soon) still had a reference to QtMwsRemoteJobManager.hpp (which is now gone).

Refs #4884

Changeset: f473785e4c5a396171422b3e411754b9be373cf5

comment:338 Changed 7 years ago by Ross Miller

More work on the Remote Job Submission stuff

1) Added location of mpirun and python executables to Facilities.xml 2) Removed property for executable from submit algorithm (because of #1) 3) Changed the password property to a masked property 4) Added validators to the other properties

Refs #4884

Changeset: 02f52aa864c50883f43ce78211cd533d8c41758f

comment:339 Changed 7 years ago by Ross Miller

Added algorithms for starting & stopping remote transactions

Added algorithms to start and stop remote transactions. Also fixed the exception throwing in the SubmitRemoteJob algorithm.

Refs #4884

Changeset: 30842741d0c1b7fe31579fe8dd43bb484d9e0002

comment:340 Changed 7 years ago by Ross Miller

API changes to RemoteJobManager class

Replaced jobOutputRead() and getJobOutput() with more generic functions for directory listing and file download.

Refs #4884

Changeset: 4712fb3caadf90f1bd0554fc6ca5a6b9241369d5

comment:341 Changed 7 years ago by Ross Miller

New algorithms for querying and downloading remote files

Algorithms make use of the changes to RemoteJobManager that were just checked in.

Refs #4884

Changeset: b77cf6f763f8f654f0072224dc88e968f500bc08

comment:342 Changed 7 years ago by Ross Miller

Better validator on ComputeResource properties

The various algorithms that deal with remote job submission all have a property to select the compute cluster to be used. This property's validator now checks against the list of compute resource names found in Facilities.xml. (Previously, the property was just a free-form string.)

Refs #4884

Changeset: 698f1003926db9bfc20fdb024d2ec9f6d5cf6f31

comment:343 Changed 7 years ago by Ross Miller

Change error message in download algorithm

Had the wrong error message when throwing an exception.

Refs #4884

Changeset: 709443fa4398170bbf5da009a5f681ae2224c148

comment:344 Changed 7 years ago by Ross Miller

Change RemoteJobManager API

Add upload function to the RemoteJobManager API.

Also changed the HTTP connection initialization code to allow us to make POST requests and remove the 'using namespace std' line.

Refs #4884

Changeset: 81493ef6c7cc41c2bc0bee2848e2a1621ad75f89

comment:345 Changed 7 years ago by Ross Miller

Add file upload algorithm

Refs #4884

Changeset: 93f656e82630e0cf5a161f28a4352ad23fec7c4d

comment:346 Changed 7 years ago by Ross Miller

Fixing bugs in the code that generates the command line...

...that's run on the remote cluster.

Refs #4884

Changeset: 299c73363ca6dffcd53d04a83bdade922e4983aa

comment:347 Changed 7 years ago by Ross Miller

Clean up SubmitRemoteJob algorithm

Changed the algorithm to simply call the requested script instead of trying to call mpirun directly. It's assumed the script will call mpirun.

Changed the generated JSON to pass some important values (number of nodes and cores per node) to the process via environment variables.

Refs #4884

Changeset: 453c84f228bc948da03d854d238f2deb578aa3c0

comment:348 Changed 7 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

Cherry-picked from original commit: 9d78735367a204d1058e8963f5c2ddc9ace76a0e

refs #4884

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: f3044d15c7d3c23e67f4f487a500568610ae1d61

comment:349 Changed 7 years ago by Ross Miller

Further work on the remote algorithms project.

Still not anywhere near complete, but some more functionality has been added: cluster info is save to / restored from the properties file and the code to fetch the specified configuration file from a URL works.

Refs #4884

Changeset: 60eaec74420aff2ac49e7595fa8d73583284c31b

comment:350 Changed 7 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: 889ea43875cefa5a9799b533bf9473eeae099628

comment:351 Changed 7 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Cherry-picked from original commit: d4663554859ff1d4d45b968e3bc88e72c8403d77

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 3c4a9dfa46469347b8d97169adee29d64a2f53d4

comment:352 Changed 7 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: ee5d40e01c84d4d8383746e85887803ad320ac2b

comment:353 Changed 7 years ago by Ross Miller

Minor comment change

refs #4884

Changeset: 42621e04493fa76a5f0f00ee2a926c251a876474

comment:354 Changed 7 years ago by Ross Miller

Small tweaks to the remote job submission code.

Setting the job name (instead of leaving it as the default of STDIN) and picking up the user id from the RemoteJobManager class. Still need to figure out how to handle the password though!! refs #4884

Changeset: 55a7ad31de988b8ea2ff6fed73f5ce914f029073

comment:355 Changed 7 years ago by Ross Miller

Fixed a signed/unsigned comparison warning

refs #4884

Changeset: 2ac92a48333ef30339a843f5657245e191263b78

comment:356 Changed 7 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Cherry-picked from original commit: 419ddcfd443ac976e10a12fb0611e4169d7d54fe

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 126dc909b02b3575c627295740bb5407dc478a75

comment:357 Changed 7 years ago by Ross Miller

Minor cleanups in preparation for a code review

Declared default constructor for RemoteJob private (and left unimplemented), removed unnecessary #include's from MantidRemoteAlg.h and commented out a currently unused paramter in RemoteAlgorithmDockWidget::xmlParseServerAttributes()

Refs #4884

Changeset: c105db264888dc38316922f90aaef6b1f05940b6

comment:358 Changed 7 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: 3ab681d4b10741baf3e597f85a1f2138484d8d64

comment:359 Changed 7 years ago by Ross Miller

Replaced a few more instances of 'algorithm' with 'task'

Refs #4884

Changeset: 5d02544eb136d8da5b06647953b9e8661815cab2

comment:360 Changed 7 years ago by Ross Miller

Filter the job status dialog

Added a slider widget to the job status dialog so that users can filter out jobs older than a specified number of days. NOTE: This is just the GUI bits; the actual filtering code isn't ready yet.

Refs #4884

Changeset: 048f7b5b973972b33afc7736e2297794f6163a78

comment:361 Changed 7 years ago by Ross Miller

Adding a JSON parser for Moab Web Services output

Refs #4884

Changeset: 55c724c621d679ea29bbc8b4a4f2fbc8017f00ce

comment:362 Changed 7 years ago by Ross Miller

Update CMakeLists to compile the new JSON parser code

Refs #4884

Changeset: b8274c6ac543e764ef36d4768cde7a4b3985dc02

comment:363 Changed 7 years ago by Ross Miller

Change Job Status Dialog Box

Changed the job status dialog box to query the MWS server for all the jobs the user has submitted. Uses the new JSON parser code to parse the (rather complicated) output. Also includes a spinbox to let the user set the maximum age of jobs to be displayed.

Refs #4884

Changeset: 5668ce9eea59d12dcbe887eca0921ad034d007e8

comment:364 Changed 7 years ago by Ross Miller

Changes to the remote job submission code

Added code to automatically pick a name for the remote job's output file and pass that as a parameter to the job submission URL. This is needed to allow the user to download the results after the remote job has run.

Refs #4884

Changeset: a3efb5491bac927f9bdc89d4fc335bde038ecadf

comment:365 Changed 7 years ago by Ross Miller

Added ability to download output files from cluster

'Download' buttons in the Job Status dialog will now actually attempt to download the output file and open it in a new workspace. Also added code to verify that there actually is an output file available before adding the download button.

Refs #4884

Changeset: cce428e28aaf13e1f11c3632d02b4c92e7636090

comment:366 Changed 7 years ago by Ross Miller

Only show jobs submitted from MantidPlot in the job status dialog

Uses the 'variables' entry in the submitted JSON to set an entry called "SUBMITTING_APP" when the job is submitted. The jobStatusAll() function checks for this entry and only includes jobs that have it set in its result list.

Refs #4884

Changeset: 0641a7112a0f5b3e8db720d8cc127f1ec10d5d6d

comment:367 Changed 7 years ago by Ross Miller

Disable previous change to job status dialog

Commit 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21 changed the job status dialog box to only show jobs that were actually submitted from MantidPlot. Unfortunately, there's a bug in Moab Web Services (support case #15864) that renders this useless. Once the folks at Adaptive Computing fix their bug, I'll uncomment one line of code to re-enable this feature.

Refs #4884

Changeset: beb827e2d37898bb7cfde904c392e3e0c9783bb3

comment:368 Changed 7 years ago by Ross Miller

Several changes to job status dialog (and job submission in general)

1) HTTPS is now enabled and enforced. No unencrypted connections are allowed. (Note: We're accepting the certificate from the server as-is with no validation. That's not really correct, but it's the best we can do for now.) 2) Jobs in the job status dialog are now sorted by their ID's 3) Had to disable the use of QTemporaryFile objects for downloading the output files because the Load algorithm runs in a separate thread and the object was cleaning up its file before the algorithm had a chance to run.

Refs #4884

Changeset: 861efc74ccb7f14680519200b6bffed32f3b6f6d

comment:369 Changed 7 years ago by Ross Miller

Preliminary support for Combo Boxes in the job submission dialog

NOTE: NOT FULLY TESTED YET! Refs #4884

Changeset: 6c79c23d7a3b039b48e7aac4ab4d16e736b2aad2

comment:370 Changed 7 years ago by Ross Miller

Better error messages if there's a problem parsing the XML config file

Refs #4884

Changeset: 9dbc60a47a31a7bd2f3f5c87d24ba17697f0f59a

comment:371 Changed 7 years ago by Ross Miller

Jub submission GUI now supports checkboxes for boolean values

Refs #4884

Changeset: d0b62e3c8bdc92c99578d26e90a94ae3258ef0e9

comment:372 Changed 7 years ago by Ross Miller

Enable session cookies in MWSRemoteJobManager

Refs #4884

Changeset: f0da9dd933a6c255f37bf5153292a806eb21508b

comment:373 Changed 7 years ago by Ross Miller

Starting to make GUI changes for the new job submission design

Refs #4884

Changeset: 005c641e6f0f4da085ec9f4d3315273537ca1e96

comment:374 Changed 7 years ago by Ross Miller

More minor GUI changes plus changing a class name

Refs #4884

Changeset: aff3963ff71a529d0f4d30109d39bea7cabc838e

comment:375 Changed 7 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: 21782bbe0e7a268f3832ff72cd632360a9f31b5a

comment:376 Changed 7 years ago by Ross Miller

Make the remote cluster list a singleton object

This is in preparation for the new design for remote job submission. In the new design, individual algorithms will need a way to access the cluster list, so it can't be stored as part of the dock widget.

Refs #4884

Changeset: 7eebaca5b5843041ade23af7ea6217a3388242cb

comment:377 Changed 7 years ago by Ross Miller

Starting to add functions necessary for the new design of the...

...remote job submission code. Specifically, adding functions for starting & stopping transactions.

Refs #4884

Changeset: 1a0e7f61e1f94443cd690ab4d17313dd2f2e138c

comment:378 Changed 7 years ago by Ross Miller

Comment & whitespace cleanups only - no code changes

Refs #4884

Changeset: 69243ecb365628e617ae69d50b4e161e62885f2e

comment:379 Changed 7 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: 1025d01c47992a3059fab58169b3d7fff727a78f

comment:380 Changed 7 years ago by Ross Miller

Add Fermi cluster to Facilities.xml

Adding a description of the Fermi cluster to Facilities.xml

Refs #4884

Changeset: 3ebee527fe79ad5f6f66ed2adf8d10bd03d8ed29

comment:381 Changed 7 years ago by Ross Miller

Fix typo in a comment

refs #4884

Changeset: 8aec5169e82b5daa15d883dc9b11bb53a683c12d

comment:382 Changed 7 years ago by Ross Miller

First cut at using Algorithms for job submission

Not anywhere near complete, but wanted to get it checked in.

Refs #4884

Changeset: 9dbda10f1bb69251b1087678eb851d5ca3688d2a

comment:383 Changed 7 years ago by Ross Miller

Remove QtMwsRemoteJobManager

Got rid of the QtMwsRemoteJobManager class and made MWSRemoteJobManager the concrete class. Had to add functions to set the username and password to the RemoteJobManager API. (Need to do this so python scripts can use the remote job submission stuff.)

Refs #4884

Changeset: 8f72194740d39ca004c42461de3dababfd36e416

comment:384 Changed 7 years ago by Ross Miller

Remove a reference to QtMwsRemoteJobManager.hpp

MantidRemoteCluster.cpp (which itself is going to be deleted soon) still had a reference to QtMwsRemoteJobManager.hpp (which is now gone).

Refs #4884

Changeset: 7d9cbb9b43009dd4d4dd043f5f98372f9d3c105c

comment:385 Changed 7 years ago by Ross Miller

More work on the Remote Job Submission stuff

1) Added location of mpirun and python executables to Facilities.xml 2) Removed property for executable from submit algorithm (because of #1) 3) Changed the password property to a masked property 4) Added validators to the other properties

Refs #4884

Changeset: b475f32288533782c69af2a9440344cdda01b8eb

comment:386 Changed 7 years ago by Ross Miller

Added algorithms for starting & stopping remote transactions

Added algorithms to start and stop remote transactions. Also fixed the exception throwing in the SubmitRemoteJob algorithm.

Refs #4884

Changeset: f33468e919d757332d54297516001d0d7ab20a23

comment:387 Changed 7 years ago by Ross Miller

API changes to RemoteJobManager class

Replaced jobOutputRead() and getJobOutput() with more generic functions for directory listing and file download.

Refs #4884

Changeset: fc556fa1660cb338e5dd0cbbf4b8252815b299f1

comment:388 Changed 7 years ago by Ross Miller

New algorithms for querying and downloading remote files

Algorithms make use of the changes to RemoteJobManager that were just checked in.

Refs #4884

Changeset: 67cb95f540aa1552548ccea0c3b15fd61704952b

comment:389 Changed 7 years ago by Ross Miller

Better validator on ComputeResource properties

The various algorithms that deal with remote job submission all have a property to select the compute cluster to be used. This property's validator now checks against the list of compute resource names found in Facilities.xml. (Previously, the property was just a free-form string.)

Refs #4884

Changeset: 320291b60f13f539b0bd647a55c3544385566fc9

comment:390 Changed 7 years ago by Ross Miller

Change error message in download algorithm

Had the wrong error message when throwing an exception.

Refs #4884

Changeset: 649b3ce6d3c2d465c39abde864e4ff3ba2aa238f

comment:391 Changed 7 years ago by Ross Miller

Change RemoteJobManager API

Add upload function to the RemoteJobManager API.

Also changed the HTTP connection initialization code to allow us to make POST requests and remove the 'using namespace std' line.

Refs #4884

Changeset: 334d93d95a3be58c76360a148968e75427988377

comment:392 Changed 7 years ago by Ross Miller

Add file upload algorithm

Refs #4884

Changeset: f5cbc202ade184f72f7e257bd33993300dac804c

comment:393 Changed 7 years ago by Ross Miller

Fixing bugs in the code that generates the command line...

...that's run on the remote cluster.

Refs #4884

Changeset: a2c48989f170306caf778683b405e13c79fcd59d

comment:394 Changed 7 years ago by Ross Miller

Clean up SubmitRemoteJob algorithm

Changed the algorithm to simply call the requested script instead of trying to call mpirun directly. It's assumed the script will call mpirun.

Changed the generated JSON to pass some important values (number of nodes and cores per node) to the process via environment variables.

Refs #4884

Changeset: 162a8a701dff16f50b44b654c7e923fcb031ef33

comment:395 Changed 7 years ago by Ross Miller

Removed last of the remote job submission GUI code...

...from the MantidPlot tree. Most of it has been supercedeed by the *Remote* algorithms and the parts that hadn't been were out of date anyway. About the only thing we're likely to need in the future is a dialog box for querying jobs on the remote system and it will probably be easier to create a new one than to try and salvage the old code.

Refs #4884

Changeset: 4eccc8daa19924adc8ba659c2aac9acec6b95dba

comment:396 Changed 7 years ago by Ross Miller

Add CMake flag to enable/disable compiling

Add a flag to CMake to enable/disable building the entire remote job submission package. Flag defaults to 'disable'.

Note: A few things are still compiled even if the flag is disabled: the RemoteJobManagerFactory and a few minor changes to FacilityInfo.

Refs #4884

Changeset: 31ff385c4f8be16594c086272552e1c7593a5ded

comment:397 Changed 7 years ago by Ross Miller

Update algorithm categories

The various algorithms for handling remote job submission had categories left as 'User Algorithm' (from createAlg.py). Change this to 'Remote'.

Refs #4884

Changeset: 7b3154f60a1967b62a0137e87ab79d5c849d6dba

comment:398 Changed 7 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

Cherry-picked from original commit: 9d78735367a204d1058e8963f5c2ddc9ace76a0e

refs #4884

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 4ddbdff99390bf766c234e4921fc85baf5a9269a

comment:399 Changed 7 years ago by Ross Miller

Further work on the remote algorithms project.

Still not anywhere near complete, but some more functionality has been added: cluster info is save to / restored from the properties file and the code to fetch the specified configuration file from a URL works.

Refs #4884

Changeset: 3ffcbf2409c122b440639147668073a1705cc2b6

comment:400 Changed 7 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: 50c37411ae7b968c11454512cad94ca489a5f85c

comment:401 Changed 7 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Cherry-picked from original commit: d4663554859ff1d4d45b968e3bc88e72c8403d77

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: c005d09a433a4d010e613198106461ce81da9a6d

comment:402 Changed 7 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: d7dc42f00eec6cd29e90788451d8df689f18c0b8

comment:403 Changed 7 years ago by Ross Miller

Minor comment change

refs #4884

Changeset: 783b574f6e9c2da3472a60b5636b0ee50aeaf493

comment:404 Changed 7 years ago by Ross Miller

Small tweaks to the remote job submission code.

Setting the job name (instead of leaving it as the default of STDIN) and picking up the user id from the RemoteJobManager class. Still need to figure out how to handle the password though!! refs #4884

Changeset: 5e772b83c6dc1f0b8f8d2473aedf36bba20f570b

comment:405 Changed 7 years ago by Ross Miller

Fixed a signed/unsigned comparison warning

refs #4884

Changeset: 274f0e2b1ba032858907bc2b7f094a6f5098f08c

comment:406 Changed 7 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Cherry-picked from original commit: 419ddcfd443ac976e10a12fb0611e4169d7d54fe

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 70d1908d1b00c52a74daa5d52ca8e6e13760cae6

comment:407 Changed 7 years ago by Ross Miller

Minor cleanups in preparation for a code review

Declared default constructor for RemoteJob private (and left unimplemented), removed unnecessary #include's from MantidRemoteAlg.h and commented out a currently unused paramter in RemoteAlgorithmDockWidget::xmlParseServerAttributes()

Refs #4884

Changeset: fddd5433261e90d10811a6c2f16ae34e23c6fcd2

comment:408 Changed 7 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: b03f9268b6701f38c263856001d35cf7d997344b

comment:409 Changed 7 years ago by Ross Miller

Replaced a few more instances of 'algorithm' with 'task'

Refs #4884

Changeset: 9a20f5de6fb932a645cc547aa1c8c49c6c21bb9a

comment:410 Changed 7 years ago by Ross Miller

Filter the job status dialog

Added a slider widget to the job status dialog so that users can filter out jobs older than a specified number of days. NOTE: This is just the GUI bits; the actual filtering code isn't ready yet.

Refs #4884

Changeset: 7b3b568e67356843936b4c4f0168d6acf61d1c12

comment:411 Changed 7 years ago by Ross Miller

Adding a JSON parser for Moab Web Services output

Refs #4884

Changeset: 3568e56425e0015167fe3f57251595d5cff213b1

comment:412 Changed 7 years ago by Ross Miller

Update CMakeLists to compile the new JSON parser code

Refs #4884

Changeset: 15eaf2c9bdf7b36b806cc2e7bec00b77dc6cd737

comment:413 Changed 7 years ago by Ross Miller

Change Job Status Dialog Box

Changed the job status dialog box to query the MWS server for all the jobs the user has submitted. Uses the new JSON parser code to parse the (rather complicated) output. Also includes a spinbox to let the user set the maximum age of jobs to be displayed.

Refs #4884

Changeset: c5f85e4446308195efb0ad90bca8c5111d604b16

comment:414 Changed 7 years ago by Ross Miller

Changes to the remote job submission code

Added code to automatically pick a name for the remote job's output file and pass that as a parameter to the job submission URL. This is needed to allow the user to download the results after the remote job has run.

Refs #4884

Changeset: 9b03653a135c61eea3fee571dab044baf982573a

comment:415 Changed 7 years ago by Ross Miller

Added ability to download output files from cluster

'Download' buttons in the Job Status dialog will now actually attempt to download the output file and open it in a new workspace. Also added code to verify that there actually is an output file available before adding the download button.

Refs #4884

Changeset: 8d34372a5ae4fa6fdf78742e07d44be0729c9fc8

comment:416 Changed 7 years ago by Ross Miller

Only show jobs submitted from MantidPlot in the job status dialog

Uses the 'variables' entry in the submitted JSON to set an entry called "SUBMITTING_APP" when the job is submitted. The jobStatusAll() function checks for this entry and only includes jobs that have it set in its result list.

Refs #4884

Changeset: 869d42dfe05933141b1a797ede755878b27424d3

comment:417 Changed 7 years ago by Ross Miller

Disable previous change to job status dialog

Commit 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21 changed the job status dialog box to only show jobs that were actually submitted from MantidPlot. Unfortunately, there's a bug in Moab Web Services (support case #15864) that renders this useless. Once the folks at Adaptive Computing fix their bug, I'll uncomment one line of code to re-enable this feature.

Refs #4884

Changeset: 5f6e396ebb1545f4b0bf2423aec6d08d7763cbef

comment:418 Changed 7 years ago by Ross Miller

Several changes to job status dialog (and job submission in general)

1) HTTPS is now enabled and enforced. No unencrypted connections are allowed. (Note: We're accepting the certificate from the server as-is with no validation. That's not really correct, but it's the best we can do for now.) 2) Jobs in the job status dialog are now sorted by their ID's 3) Had to disable the use of QTemporaryFile objects for downloading the output files because the Load algorithm runs in a separate thread and the object was cleaning up its file before the algorithm had a chance to run.

Refs #4884

Changeset: 641734c23bfe7d4f8bb3149efce7f1ea7d3458de

comment:419 Changed 7 years ago by Ross Miller

Preliminary support for Combo Boxes in the job submission dialog

NOTE: NOT FULLY TESTED YET! Refs #4884

Changeset: 0711ae53df232b2a9a29d69365059b64a775b5cb

comment:420 Changed 7 years ago by Ross Miller

Better error messages if there's a problem parsing the XML config file

Refs #4884

Changeset: 4d4b8f6819b48565fd7bdd5a363a74670a0204d5

comment:421 Changed 7 years ago by Ross Miller

Jub submission GUI now supports checkboxes for boolean values

Refs #4884

Changeset: 48cbaa6fed6ad890882671931ca0dc8f1f4b70f0

comment:422 Changed 7 years ago by Ross Miller

Enable session cookies in MWSRemoteJobManager

Refs #4884

Changeset: 2495f74a74ec847a2baf96b424e0f0bc14ca71a5

comment:423 Changed 7 years ago by Ross Miller

Starting to make GUI changes for the new job submission design

Refs #4884

Changeset: 9d45e9a346c0136c055997e138a380c47795dce2

comment:424 Changed 7 years ago by Ross Miller

More minor GUI changes plus changing a class name

Refs #4884

Changeset: 14785e131ad1199e7926d09915e5460681cd587d

comment:425 Changed 7 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: 40d2b7b41b67b397608e13d7caeb756a337347f7

comment:426 Changed 7 years ago by Ross Miller

Make the remote cluster list a singleton object

This is in preparation for the new design for remote job submission. In the new design, individual algorithms will need a way to access the cluster list, so it can't be stored as part of the dock widget.

Refs #4884

Changeset: 713be803af3127d0e1149e6a29bf4c61035bdda8

comment:427 Changed 7 years ago by Ross Miller

Starting to add functions necessary for the new design of the...

...remote job submission code. Specifically, adding functions for starting & stopping transactions.

Refs #4884

Changeset: ffc4b8b8b64a23455d91fc2a66f010efef516e51

comment:428 Changed 7 years ago by Ross Miller

Comment & whitespace cleanups only - no code changes

Refs #4884

Changeset: 58b426b87c1c08315c11fb359db4ba8a32fcb80b

comment:429 Changed 7 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: bbc44f2fc8375986c00231d1285f8de834277cfb

comment:430 Changed 7 years ago by Ross Miller

Add Fermi cluster to Facilities.xml

Adding a description of the Fermi cluster to Facilities.xml

Refs #4884

Changeset: c6cd12914726b93ea8a39f4d75620c3a32a1af97

comment:431 Changed 7 years ago by Ross Miller

Fix typo in a comment

refs #4884

Changeset: 3df1950c8b4f1a366c91fbc88c9093d17fc3d325

comment:432 Changed 7 years ago by Ross Miller

First cut at using Algorithms for job submission

Not anywhere near complete, but wanted to get it checked in.

Refs #4884

Changeset: 2c62ba4bf39e6f79331a4a34b945ba41ca0a4836

comment:433 Changed 7 years ago by Ross Miller

Remove QtMwsRemoteJobManager

Got rid of the QtMwsRemoteJobManager class and made MWSRemoteJobManager the concrete class. Had to add functions to set the username and password to the RemoteJobManager API. (Need to do this so python scripts can use the remote job submission stuff.)

Refs #4884

Changeset: f984fc9353a42de31395c00be5a8d74afab61339

comment:434 Changed 7 years ago by Ross Miller

Remove a reference to QtMwsRemoteJobManager.hpp

MantidRemoteCluster.cpp (which itself is going to be deleted soon) still had a reference to QtMwsRemoteJobManager.hpp (which is now gone).

Refs #4884

Changeset: 861598f7e0c1163942df63a01d0205bcda8e7266

comment:435 Changed 7 years ago by Ross Miller

More work on the Remote Job Submission stuff

1) Added location of mpirun and python executables to Facilities.xml 2) Removed property for executable from submit algorithm (because of #1) 3) Changed the password property to a masked property 4) Added validators to the other properties

Refs #4884

Changeset: 9e4e1814ac2bc005e8300e66b07360f0bac6780a

comment:436 Changed 7 years ago by Ross Miller

Added algorithms for starting & stopping remote transactions

Added algorithms to start and stop remote transactions. Also fixed the exception throwing in the SubmitRemoteJob algorithm.

Refs #4884

Changeset: 4a7eeb6fed8a53a247994f17824e03b2213c7449

comment:437 Changed 7 years ago by Ross Miller

API changes to RemoteJobManager class

Replaced jobOutputRead() and getJobOutput() with more generic functions for directory listing and file download.

Refs #4884

Changeset: 744c3d4fa66324ed9c5d872abc8a7a4d0a96166b

comment:438 Changed 7 years ago by Ross Miller

New algorithms for querying and downloading remote files

Algorithms make use of the changes to RemoteJobManager that were just checked in.

Refs #4884

Changeset: fe7907d329c63cdad1d23655b6aebec94dec4e3c

comment:439 Changed 7 years ago by Ross Miller

Better validator on ComputeResource properties

The various algorithms that deal with remote job submission all have a property to select the compute cluster to be used. This property's validator now checks against the list of compute resource names found in Facilities.xml. (Previously, the property was just a free-form string.)

Refs #4884

Changeset: 4fb4a96b276400c8b5b2a56fc88b35f707abeb80

comment:440 Changed 7 years ago by Ross Miller

Change error message in download algorithm

Had the wrong error message when throwing an exception.

Refs #4884

Changeset: 13e0747f101f608c68c3217fe06356916ba2de0f

comment:441 Changed 7 years ago by Ross Miller

Change RemoteJobManager API

Add upload function to the RemoteJobManager API.

Also changed the HTTP connection initialization code to allow us to make POST requests and remove the 'using namespace std' line.

Refs #4884

Changeset: 8e2c570851ca17f854c883a737ca7767ef6bf36c

comment:442 Changed 7 years ago by Ross Miller

Add file upload algorithm

Refs #4884

Changeset: 289dd864d71fe70e8a440a4ce0a57240c483424e

comment:443 Changed 7 years ago by Ross Miller

Fixing bugs in the code that generates the command line...

...that's run on the remote cluster.

Refs #4884

Changeset: 30b5413ea470c6bb3d52a35019f4d3c13034c9d6

comment:444 Changed 7 years ago by Ross Miller

Clean up SubmitRemoteJob algorithm

Changed the algorithm to simply call the requested script instead of trying to call mpirun directly. It's assumed the script will call mpirun.

Changed the generated JSON to pass some important values (number of nodes and cores per node) to the process via environment variables.

Refs #4884

Changeset: 531a230b5bb701e740bf82460f5d9a276eb0975b

comment:445 Changed 7 years ago by Ross Miller

Removed last of the remote job submission GUI code...

...from the MantidPlot tree. Most of it has been supercedeed by the *Remote* algorithms and the parts that hadn't been were out of date anyway. About the only thing we're likely to need in the future is a dialog box for querying jobs on the remote system and it will probably be easier to create a new one than to try and salvage the old code.

Refs #4884

Changeset: 72d310a812051b554297df278fb0475c50279dd1

comment:446 Changed 7 years ago by Ross Miller

Add CMake flag to enable/disable compiling

Add a flag to CMake to enable/disable building the entire remote job submission package. Flag defaults to 'disable'.

Note: A few things are still compiled even if the flag is disabled: the RemoteJobManagerFactory and a few minor changes to FacilityInfo.

Refs #4884

Changeset: 2efa9aa506b60b513aab8fff4c877dff33558265

comment:447 Changed 7 years ago by Ross Miller

Update algorithm categories

The various algorithms for handling remote job submission had categories left as 'User Algorithm' (from createAlg.py). Change this to 'Remote'.

Refs #4884

Changeset: 57c98adcf614f0ac944747112af13d03c8706e8d

comment:448 Changed 7 years ago by Ross Miller

First cut at GUI elements for the remote algorithsm work

Cherry-picked from original commit: 9d78735367a204d1058e8963f5c2ddc9ace76a0e

refs #4884

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 4ddbdff99390bf766c234e4921fc85baf5a9269a

comment:449 Changed 7 years ago by Ross Miller

Further work on the remote algorithms project.

Still not anywhere near complete, but some more functionality has been added: cluster info is save to / restored from the properties file and the code to fetch the specified configuration file from a URL works.

Refs #4884

Changeset: 3ffcbf2409c122b440639147668073a1705cc2b6

comment:450 Changed 7 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Changeset: 50c37411ae7b968c11454512cad94ca489a5f85c

comment:451 Changed 7 years ago by Ross Miller

Moving files related to remote job submisison to their own directory.

Refs #4884

Cherry-picked from original commit: d4663554859ff1d4d45b968e3bc88e72c8403d77

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: c005d09a433a4d010e613198106461ce81da9a6d

comment:452 Changed 7 years ago by Ross Miller

Now have enough done to actually submit a remote job to Chadwick

refs #4884

Changeset: d7dc42f00eec6cd29e90788451d8df689f18c0b8

comment:453 Changed 7 years ago by Ross Miller

Minor comment change

refs #4884

Changeset: 783b574f6e9c2da3472a60b5636b0ee50aeaf493

comment:454 Changed 7 years ago by Ross Miller

Small tweaks to the remote job submission code.

Setting the job name (instead of leaving it as the default of STDIN) and picking up the user id from the RemoteJobManager class. Still need to figure out how to handle the password though!! refs #4884

Changeset: 5e772b83c6dc1f0b8f8d2473aedf36bba20f570b

comment:455 Changed 7 years ago by Ross Miller

Fixed a signed/unsigned comparison warning

refs #4884

Changeset: 274f0e2b1ba032858907bc2b7f094a6f5098f08c

comment:456 Changed 7 years ago by Ross Miller

Added job status display

Added code to display the status of previously submiitted remote jobs. Also added code to ask for a password the first time the user needs to send anything to Moab Web Services. (The password is held in memory after that.)

refs #4884

Cherry-picked from original commit: 419ddcfd443ac976e10a12fb0611e4169d7d54fe

Conflicts:

Code/Mantid/MantidPlot/CMakeLists.txt

Changeset: 70d1908d1b00c52a74daa5d52ca8e6e13760cae6

comment:457 Changed 7 years ago by Ross Miller

Minor cleanups in preparation for a code review

Declared default constructor for RemoteJob private (and left unimplemented), removed unnecessary #include's from MantidRemoteAlg.h and commented out a currently unused paramter in RemoteAlgorithmDockWidget::xmlParseServerAttributes()

Refs #4884

Changeset: fddd5433261e90d10811a6c2f16ae34e23c6fcd2

comment:458 Changed 7 years ago by Ross Miller

Changed all instances of "Remote Algorithm" to "Remote Task"

Refs #4884

Changeset: b03f9268b6701f38c263856001d35cf7d997344b

comment:459 Changed 7 years ago by Ross Miller

Replaced a few more instances of 'algorithm' with 'task'

Refs #4884

Changeset: 9a20f5de6fb932a645cc547aa1c8c49c6c21bb9a

comment:460 Changed 7 years ago by Ross Miller

Filter the job status dialog

Added a slider widget to the job status dialog so that users can filter out jobs older than a specified number of days. NOTE: This is just the GUI bits; the actual filtering code isn't ready yet.

Refs #4884

Changeset: 7b3b568e67356843936b4c4f0168d6acf61d1c12

comment:461 Changed 7 years ago by Ross Miller

Adding a JSON parser for Moab Web Services output

Refs #4884

Changeset: 3568e56425e0015167fe3f57251595d5cff213b1

comment:462 Changed 7 years ago by Ross Miller

Update CMakeLists to compile the new JSON parser code

Refs #4884

Changeset: 15eaf2c9bdf7b36b806cc2e7bec00b77dc6cd737

comment:463 Changed 7 years ago by Ross Miller

Change Job Status Dialog Box

Changed the job status dialog box to query the MWS server for all the jobs the user has submitted. Uses the new JSON parser code to parse the (rather complicated) output. Also includes a spinbox to let the user set the maximum age of jobs to be displayed.

Refs #4884

Changeset: c5f85e4446308195efb0ad90bca8c5111d604b16

comment:464 Changed 7 years ago by Ross Miller

Changes to the remote job submission code

Added code to automatically pick a name for the remote job's output file and pass that as a parameter to the job submission URL. This is needed to allow the user to download the results after the remote job has run.

Refs #4884

Changeset: 9b03653a135c61eea3fee571dab044baf982573a

comment:465 Changed 7 years ago by Ross Miller

Added ability to download output files from cluster

'Download' buttons in the Job Status dialog will now actually attempt to download the output file and open it in a new workspace. Also added code to verify that there actually is an output file available before adding the download button.

Refs #4884

Changeset: 8d34372a5ae4fa6fdf78742e07d44be0729c9fc8

comment:466 Changed 7 years ago by Ross Miller

Only show jobs submitted from MantidPlot in the job status dialog

Uses the 'variables' entry in the submitted JSON to set an entry called "SUBMITTING_APP" when the job is submitted. The jobStatusAll() function checks for this entry and only includes jobs that have it set in its result list.

Refs #4884

Changeset: 869d42dfe05933141b1a797ede755878b27424d3

comment:467 Changed 7 years ago by Ross Miller

Disable previous change to job status dialog

Commit 4af5da7d6f3b6da5b94c3d0a3f0831ee03c58c21 changed the job status dialog box to only show jobs that were actually submitted from MantidPlot. Unfortunately, there's a bug in Moab Web Services (support case #15864) that renders this useless. Once the folks at Adaptive Computing fix their bug, I'll uncomment one line of code to re-enable this feature.

Refs #4884

Changeset: 5f6e396ebb1545f4b0bf2423aec6d08d7763cbef

comment:468 Changed 7 years ago by Ross Miller

Several changes to job status dialog (and job submission in general)

1) HTTPS is now enabled and enforced. No unencrypted connections are allowed. (Note: We're accepting the certificate from the server as-is with no validation. That's not really correct, but it's the best we can do for now.) 2) Jobs in the job status dialog are now sorted by their ID's 3) Had to disable the use of QTemporaryFile objects for downloading the output files because the Load algorithm runs in a separate thread and the object was cleaning up its file before the algorithm had a chance to run.

Refs #4884

Changeset: 641734c23bfe7d4f8bb3149efce7f1ea7d3458de

comment:469 Changed 7 years ago by Ross Miller

Preliminary support for Combo Boxes in the job submission dialog

NOTE: NOT FULLY TESTED YET! Refs #4884

Changeset: 0711ae53df232b2a9a29d69365059b64a775b5cb

comment:470 Changed 7 years ago by Ross Miller

Better error messages if there's a problem parsing the XML config file

Refs #4884

Changeset: 4d4b8f6819b48565fd7bdd5a363a74670a0204d5

comment:471 Changed 7 years ago by Ross Miller

Jub submission GUI now supports checkboxes for boolean values

Refs #4884

Changeset: 48cbaa6fed6ad890882671931ca0dc8f1f4b70f0

comment:472 Changed 7 years ago by Ross Miller

Enable session cookies in MWSRemoteJobManager

Refs #4884

Changeset: 2495f74a74ec847a2baf96b424e0f0bc14ca71a5

comment:473 Changed 7 years ago by Ross Miller

Starting to make GUI changes for the new job submission design

Refs #4884

Changeset: 9d45e9a346c0136c055997e138a380c47795dce2

comment:474 Changed 7 years ago by Ross Miller

More minor GUI changes plus changing a class name

Refs #4884

Changeset: 14785e131ad1199e7926d09915e5460681cd587d

comment:475 Changed 7 years ago by Ross Miller

Changed the name of 2 files to match the new class name

Refs #4884

Changeset: 40d2b7b41b67b397608e13d7caeb756a337347f7

comment:476 Changed 7 years ago by Ross Miller

Make the remote cluster list a singleton object

This is in preparation for the new design for remote job submission. In the new design, individual algorithms will need a way to access the cluster list, so it can't be stored as part of the dock widget.

Refs #4884

Changeset: 713be803af3127d0e1149e6a29bf4c61035bdda8

comment:477 Changed 7 years ago by Ross Miller

Starting to add functions necessary for the new design of the...

...remote job submission code. Specifically, adding functions for starting & stopping transactions.

Refs #4884

Changeset: ffc4b8b8b64a23455d91fc2a66f010efef516e51

comment:478 Changed 7 years ago by Ross Miller

Comment & whitespace cleanups only - no code changes

Refs #4884

Changeset: 58b426b87c1c08315c11fb359db4ba8a32fcb80b

comment:479 Changed 7 years ago by Ross Miller

Changing the RemoteJobManager infrastructure

Instances of RemoteJobManager are created from XML elements in Facilities.xml (rather than being entered by the user and stored in the user's properties file).

Refs #4884

Changeset: bbc44f2fc8375986c00231d1285f8de834277cfb

comment:480 Changed 7 years ago by Ross Miller

Add Fermi cluster to Facilities.xml

Adding a description of the Fermi cluster to Facilities.xml

Refs #4884

Changeset: c6cd12914726b93ea8a39f4d75620c3a32a1af97

comment:481 Changed 7 years ago by Ross Miller

Fix typo in a comment

refs #4884

Changeset: 3df1950c8b4f1a366c91fbc88c9093d17fc3d325

comment:482 Changed 7 years ago by Ross Miller

First cut at using Algorithms for job submission

Not anywhere near complete, but wanted to get it checked in.

Refs #4884

Changeset: 2c62ba4bf39e6f79331a4a34b945ba41ca0a4836

comment:483 Changed 7 years ago by Ross Miller

Remove QtMwsRemoteJobManager

Got rid of the QtMwsRemoteJobManager class and made MWSRemoteJobManager the concrete class. Had to add functions to set the username and password to the RemoteJobManager API. (Need to do this so python scripts can use the remote job submission stuff.)

Refs #4884

Changeset: f984fc9353a42de31395c00be5a8d74afab61339

comment:484 Changed 7 years ago by Ross Miller

Remove a reference to QtMwsRemoteJobManager.hpp

MantidRemoteCluster.cpp (which itself is going to be deleted soon) still had a reference to QtMwsRemoteJobManager.hpp (which is now gone).

Refs #4884

Changeset: 861598f7e0c1163942df63a01d0205bcda8e7266

comment:485 Changed 7 years ago by Ross Miller

More work on the Remote Job Submission stuff

1) Added location of mpirun and python executables to Facilities.xml 2) Removed property for executable from submit algorithm (because of #1) 3) Changed the password property to a masked property 4) Added validators to the other properties

Refs #4884

Changeset: 9e4e1814ac2bc005e8300e66b07360f0bac6780a

comment:486 Changed 7 years ago by Ross Miller

Added algorithms for starting & stopping remote transactions

Added algorithms to start and stop remote transactions. Also fixed the exception throwing in the SubmitRemoteJob algorithm.

Refs #4884

Changeset: 4a7eeb6fed8a53a247994f17824e03b2213c7449

comment:487 Changed 7 years ago by Ross Miller

API changes to RemoteJobManager class

Replaced jobOutputRead() and getJobOutput() with more generic functions for directory listing and file download.

Refs #4884

Changeset: 744c3d4fa66324ed9c5d872abc8a7a4d0a96166b

comment:488 Changed 7 years ago by Ross Miller

New algorithms for querying and downloading remote files

Algorithms make use of the changes to RemoteJobManager that were just checked in.

Refs #4884

Changeset: fe7907d329c63cdad1d23655b6aebec94dec4e3c

comment:489 Changed 7 years ago by Ross Miller

Better validator on ComputeResource properties

The various algorithms that deal with remote job submission all have a property to select the compute cluster to be used. This property's validator now checks against the list of compute resource names found in Facilities.xml. (Previously, the property was just a free-form string.)

Refs #4884

Changeset: 4fb4a96b276400c8b5b2a56fc88b35f707abeb80

comment:490 Changed 7 years ago by Ross Miller

Change error message in download algorithm

Had the wrong error message when throwing an exception.

Refs #4884

Changeset: 13e0747f101f608c68c3217fe06356916ba2de0f

comment:491 Changed 7 years ago by Ross Miller

Change RemoteJobManager API

Add upload function to the RemoteJobManager API.

Also changed the HTTP connection initialization code to allow us to make POST requests and remove the 'using namespace std' line.

Refs #4884

Changeset: 8e2c570851ca17f854c883a737ca7767ef6bf36c

comment:492 Changed 7 years ago by Ross Miller

Add file upload algorithm

Refs #4884

Changeset: 289dd864d71fe70e8a440a4ce0a57240c483424e

comment:493 Changed 7 years ago by Ross Miller

Fixing bugs in the code that generates the command line...

...that's run on the remote cluster.

Refs #4884

Changeset: 30b5413ea470c6bb3d52a35019f4d3c13034c9d6

comment:494 Changed 7 years ago by Ross Miller

Clean up SubmitRemoteJob algorithm

Changed the algorithm to simply call the requested script instead of trying to call mpirun directly. It's assumed the script will call mpirun.

Changed the generated JSON to pass some important values (number of nodes and cores per node) to the process via environment variables.

Refs #4884

Changeset: 531a230b5bb701e740bf82460f5d9a276eb0975b

comment:495 Changed 7 years ago by Ross Miller

Removed last of the remote job submission GUI code...

...from the MantidPlot tree. Most of it has been supercedeed by the *Remote* algorithms and the parts that hadn't been were out of date anyway. About the only thing we're likely to need in the future is a dialog box for querying jobs on the remote system and it will probably be easier to create a new one than to try and salvage the old code.

Refs #4884

Changeset: 72d310a812051b554297df278fb0475c50279dd1

comment:496 Changed 7 years ago by Ross Miller

Add CMake flag to enable/disable compiling

Add a flag to CMake to enable/disable building the entire remote job submission package. Flag defaults to 'disable'.

Note: A few things are still compiled even if the flag is disabled: the RemoteJobManagerFactory and a few minor changes to FacilityInfo.

Refs #4884

Changeset: 2efa9aa506b60b513aab8fff4c877dff33558265

comment:497 Changed 7 years ago by Ross Miller

Update algorithm categories

The various algorithms for handling remote job submission had categories left as 'User Algorithm' (from createAlg.py). Change this to 'Remote'.

Refs #4884

Changeset: 57c98adcf614f0ac944747112af13d03c8706e8d

comment:498 Changed 7 years ago by Mathieu Doucet

  • Blocking 6924 added

comment:499 Changed 7 years ago by Ross Miller

Merge branch 'RACleanup' into develop into 6856_ConvertToDiffractionMDWS_v2

comment:500 Changed 7 years ago by Ross Miller

Merge branch 'RACleanup' into develop into 6856_ConvertToDiffractionMDWS_v2

comment:501 Changed 7 years ago by Ross Miller

Moved the RemoteAlgorithms classes into their own namespace

Refs #4884

Changeset: 797dc5cd845f6bd31c55423e631b8f77f05918df

comment:502 Changed 7 years ago by Ross Miller

  • Status changed from new to accepted

comment:503 Changed 7 years ago by Ross Miller

  • Status changed from accepted to verify
  • Resolution set to fixed

Closing this ticket. The code is by no means complete, but it's got enough functionality that others want to start using it. The CMake flag to enable this code still defaults to off, so most people won't ever see it. Testing it on the build servers requires using the special builds under the "Remote Jobs Builds" tab.

comment:504 Changed 7 years ago by Ross Miller

  • Status changed from verify to reopened
  • Resolution fixed deleted

Re-opening the ticket because of build failures on MacOS

comment:505 Changed 7 years ago by Ross Miller

Build cleanups for MacOS

std::ifstream and std::ofstream constructors require const char * rather than strings

Refs #4884

Changeset: 2c0aab6c0fa70ddb0f1df5c7ca49cb21c3de4847

comment:506 Changed 7 years ago by Ross Miller

Changed the order of a couple of libraries in CMakeLists

I'm hoping this will fix the 'undefined symbol' erro on the Mac build

Refs #4884

Changeset: b034b8a30dc539190a71f9358cf3e8ce0563e9f6

comment:507 Changed 7 years ago by Ross Miller

  • Status changed from reopened to accepted

comment:508 Changed 7 years ago by Ross Miller

  • Status changed from accepted to verify
  • Resolution set to fixed

Builds are working on Linux, Mac & Windows now, so closing the ticket. Again, code is not yet complete, but others want to start using it. The CMake flag that enables this chunk of code still defaults to off.

comment:509 Changed 7 years ago by Mathieu Doucet

  • Status changed from verify to verifying
  • Milestone changed from Unassigned to Release 2.6
  • Tester set to Mathieu Doucet

comment:510 Changed 7 years ago by Mathieu Doucet

  • Status changed from verifying to closed

The submission works.

Note: The job querying part will be done in another ticket

comment:511 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 5731

Note: See TracTickets for help on using tickets.