Ticket #7636 (closed: fixed)

Opened 7 years ago

Last modified 5 years ago

[ICAT] Publish algorithm

Reported by: Jay Rainey Owned by: Jay Rainey
Priority: major Milestone: Release 3.1
Component: Framework Keywords: ICAT4,publish
Cc: Blocked By: #8581, #8599
Blocking: #8537 Tester: Roman Tolchenov

Description (last modified by Jay Rainey) (diff)

The goal is to allow the users of Mantid to publish their modified data to ICAT.

Change History

comment:1 Changed 7 years ago by Jay Rainey

  • Milestone changed from Release 3.0 to Backlog

comment:2 Changed 7 years ago by Jay Rainey

  • Milestone changed from Backlog to Release 3.1

comment:3 Changed 7 years ago by Jay Rainey

  • Summary changed from Publish results to ICAT to [ICAT] Publish results

comment:4 Changed 7 years ago by Jay Rainey

  • Blocking 8537 added

comment:5 Changed 7 years ago by Jay Rainey

  • Description modified (diff)

comment:6 Changed 7 years ago by Jay Rainey

  • Description modified (diff)

comment:7 Changed 7 years ago by Jay Rainey

  • Keywords ICAT4,publish added; ICAT4 removed

comment:8 Changed 7 years ago by Jay Rainey

  • Description modified (diff)

comment:9 Changed 7 years ago by Jay Rainey

  • Status changed from new to inprogress

Removed unused method and forward declaration. Refs #7636.

Changeset: 28f3c5325466aef1135092f035ac2c10f74d4231

comment:10 Changed 7 years ago by Jay Rainey

Create the method outline for the uploadURL. Refs #7636.

Changeset: ed4db5a5d0884037d2887a7c5f968765bb3053f3

comment:11 Changed 7 years ago by Jay Rainey

Generate upload URL in ICat4. Refs #7636.

  • In order to publish (upload/PUT) we need to know the dataset the user wants to PUT that file to. This is "calculated" in the getDatasetIdFromFileName method, which uses the related filename (extracted from path or workspace in the algorithm) & obtains the related datasetID.

Changeset: 110546238a4693f5172874d9384c5c6022fc9db1

comment:12 Changed 7 years ago by Jay Rainey

Added CatalogPublish algorithm. Refs #7636.

Changeset: cc3a7eccfdb7991dd771e6a8cc849cd91f3c9f3b

comment:13 Changed 7 years ago by Jay Rainey

Corrected indent in cmakelist. Refs #7636.

Changeset: 095226c24539ffaad0bc79b4d2717f7d28d7db35

comment:14 Changed 7 years ago by Jay Rainey

Added correct ICAT call to method. Refs #7636.

Changeset: 96686ca348b6e92e92856e0a9e94b2218156d733

comment:16 Changed 7 years ago by Jay Rainey

Fix compiler warning. Refs #7636.

Changeset: ff6edb140d78ad629308c1f29c9aa3e040766f9b

comment:17 Changed 7 years ago by Jay Rainey

Removed unused exception. Refs #7636.

Changeset: 70fd22cd19e614cedf45355916220b39cda48244

comment:18 Changed 7 years ago by Jay Rainey

Published a commit to test it on windows. The changes were for testing purposes, and won't be in the branch when tested.

Last edited 7 years ago by Jay Rainey (previous) (diff)

comment:19 Changed 7 years ago by Jay Rainey

Set encoding type. Upload now works correctly. Refs #7636.

  • Added IsDataFile method to detect if the file being uploaded is datafile, in order to set the correct mode.

Changeset: dc0628641b8e1602d57f03b27bc1de131bcfedcb

comment:20 Changed 7 years ago by Jay Rainey

  • Blocked By 8581 added

comment:21 Changed 7 years ago by Jay Rainey

Merge remote-tracking branch 'origin/master' into feature/7636_icat_publish. Refs #7636.

Conflicts:

  • Code/Mantid/Framework/ICat/inc/MantidICat/ICat4/ICat4Catalog.h

Changeset: e28c983e7beb1afee3408ce8d14924a56efe9ceb

Last edited 7 years ago by Jay Rainey (previous) (diff)

comment:22 Changed 7 years ago by Jay Rainey

  • Blocked By 8599 added

comment:23 Changed 7 years ago by Jay Rainey

Helper method to obtain file name from a given path. Refs #7636.

Changeset: 08f41c17ae873fb50cd96cb54f13dac9910e0996

comment:24 Changed 7 years ago by Jay Rainey

Added workspace property. Refs #7636.

  • Removed unused headers.
  • Forgot to remove extractFileName code from exec.

Changeset: 4caa02b08d6c619504ff9a1da54925fef13b3692

comment:25 Changed 7 years ago by Jay Rainey

Encapsulate publish functionality. Refs #7636.

Changeset: a1a97ea8e4390cbbb92a9680e694e8040c8e4d9e

comment:26 Changed 7 years ago by Jay Rainey

Upload of datafile works correctly. Refs #7636.

Changeset: e0912e3fd456e6ef0eab18aa87f6182390480d0b

comment:27 Changed 7 years ago by Jay Rainey

Let saveNexus take care of workspace groups. Refs #7636.

Changeset: 494cbe294739e18a8fd89f88bbf92c5e391af871

comment:28 Changed 7 years ago by Jay Rainey

Merge remote-tracking branch 'origin/master' into feature/7636_icat_publish. Refs #7636.

Conflicts:

  • Code/Mantid/Framework/ICat/inc/MantidICat/ICat4/ICat4Catalog.h

Changeset: 9cf9ba0c9b69bc52b3f00ebe77ff648245f6f25f

Last edited 7 years ago by Jay Rainey (previous) (diff)

comment:29 Changed 7 years ago by Jay Rainey

Added setICATProxySettings to publish method. Refs #7636.

Changeset: 798963a3c6201b2990d2b097313da004da3ff7af

comment:30 Changed 7 years ago by Jay Rainey

Add publishing of workspace support. Refs #7636.

  • If user has selected a workspace, we will save it to their default directory then upload that file to the IDS.

Changeset: 53779aed414375639c154bd046e390ffd32fecd8

comment:31 Changed 7 years ago by Jay Rainey

  • Description modified (diff)

comment:32 Changed 7 years ago by Jay Rainey

Add const to ICatalog and related classes. Refs #7636.

Changeset: c4b7daccd3a9c929c94bf83247a1797afb0c325d

comment:33 Changed 7 years ago by Jay Rainey

Publish workspace history if workspace was selected. Refs #7636.

Changeset: c56101c3410154bec80360c236028269af7015f0

comment:34 Changed 7 years ago by Jay Rainey

Remove hard-coded download URL. Refs #7636.

Changeset: cf52c9a36892dd7aa7e14d1f4f4714fab6de3bae

comment:35 Changed 7 years ago by Jay Rainey

  • Status changed from inprogress to verify
  • Resolution set to fixed
Last edited 7 years ago by Jay Rainey (previous) (diff)

comment:36 Changed 7 years ago by Jay Rainey

  • Summary changed from [ICAT] Publish results to [ICAT] Publish results algorithm

comment:37 Changed 7 years ago by Jay Rainey

  • Summary changed from [ICAT] Publish results algorithm to [ICAT] Publish algorithm

comment:38 Changed 7 years ago by Nick Draper

  • Status changed from verify to verifying
  • Tester set to Nick Draper

comment:39 Changed 7 years ago by Nick Draper

  • Status changed from verifying to reopened
  • Resolution fixed deleted

I ran into a few problems testing this on windows:

  1. The parameters of the algorithm need to be improved / harmonised [filepath -> FileName CreateFileName -> NameInCatalog? (default use existing filename)]
  2. I was refused to upload an ILL file, we should add a InvestigationNumber property (defualt extract from file).
  3. Trying to search for a file I uploaded failed, the instrument field drop down was empty, no instruments to choose from, and entering the uploaded filename and searching gave the response "You have not input any terms to search for."

comment:40 Changed 7 years ago by Jay Rainey

  • Status changed from reopened to inprogress
Last edited 7 years ago by Jay Rainey (previous) (diff)

comment:41 Changed 7 years ago by Jay Rainey

Last edited 7 years ago by Jay Rainey (previous) (diff)

comment:42 Changed 7 years ago by Jay Rainey

Last edited 7 years ago by Jay Rainey (previous) (diff)

comment:43 Changed 7 years ago by Jay Rainey

Last edited 7 years ago by Jay Rainey (previous) (diff)

comment:44 Changed 7 years ago by Jay Rainey

Use SSLManager to make use of same context. Refs #7636.

Changeset: 5cd084e7682f0f5883c401433cd6a7910e8fd3fd

comment:45 Changed 7 years ago by Jay Rainey

Harmonise publish algorithm parameters. Refs #7636.

Changeset: 0f29f76421f2dfc2a3a951ae97ffef64705172d6

comment:46 Changed 7 years ago by Jay Rainey

Remove bad merge code from develop. Refs #7636.

Changeset: dd1b033adf220fe351151a7e557d4eaf5160a29a

comment:47 Changed 7 years ago by Jay Rainey

Allow user to upload any file if they specify inves num. Refs #7636.

Changeset: 8c06f260008b0aa52f7a050052d8540ecb769f12

comment:48 Changed 7 years ago by Jay Rainey

Use filename including extension. Refs #7636.

Changeset: fc2fefe23373cd64b5a83e00b6511b066bd48a37

comment:49 Changed 7 years ago by Jay Rainey

  • Status changed from inprogress to verify
  • Resolution set to fixed
Last edited 7 years ago by Jay Rainey (previous) (diff)

comment:50 Changed 7 years ago by Jay Rainey

Remove bad merge code from develop. Refs #7636.

Changeset: 38d87cdaffe603cf9574048a31ffbb9a59a5c703

comment:51 Changed 7 years ago by Samuel Jackson

  • Status changed from verify to verifying
  • Tester changed from Nick Draper to Samuel Jackson

comment:52 Changed 7 years ago by Samuel Jackson

  • Status changed from verifying to verify
  • Tester Samuel Jackson deleted

comment:53 Changed 7 years ago by Jay Rainey

  • Status changed from verify to reopened
  • Resolution fixed deleted

Fixing a "hang" when the user selects a file (with an invalid filename) & does not specify the investigation number.

comment:54 Changed 7 years ago by Russell Taylor

There's an MSVC compiler warning concerning the use of Poco::SharedPtr that needs addressing.

comment:55 Changed 7 years ago by Jay Rainey

  • Status changed from reopened to inprogress

Use the correct datafile name for generated py script. Refs #7636.

  • Use the datafile name to obtain the dataset ID unless the user wants to provide the specific run number.

Changeset: c73f45a8dfbe5cdc325046b07ec03b652fc204f1

comment:56 Changed 7 years ago by Jay Rainey

Improved error message. Refs #7636.

Changeset: 6be9870ca8fd79dd9d2ddbb2947fe4a89bf4ad9a

comment:57 Changed 7 years ago by Jay Rainey

Handle errors returned by the IDS. Refs #7636.

  • It's not possible to throw a runtime_error as poco catches it, then throws another I/O error instead! This is related to a bug in POCO 1.4 (http://sourceforge.net/p/poco/bugs/403/). Instead, I have logged the message and informed the user of the possible problems.
  • This "error handling" will be improved on the GUI, but needs to exist as is for when user's write scripts against this algorithm.

Changeset: 277579a8fa2f03277237bde09d63743320efd557

comment:58 Changed 7 years ago by Jay Rainey

User must be logged into ICAT to publish. Refs #7636.

Changeset: decfc706be40942f9138ecd777cba15f84714d07

comment:59 Changed 7 years ago by Jay Rainey

Use raw pointer instead of shared. Refs #7636.

Changeset: 4f753192a00462ad12b94ed879bfb5ee9f4f2c84

comment:60 Changed 7 years ago by Jay Rainey

  • Status changed from inprogress to verify
  • Resolution set to fixed

Note

  • You can only publish data to investigations of which you are an investigator on.
  • This ticket needs to be tested on multiple platforms.
  • You can only test this ticket if you have results in "my data". (E.g. you are an investigator in investigations.)

To test

Change Facilities.xml

You will need to change the soapendpoint and externaldownload in Facilities.xml to:

  soapendpoint     -> "https://icatdev.isis.cclrc.ac.uk/ICATService/ICAT"
  externaldownload -> "https://isisoxfordvmsrv.isis.cclrc.ac.uk/ids/"

Publishing a file (with a valid name)

  1. Log into the catalog.
  2. From the Algorithms menu input CatalogPublish and press enter.
  3. Select a file to upload that has a valid name (E.G. It must have the instrument name follow by a valid rb number that is in your investigations). For example, GEM62839.txt. To obtain a valid filename you could search My data' in the ICAT GUI, select an investigation then download a file.
  4. Verify that the file has been uploaded to the IDS by searching for the file in the archives. I recommend entering the name of the file in the Datafile name: field and selecting the correct instrument for a quicker search response.

Publishing a workspace (single and grouped)

  1. Load a workspace into Mantid.
  2. Log into the catalog.
  3. Select the CatalogPublish algorithm as before.
  4. This time select the workspace from the dropdown (the filepath should be empty, otherwise you will get an error).
  5. Click Run. The file should now have been uploaded to the IDS.
  6. Verify that the file has been uploaded by finding it in the archives. (Perhaps also loading it into Mantid & comparing it against the workspace you uploaded.)
  7. A feature of the publishing algorithm is to also upload the history of the workspace in a .py file. This should also be verified. (It will have the same name as you have named the file, but with a .py extension.)

Note: You must test uploading of both a single AND a grouped workspace.

Testing the publishing parameters

  1. You should provide a name to save the file in the catalog via the NameInCatalog box. By default, the filename of the file you are uploading is used.
  2. Attempt to publish with an invalid name, investigation ID, and both.
  3. Try to break it!

In addition, verify that there are no compiler warnings (specifically on Windows), that the code is clean, and that I have addressed Nick's comment.

Last edited 7 years ago by Jay Rainey (previous) (diff)

comment:61 Changed 7 years ago by Jay Rainey

Include necessary header & use shared pointer. Refs #7636.

Changeset: 2a4a3303c1a4546946b2b0a2badec10249a53d16

comment:62 Changed 7 years ago by Roman Tolchenov

  • Status changed from verify to verifying
  • Tester set to Roman Tolchenov

comment:63 Changed 7 years ago by Jay Rainey

Better error handling. Refs #7636.

Changeset: c9cc8157b7f6872c0cfa06a31a99b205ab1a3052

comment:64 Changed 7 years ago by Roman Tolchenov

  • Status changed from verifying to closed

Merge remote-tracking branch 'origin/feature/7636_icat_publish'

Full changeset: 1e501ca0726ffbe9314b6fbd66d28829a6705b3d

comment:65 Changed 7 years ago by Jay Rainey

Testing publishing on windows. Refs #7636.

Changeset: 3ffc02a234a5846be6901f43669a7d5a130d10a5

comment:66 Changed 7 years ago by Jay Rainey

Merge branch 'feature/7636_icat_publish'. Refs #7636.

Conflicts:

  • Code/Mantid/Framework/ICat/src/ICat4/ICat4Catalog.cpp
  • Code/Mantid/instrument/Facilities.xml

Changeset: 10f8dbf60fbdd44136c2b03e006b3eb1c339e168

comment:67 Changed 7 years ago by Jay Rainey

Use SSLManager to make use of same context. Refs #7636.

Changeset: b61d40675911ec274c22c39709333fd860dd9d55

comment:68 Changed 7 years ago by Jay Rainey

Remove bad merge code from develop. Refs #7636.

Changeset: 38d87cdaffe603cf9574048a31ffbb9a59a5c703

comment:69 Changed 7 years ago by Nick Draper

  • Component changed from Framework to User Interface

comment:70 Changed 7 years ago by Nick Draper

  • Component changed from User Interface to Framework

comment:71 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 8481

Note: See TracTickets for help on using tickets.