Ticket #7636 (closed: fixed)
[ICAT] Publish algorithm
Reported by: | Jay Rainey | Owned by: | Jay Rainey |
---|---|---|---|
Priority: | major | Milestone: | Release 3.1 |
Component: | Framework | Keywords: | ICAT4,publish |
Cc: | Blocked By: | #8581, #8599 | |
Blocking: | #8537 | Tester: | Roman Tolchenov |
Description (last modified by Jay Rainey) (diff)
The goal is to allow the users of Mantid to publish their modified data to ICAT.
Change History
comment:3 Changed 7 years ago by Jay Rainey
- Summary changed from Publish results to ICAT to [ICAT] Publish results
comment:9 Changed 7 years ago by Jay Rainey
- Status changed from new to inprogress
Removed unused method and forward declaration. Refs #7636.
Changeset: 28f3c5325466aef1135092f035ac2c10f74d4231
comment:10 Changed 7 years ago by Jay Rainey
Create the method outline for the uploadURL. Refs #7636.
Changeset: ed4db5a5d0884037d2887a7c5f968765bb3053f3
comment:11 Changed 7 years ago by Jay Rainey
Generate upload URL in ICat4. Refs #7636.
- In order to publish (upload/PUT) we need to know the dataset the user wants to PUT that file to. This is "calculated" in the getDatasetIdFromFileName method, which uses the related filename (extracted from path or workspace in the algorithm) & obtains the related datasetID.
Changeset: 110546238a4693f5172874d9384c5c6022fc9db1
comment:12 Changed 7 years ago by Jay Rainey
Added CatalogPublish algorithm. Refs #7636.
Changeset: cc3a7eccfdb7991dd771e6a8cc849cd91f3c9f3b
comment:13 Changed 7 years ago by Jay Rainey
Corrected indent in cmakelist. Refs #7636.
Changeset: 095226c24539ffaad0bc79b4d2717f7d28d7db35
comment:14 Changed 7 years ago by Jay Rainey
Added correct ICAT call to method. Refs #7636.
Changeset: 96686ca348b6e92e92856e0a9e94b2218156d733
comment:15 Changed 7 years ago by Martyn Gigg
There is a compiler warning that needs addressing: http://download.mantidproject.org/jenkins/view/Develop%20Clean%20Builds%20and%20Tests/job/is_clean_rhel6_develop/292/warnings20Result/
comment:16 Changed 7 years ago by Jay Rainey
Fix compiler warning. Refs #7636.
Changeset: ff6edb140d78ad629308c1f29c9aa3e040766f9b
comment:17 Changed 7 years ago by Jay Rainey
Removed unused exception. Refs #7636.
Changeset: 70fd22cd19e614cedf45355916220b39cda48244
comment:18 Changed 7 years ago by Jay Rainey
Published a commit to test it on windows. The changes were for testing purposes, and won't be in the branch when tested.
comment:19 Changed 7 years ago by Jay Rainey
Set encoding type. Upload now works correctly. Refs #7636.
- Added IsDataFile method to detect if the file being uploaded is datafile, in order to set the correct mode.
Changeset: dc0628641b8e1602d57f03b27bc1de131bcfedcb
comment:21 Changed 7 years ago by Jay Rainey
Merge remote-tracking branch 'origin/master' into feature/7636_icat_publish. Refs #7636.
Conflicts:
- Code/Mantid/Framework/ICat/inc/MantidICat/ICat4/ICat4Catalog.h
Changeset: e28c983e7beb1afee3408ce8d14924a56efe9ceb
comment:23 Changed 7 years ago by Jay Rainey
Helper method to obtain file name from a given path. Refs #7636.
Changeset: 08f41c17ae873fb50cd96cb54f13dac9910e0996
comment:24 Changed 7 years ago by Jay Rainey
Added workspace property. Refs #7636.
- Removed unused headers.
- Forgot to remove extractFileName code from exec.
Changeset: 4caa02b08d6c619504ff9a1da54925fef13b3692
comment:25 Changed 7 years ago by Jay Rainey
Encapsulate publish functionality. Refs #7636.
Changeset: a1a97ea8e4390cbbb92a9680e694e8040c8e4d9e
comment:26 Changed 7 years ago by Jay Rainey
Upload of datafile works correctly. Refs #7636.
Changeset: e0912e3fd456e6ef0eab18aa87f6182390480d0b
comment:27 Changed 7 years ago by Jay Rainey
Let saveNexus take care of workspace groups. Refs #7636.
Changeset: 494cbe294739e18a8fd89f88bbf92c5e391af871
comment:28 Changed 7 years ago by Jay Rainey
Merge remote-tracking branch 'origin/master' into feature/7636_icat_publish. Refs #7636.
Conflicts:
- Code/Mantid/Framework/ICat/inc/MantidICat/ICat4/ICat4Catalog.h
Changeset: 9cf9ba0c9b69bc52b3f00ebe77ff648245f6f25f
comment:29 Changed 7 years ago by Jay Rainey
Added setICATProxySettings to publish method. Refs #7636.
Changeset: 798963a3c6201b2990d2b097313da004da3ff7af
comment:30 Changed 7 years ago by Jay Rainey
Add publishing of workspace support. Refs #7636.
- If user has selected a workspace, we will save it to their default directory then upload that file to the IDS.
Changeset: 53779aed414375639c154bd046e390ffd32fecd8
comment:32 Changed 7 years ago by Jay Rainey
Add const to ICatalog and related classes. Refs #7636.
Changeset: c4b7daccd3a9c929c94bf83247a1797afb0c325d
comment:33 Changed 7 years ago by Jay Rainey
Publish workspace history if workspace was selected. Refs #7636.
Changeset: c56101c3410154bec80360c236028269af7015f0
comment:34 Changed 7 years ago by Jay Rainey
Remove hard-coded download URL. Refs #7636.
Changeset: cf52c9a36892dd7aa7e14d1f4f4714fab6de3bae
comment:35 Changed 7 years ago by Jay Rainey
- Status changed from inprogress to verify
- Resolution set to fixed
comment:36 Changed 7 years ago by Jay Rainey
- Summary changed from [ICAT] Publish results to [ICAT] Publish results algorithm
comment:37 Changed 7 years ago by Jay Rainey
- Summary changed from [ICAT] Publish results algorithm to [ICAT] Publish algorithm
comment:38 Changed 7 years ago by Nick Draper
- Status changed from verify to verifying
- Tester set to Nick Draper
comment:39 Changed 7 years ago by Nick Draper
- Status changed from verifying to reopened
- Resolution fixed deleted
I ran into a few problems testing this on windows:
- The parameters of the algorithm need to be improved / harmonised [filepath -> FileName CreateFileName -> NameInCatalog? (default use existing filename)]
- I was refused to upload an ILL file, we should add a InvestigationNumber property (defualt extract from file).
- Trying to search for a file I uploaded failed, the instrument field drop down was empty, no instruments to choose from, and entering the uploaded filename and searching gave the response "You have not input any terms to search for."
comment:41 Changed 7 years ago by Jay Rainey
comment:42 Changed 7 years ago by Jay Rainey
comment:43 Changed 7 years ago by Jay Rainey
comment:44 Changed 7 years ago by Jay Rainey
Use SSLManager to make use of same context. Refs #7636.
Changeset: 5cd084e7682f0f5883c401433cd6a7910e8fd3fd
comment:45 Changed 7 years ago by Jay Rainey
Harmonise publish algorithm parameters. Refs #7636.
Changeset: 0f29f76421f2dfc2a3a951ae97ffef64705172d6
comment:46 Changed 7 years ago by Jay Rainey
Remove bad merge code from develop. Refs #7636.
Changeset: dd1b033adf220fe351151a7e557d4eaf5160a29a
comment:47 Changed 7 years ago by Jay Rainey
Allow user to upload any file if they specify inves num. Refs #7636.
Changeset: 8c06f260008b0aa52f7a050052d8540ecb769f12
comment:48 Changed 7 years ago by Jay Rainey
Use filename including extension. Refs #7636.
Changeset: fc2fefe23373cd64b5a83e00b6511b066bd48a37
comment:49 Changed 7 years ago by Jay Rainey
- Status changed from inprogress to verify
- Resolution set to fixed
comment:50 Changed 7 years ago by Jay Rainey
Remove bad merge code from develop. Refs #7636.
Changeset: 38d87cdaffe603cf9574048a31ffbb9a59a5c703
comment:51 Changed 7 years ago by Samuel Jackson
- Status changed from verify to verifying
- Tester changed from Nick Draper to Samuel Jackson
comment:52 Changed 7 years ago by Samuel Jackson
- Status changed from verifying to verify
- Tester Samuel Jackson deleted
comment:53 Changed 7 years ago by Jay Rainey
- Status changed from verify to reopened
- Resolution fixed deleted
Fixing a "hang" when the user selects a file (with an invalid filename) & does not specify the investigation number.
comment:54 Changed 7 years ago by Russell Taylor
There's an MSVC compiler warning concerning the use of Poco::SharedPtr that needs addressing.
comment:55 Changed 7 years ago by Jay Rainey
- Status changed from reopened to inprogress
Use the correct datafile name for generated py script. Refs #7636.
- Use the datafile name to obtain the dataset ID unless the user wants to provide the specific run number.
Changeset: c73f45a8dfbe5cdc325046b07ec03b652fc204f1
comment:56 Changed 7 years ago by Jay Rainey
Improved error message. Refs #7636.
Changeset: 6be9870ca8fd79dd9d2ddbb2947fe4a89bf4ad9a
comment:57 Changed 7 years ago by Jay Rainey
Handle errors returned by the IDS. Refs #7636.
- It's not possible to throw a runtime_error as poco catches it, then throws another I/O error instead! This is related to a bug in POCO 1.4 (http://sourceforge.net/p/poco/bugs/403/). Instead, I have logged the message and informed the user of the possible problems.
- This "error handling" will be improved on the GUI, but needs to exist as is for when user's write scripts against this algorithm.
Changeset: 277579a8fa2f03277237bde09d63743320efd557
comment:58 Changed 7 years ago by Jay Rainey
User must be logged into ICAT to publish. Refs #7636.
Changeset: decfc706be40942f9138ecd777cba15f84714d07
comment:59 Changed 7 years ago by Jay Rainey
Use raw pointer instead of shared. Refs #7636.
Changeset: 4f753192a00462ad12b94ed879bfb5ee9f4f2c84
comment:60 Changed 7 years ago by Jay Rainey
- Status changed from inprogress to verify
- Resolution set to fixed
Note
- You can only publish data to investigations of which you are an investigator on.
- This ticket needs to be tested on multiple platforms.
- You can only test this ticket if you have results in "my data". (E.g. you are an investigator in investigations.)
To test
Change Facilities.xml
You will need to change the soapendpoint and externaldownload in Facilities.xml to:
soapendpoint -> "https://icatdev.isis.cclrc.ac.uk/ICATService/ICAT" externaldownload -> "https://isisoxfordvmsrv.isis.cclrc.ac.uk/ids/"
Publishing a file (with a valid name)
- Log into the catalog.
- From the Algorithms menu input CatalogPublish and press enter.
- Select a file to upload that has a valid name (E.G. It must have the instrument name follow by a valid rb number that is in your investigations). For example, GEM62839.txt. To obtain a valid filename you could search My data' in the ICAT GUI, select an investigation then download a file.
- Verify that the file has been uploaded to the IDS by searching for the file in the archives. I recommend entering the name of the file in the Datafile name: field and selecting the correct instrument for a quicker search response.
Publishing a workspace (single and grouped)
- Load a workspace into Mantid.
- Log into the catalog.
- Select the CatalogPublish algorithm as before.
- This time select the workspace from the dropdown (the filepath should be empty, otherwise you will get an error).
- Click Run. The file should now have been uploaded to the IDS.
- Verify that the file has been uploaded by finding it in the archives. (Perhaps also loading it into Mantid & comparing it against the workspace you uploaded.)
- A feature of the publishing algorithm is to also upload the history of the workspace in a .py file. This should also be verified. (It will have the same name as you have named the file, but with a .py extension.)
Note: You must test uploading of both a single AND a grouped workspace.
Testing the publishing parameters
- You should provide a name to save the file in the catalog via the NameInCatalog box. By default, the filename of the file you are uploading is used.
- Attempt to publish with an invalid name, investigation ID, and both.
- Try to break it!
In addition, verify that there are no compiler warnings (specifically on Windows), that the code is clean, and that I have addressed Nick's comment.
comment:61 Changed 7 years ago by Jay Rainey
Include necessary header & use shared pointer. Refs #7636.
Changeset: 2a4a3303c1a4546946b2b0a2badec10249a53d16
comment:62 Changed 7 years ago by Roman Tolchenov
- Status changed from verify to verifying
- Tester set to Roman Tolchenov
comment:63 Changed 7 years ago by Jay Rainey
Better error handling. Refs #7636.
Changeset: c9cc8157b7f6872c0cfa06a31a99b205ab1a3052
comment:64 Changed 7 years ago by Roman Tolchenov
- Status changed from verifying to closed
Merge remote-tracking branch 'origin/feature/7636_icat_publish'
Full changeset: 1e501ca0726ffbe9314b6fbd66d28829a6705b3d
comment:65 Changed 7 years ago by Jay Rainey
Testing publishing on windows. Refs #7636.
Changeset: 3ffc02a234a5846be6901f43669a7d5a130d10a5
comment:66 Changed 7 years ago by Jay Rainey
Merge branch 'feature/7636_icat_publish'. Refs #7636.
Conflicts:
- Code/Mantid/Framework/ICat/src/ICat4/ICat4Catalog.cpp
- Code/Mantid/instrument/Facilities.xml
Changeset: 10f8dbf60fbdd44136c2b03e006b3eb1c339e168
comment:67 Changed 7 years ago by Jay Rainey
Use SSLManager to make use of same context. Refs #7636.
Changeset: b61d40675911ec274c22c39709333fd860dd9d55
comment:68 Changed 7 years ago by Jay Rainey
Remove bad merge code from develop. Refs #7636.
Changeset: 38d87cdaffe603cf9574048a31ffbb9a59a5c703
comment:71 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 8481