Add DOI to icat publishing

[Jay Rainey]

This will need to be discussed further with Tom.

[Jay Rainey]

Having spoken with Tom, the following was concluded:

1. The DOI API is developed by the SCP, which is unlikely to be released by the code-freeze.
2. Generating a DOI for a datafile is optional.
3. When a user publishes a datafile they are automatically made private. Tom suggested that I should implement a checkbox to provide the user with an optional to make data private, and by default make it public.

[Nick Draper]

Bulk move of tickets out of triage (new) to assigned at the introduction of the triage state

[Jay Rainey]

To add DOI generation support for publishing I need to:

1. Generate gSoap files against the DOI generator end-point, which are:
   • https://data4.isis.stfc.ac.uk/doi/DOIService?wsdl Develop
   • https://sig-03.esc.rl.ac.uk:8181/doi/DOIService?wsdl Production
2. Add the end-point (dev for now until changes below are made) to Facilities.xml.
3. Update ICatalogInfoService to include a new method (generateDOIForDatafile) that makes use of the gSoap methods above (specifically: registerDatafileDOI) and returns the generated DOI (to output on LOG).
4. Add and implement the functionality of the new method above in ICat4Catalog.
   • This includes obtaining the end-point from Facilities.xml t oaccomdate other facilities using th
5. Add two new properties to CatalogPublish:
   • InvestigationID, "",`The database row ID of the investigation that you want to publish to. (Note: this is required to generate a DOI.)
   • GenerateDOI, true, "Generates a DOI for the datafile being published. Once a DOI is generated the datafile will be made public."
6. Add logic to exec in CatalogPublish to make use of the properties and call generateDOIForDatafile from the catalog. Note: if a file fails to upload (e.g. already exists etc...) a DOI must not be generated.
7. Update CatalogPublishDialog to have a checkbox for GenerateDOI. InvestigationID will be hidden on the GUI and obtained from the table generated getPublishInvestigations (once #9223 is complete).
   • The generateDOI (make public) checkbox on the GUI should be checked by default.
   • To prevent users accidentally generating DOIs that they may not have wanted to, a QMessageBox should be generated IF the user has checked generateDOI to confirm this choice.

[Jay Rainey]

Added DOI endpoint to Facilities.xml. Refs #8538.

Changeset: ecea7bceea7a5f6e89825f0885f658635ed819ee

[Jay Rainey]

Added tests for doi in CatalogInfo. Refs #8538.

• Also added missing testing for external download url (IDS).

Changeset: 866ff0fed7beaef1d9e1f6584e2d5b14f300ca66

[Jay Rainey]

Added doiEndPoint method to CatalogInfo. Refs #8538.

Changeset: bb5a48f9e62af3561912ef98cc66bbe1d45ec288

[Jay Rainey]

Added gSoap generated code for endpoint. Refs #8538.

Changeset: b0e3af73c6cd51e3c382130ac92a90df010ce970

[Jay Rainey]

Add registerDatafileDOI to interface & inherited classes. Refs #8538.

Changeset: 78723d52a70a65d93f2a6a8e4ef05e2605a23f55

[Jay Rainey]

Refactor CatalogPublish error handling. Refs #8538.

Changeset: f968514b0e5cf84b9d5e7cd0ce9092e08c230d61

[Jay Rainey]

Merge 'origin/master' into feature/8538_icat_publish_dois. Refs #8538.

Changeset: 85af5037527c89ad144f3d6cfe51c90a808160d8

[Jay Rainey]

Add properties and logic to catalogPublish algorithm. Refs #8538.

Changeset: c8c27c2b692a8ba8b278f3e4d31111d11c0c6503

[Jay Rainey]

Use getRef for ease of use. Refs #8538.

Changeset: edf82a7693b662701d215009200d6ad14eb01e17

[Jay Rainey]

Rename parameter on registerDOI. Refs #8538.

Changeset: df97d12c6a40d9c615707c988504f210e3864a0e

[Jay Rainey]

Make SSLContext method generic via templates. Refs #8538.

• This means we can make use of this for ICATProxy and the DOIProxy objects.

Changeset: 81fabd7e4e92859852b785a5ece696a0a3128114

[Jay Rainey]

Make throwErrorMessage generic via templates. Refs #8538.

Changeset: e228227527335ad62470427209d9453a663b0492

[Jay Rainey]

Renamed method. Refs #8538.

Changeset: ff5abaf1de9c4ad67afa5a58c089c8eb488b6e9e

[Jay Rainey]

Added DOI catalog implementation. Refs #8538.

Changeset: 3b19191643d0454738959cddb4aae4ef78d5bbcb

[Jay Rainey]

Added DatabaseID to investigation workspace. Refs #8538.

• Removed investigationID check as it's handled on the DOI server.

Changeset: bc1936c94715a789ca0c6d201a0d60c5fde99262

[Jay Rainey]

Use specified namespace instead of ns1. Refs #8538.

Changeset: d0ce080430140ee01bfe1988cb13033bd21b19dd

[Jay Rainey]

Updated readme to include namespace generation. Refs #8538.

• The -n option overrides the ns namespace prefix. This is required to prevent warnings if several end-points have been generated without a namespace set.

Changeset: a1a7698e8babbd2ea1855bdc3993e553f5eab424

[Jay Rainey]

Add make public checkbox to publish GUI. Refs #8538.

Changeset: ae0b0628c9ff6cf693e37113d96de3bf62743f71

[Jay Rainey]

Set DatabaseID property of Publish algorithm via GUI. Refs #8538.

Changeset: b18ec0d8d086f29259c9974815f5e75c3face193

[Jay Rainey]

Fix include. Refs #8538.

Changeset: dc3d16e311644f5530b43c0c98230b07d3f7992d

[Russell Taylor]

The system test that validates of the Facilities file against its schema is failing because "doiendpoint" does not feature in the schema.

[Martyn Gigg]

There are quite a lot of new cppcheck ​warnings for the generated code. I think we will need to add another EXCLUDE for the new gsoap-generated code in Code/Mantid/Build/CMake/CppCheckSetup.cmake.

[Jay Rainey]

Add CPPEXCLUDE for DOI gSoap generated code. Refs #8538.

Changeset: 0cf95f1071239e1699d0dbb1e3d42fa6205b5a07

[Jay Rainey]

Added doiendpoint to FacilitiesScheme. Refs #8538.

Changeset: 629055c43d37ea88ecc779772496f6f2a2190c0a

[Jay Rainey]

Added doi field to datafile table. Refs #8538.

Changeset: b990be39e37845766b5c0d1ea64040740807bcbe

[Jay Rainey]

Restrict doiendpoint as facilities may not support. Refs #8538.

Changeset: 4521af6562ad132455c606216ff160ac63126dab

[Jay Rainey]

Not sure why, but several commits I just made didn't make their way onto track. They were:

1. ​Merged master into branch after #9223 was completed.
2. ​Refactor ICAT SOAP calls to improve readability.
3. ​Get the error message from soap, rather than do it manually.
4. ​Output the gSOAP error code to debug - very useful.

[Jay Rainey]

Moved keepAlive above IDS methods. Refs #8538.

Changeset: 22d2990bc831c4f71965ae3201ec45413fe921be

[Jay Rainey]

Updated doiendpoint (still using dev). Refs #8538.

Changeset: 63ea25c2a4474d02abf317dfc46f7693d904239c

[Jay Rainey]

Updated doiendpoint to new development url. Refs #8538.

Changeset: 0f951ee96e6fce41e1a043c06590b5111a51098a

[Jay Rainey]

Merge 'origin/master' into feature/8538_icat_publish_dois. Refs #8538.

Conflicts:

• Code/Mantid/Framework/ICat/src/ICat4/ICat4Catalog.cpp

Changeset: 4b1e897e3ab5869886f7b381b5d8773bb0678296

[Jay Rainey]

Add better logging messages for publishing. Refs #8538.

Changeset: e1b0fc694f9fb735b75646739c7d11fb32e1b79e

[Jay Rainey]

The functionality is complete, but am having issues with the ICAT DOI development server (It's throwing an error when DOI generation should be valid). I've got in touch with the DOI developer and once this is sorted this can be tested.

[Jay Rainey]

Updated DOI wsdl

[Jay Rainey]

The functionality for this ticket is complete. However, there are permission issues in ICAT that are preventing DOIs being generated, which Tom is looking into. Once these have been addressed this ticket can be opened for testing. If it does not pass testing, any additional changes should occur on ​the published branch.

Background

This ticket deals with generating a DOI when the user publishes a datafile to the archives and has checked the "Make public" option on the CatalogPublish dialog.

Testing

Development settings in Facilities.xml

You must update the url of soapendpoint, externaldownload and doiendpoint to be the following to ensure only test DOIs are generated:

 <soapendpoint url="https://icatdev.isis.cclrc.ac.uk/ICATService/ICAT"></soapendpoint>
 <externaldownload url="https://isisoxfordvmsrv.isis.cclrc.ac.uk/ids/"></externaldownload>
 <doiendpoint url="https://topcatdev.isis.cclrc.ac.uk/doi/DOIWebService"></doiendpoint>

Generating a DOI

To be thorough I have attached a python script that will generate a DOI against the test server. This should be used prior to testing it within Mantid. This is a faster way to observe if there are any errors on ICAT when generating DOIs.

Generating a DOI with the provided script

Note: If the DOI has been generated, then this proves that a DOI can be generated, and saved into ICAT successfully.

Remember to change your Facilities.xml file to the developer options.

1. Run the attached script. Enter your username & password.
2. Select f for the Object type as we want to generate a DOI for a datafile.
3. Enter 32493044 into the Object id field. Note: This is the ID of a datafile in an EMU investigation. The title of this investigation is: Magnetic ionic liquid surfactants.
4. You will notice that a DOI has been generated. If an error occurs (DOI not created, response code: 400), then this ICAT permissions is is still not fixed! Contact Tom about this.
5. Now that a DOI has been generated we need to see if it saved correctly to ICAT.
6. Log into the catalog and open the Search GUI.
7. In the Investigation name field enter: Magnetic ionic liquid surfactants and click Search.
8. Double click on the investigation and the Datafiles information table will be populated.
9. Observe that a DOI now exists in the related row for that datafile.

Generating a DOI within Mantid

Remember to change your Facilities.xml file to the developer options.

1. Log into the catalog.
2. Left click Publish in the Catalog menu. The publishing GUI will open.
3. Select Make public. This option ensures a DOI is generated when the datafile is published.
4. Select a datafile to upload to ICAT.
5. Enter a name in the Name in catalog field. Copy that name to your clipboard.
6. Select an investigation where you would like to publish the datafile.
7. Click Run. If the file is uploaded successfully then a DOI will be generated.
8. Open the catalog search GUI.
9. Click Advanced search and paste the above file name into the Datafile name field.
10. Click Search and double click the investigation that is returned.

Code review

As generating a DOI is irreversible it is essential to perform a thorough code review. I recommend reading comment:8 to gain an idea of what the desired logic for this ticket was.

Note: once this ticket has been tested the ICAT DOI endpoint (doiendpoint) should be updated to that of the production server, and not the testing server.

[Nick Draper]

Moved to the backlog at the code freeze of R3.3

[Stuart Campbell]

This ticket has been transferred to github issue ​9382