ICAT4: External (HTTP) download

[Jay Rainey]

Implement functionality that uses ICAT 4.3 in order to download datafiles via HTTP (e.g. When a user does not have access to the archives).

To do this, I will need to:

1. Generate gSoap files for ICAT 4.3 and overwrite the current ICAT4 files.
2. Update broken queries to work with new database layout and query language.
3. Update Facilities.xml to include a new attribute (http://cloud089.gridpp.rl.ac.uk:8080/ids/), which is the URL used to download the files from.
4. Update CatalogInfo to be able to use the new attributes in the facilities file.
5. Implement the functionality to generate a correct URL for a given file in getDownloadURL() within ICat4Catalog.

[Jay Rainey]

Generated ICAT4.3 files and overwrote old ICAT4 files. Refs #8126.

• Updated .gitignore to allow generated files to be comitted.

Changeset: efbac9a16f33ecf7f01bf0ad073ffd187c6b8b34

[Jay Rainey]

Make ICAT use SSL. Refs #8126.

Changeset: 7e04a3b1361fe48fafa42030cd700e26c32b3488

[Jay Rainey]

Fixed broken queries. Refs #8126.

• For an investigation to see an instrument it must talk through the investigationInstrument table.

Changeset: d33dcd9781ecaca9f1d72d3bffcfa7a9ad8fb941

[Jay Rainey]

Search by ID instead of name. Refs #8126.

Changeset: 75de6b7bd21922c8ea4d1ba12bc35942cd2df652

[Jay Rainey]

Merge branch 'feature/8126_icat4_external_download' into develop. Refs #8126.

Conflicts:

Code/Mantid/Framework/ICat/src/ICat4/ICat4Catalog.cpp

Changeset: 4579d123b377a3b368787256038435eab4a9658b

[Jay Rainey]

Add external download attribute to Facilities.xml. Refs #8126.

Changeset: d3a4c350992af80c86d65de5e0e86ee786752859

[Jay Rainey]

Added new attributes to cataloginfo. Refs #8126.

Changeset: f2194fefb86f5360e9ab7fba2b23dbdfceaeab3f

[Jay Rainey]

Added external download support. Refs #8126.

Changeset: 4e300dceb2ad841dde02046da87a214ae43309d5

[Jay Rainey]

Fix broken merge. Refs #8126.

• I had a problem with code being on develop that shouldn't have been. As a quick fix I have indented the related method in order to overwrite all changes.

Changeset: 98fa15432aa180ba3182c1b8c10057b59d8720d4

[Jay Rainey]

I reverted my code back to comment:13 as I discovered that I need changes in #7637 in order to progress.

[Jay Rainey]

Updated FacilitiesSchema to include new attribute. Refs #8126.

Changeset: ef3150fd65e8807ed88ffd0753607e22c86d312b

[Jay Rainey]

Merge remote-tracking branch 'origin/master' into feature/8126_icat4_external_download. Refs #8126.

Conflicts:

Code/Mantid/Framework/ICat/src/ICat4/ICat4Catalog.cpp

Changeset: 6ce29981110d2fa9a10a5f3b9380f0fe5cad9f6f

[Jay Rainey]

Updated search query for ICat4.3. Refs #8126.

Changeset: 7b261752790d2557b23725a1e02f7e7fb8172bbb

[Jay Rainey]

Check if file is datafile before download. Refs #8126.

Changeset: 68d08d6dd5d8d15b098a83a9d51383b27375000e

[Jay Rainey]

Search by id rather than name. Refs #8126.

Changeset: 76a27a724be24fc783a828b2a3acec051e0a317a

[Jay Rainey]

Removed extra zero from label. Refs #8126.

Changeset: 9c1aec104cc9e0d024928631dc5bf1906eb46f61

[Jay Rainey]

Updated Facilities.xml to ICAT4.3. Refs #8126.

Changeset: cb7df2445b609b65694b68c800855d1cb4d4e67e

[Jay Rainey]

Merge branch 'feature/8126_icat4_external_download' into develop. Refs #8126.

Conflicts:

Code/Mantid/Framework/ICat/src/ICat4/ICat4Catalog.cpp

Changeset: 4b233e2c549f275108aa65efaa693c3a376b778f

[Jay Rainey]

This ticket addressed externally downloading datafiles from the archives. As such, you must not have access to the archives when testing this ticket.

To test

1. Enable debug in the console log.
2. Open the search window (Catalog -> Search).
3. Perform a search.
4. Selection an investigation.
5. Select a related datafile to download.
6. Select the location to save the file.
7. Verify that the file has downloaded to the correct location, and has the correct contents (load it), and verify that the size of the file is correct.

To be thorough I recommend downloading the file from the URL output in the console log to see if it has the same contents of the downloaded file. Note: this will require you to add the expected extension to the file (such as .raw).

[Jay Rainey]

(In #1807) This should not be tested until blocking tickets have been fixed.

[Jay Rainey]

(In #3827) This should not be tested until blocking tickets have been fixed.

[Jay Rainey]

(In #8209) This should not be tested until blocking tickets have been fixed.

[Jay Rainey]

(In #8202) This should not be tested until blocking tickets have been fixed.

[Jay Rainey]

Since I modified the soap end-point in Facilities.xml in this ticket, and the end-point is no longer valid (as Tom has moved it to icatisis) then I will need to update it again here to:

​https://icatisis.esc.rl.ac.uk/ICATService/ICAT

This can not be solved directly in #8128 as this ticket cannot be tested without the correct ICAT4.3 endpoint.

[Jay Rainey]

Updated soap end-point. Refs #8126.

Changeset: 277dc42c4f2c716aee9ace29dfa61543b41e5014

[Jay Rainey]

Regenerated ICAT4.3 gsoap files. Refs #8126.

• Updated .gitignore to allow generated files to be comitted.

Changeset: 967b64050c515fc1f97eef53bc7136d9d65e9436

[Jay Rainey]

Should not call inputs if not used. Refs #8126.

Changeset: fc2511f10751a755421a5c056ad0f335c6786720

[Jay Rainey]

I accidentally made the previous commit on this branch instead of #6457 (Not sure why as the ticket numbers are not similar...). It was a minor fix to prevent a crash on windows on search.

[Jay Rainey]

I am now working on fixing the windows crash noted in #6457 here as I require the ICAT4 endpoint, which has been updated here (previous ICAT4 end-point is no longer valid, and changing the end-point requires regeneration of gSoap files).

[Jay Rainey]

Removed windows specific date formats. Refs #8126.

Changeset: 9f47d0c45a0c49d6d1377400c8491e7eab631690

[Jay Rainey]

To test

1. Download a build from develop and install it on windows (as that's where the crash previously occurred).
2. Follow the instructions in comment:27.

[Owen Arnold]

Code review generally looks good.

I would suggest writing

if (extension.compare("raw") == 0 || extension.compare("nxs") == 0)
       {
         return true;
       }
       else
       {
         return false;
       }

as this:

return (extension.compare("raw") == 0 || extension.compare("nxs") == 0)

Don't change it for this release, or create a trial ticket for it, but don't bother fixing it here.

[Owen Arnold]

Merge remote-tracking branch 'origin/feature/8126_icat4_external_download'

Conflicts:

Code/Mantid/Framework/ICat/src/ICat4/ICat4Catalog.cpp

refs #8126

Changeset: 99bc13cbb9acbaf1ccedb96c7d5d4e4df92bccbb

[Owen Arnold]

Merge remote-tracking branch 'origin/feature/8126_icat4_external_download'

Conflicts:

Code/Mantid/Framework/ICat/src/ICat4/ICat4Catalog.cpp

refs #8126

Full changeset: ​99bc13cbb9acbaf1ccedb96c7d5d4e4df92bccbb

[Stuart Campbell]

This ticket has been transferred to github issue ​8971