Cross platform support for downloading archives

[Jay Rainey]

Currently, it is only possible to download files from the archive on a windows machine since the string returned by getFileLocation does not consider file paths or mount points.

Therefore, we need to create a class that will parse the string returned, and convert it to a usable path for the user's operating system. This will be done by converting the returned string from ICAT and changing it as needed.

For example, if a user has set their archive mount point to be /archive/ then we need to replace the windows specific part of the returned string with this. If no mount point exists, then we should download via HTTP.

[Martyn Gigg]

The agreed solution is described as follows.

Udate/add information to the Facilities.xml. The new structure should look like (ISIS example):

<catalog name="ICat3Catalog">
  <soapendpoint url="https://facilities01.esc.rl.ac.uk:443/ICATService/ICAT"></soapendpoint>
  <filelocation>
    <prefix regex="\\isis\inst$"></prefix>
    <windows replacement=""></prefix>
    <linux replacement="/archive"></prefix>
    <mac replacement="/archive"></prefix>
  </filelocation>
</catalog>

Note that the soapendpoint tag has been moved inside the relevant catalog tag, which will allow us to have multiple catalog tags in the future. It has also been put to lower case for consistency.

The Facilities schema (XSD) in the Code/Mantid/instrument/Schema directory will need to be updated too.

The Kernel::FacilityInfo class needs to be updated to understand the new information. My suggestion is to add a new class called CatalogInfo and have the FacilityInfo class store a list of these objects. The CatalogInfo class should store all of the info enclosed within the <catalog> tag above.

The current 'getSoapEndPoint' method should then be moved to the CatalogInfo class and additional methods added to CatalogInfo to enquire about the other values. I would also suggest an extra method on he CatalogInfo class that can transform a file location to the correct value/style for the current operating system. That way this can be well encapsulated, unit tested and available elsewhere.

The final piece of the puzzle is then using this functionality in the ICAT layer.

[Jay Rainey]

Added new catalog scheme. Refs #7866.

Changeset: 4b5b88c79d75b97d91e49ea3a8b58e5bb339c1a9

[Jay Rainey]

Added outline for CatalogInfo class. Refs #7866.

Changeset: 8f2e1dabc985d4f1d7e0ce9eeaa4e787c761439f

[Jay Rainey]

Use catalog class in FacilityInfo instead. Refs #7866.

Changeset: af396d251e3a228c0c7cfa22ff54ef32cfd13ef3

[Jay Rainey]

Removed now unnecessary methods. Refs #7866.

Changeset: 8ad3e03c7d1d49ce5677a408e120059c97b05a70

[Jay Rainey]

Methods updated for CatalogInfo. Refs #7866.

Changeset: 8756f39739d95f149ca4edb52faaa8db3012a914

[Jay Rainey]

Updated regex in Facilities.xml. Refs #7866.

Changeset: 8f2ae56c5da003ffa964ea36dd6a1dca75149b89

[Jay Rainey]

Updated transform method and created helper methods.Refs #7866.

Changeset: ae10345c859c4f75d53ecea234eefee51f83b88d

[Jay Rainey]

Updated regex in Facilities.xml. Refs #7866.

• Now accounts for Instruments$ in the search path.

Changeset: 10618f59a3d6aac0436df374efda21fab8e5375e

[Jay Rainey]

Added unit test for CatalogInfo. Refs #7866.

Changeset: 8bd0f1169f6ff1110dfce4bb9d25299c719472bb

[Jay Rainey]

Silence compiler warning. Refs #7866.

Changeset: bb5bcce55272ce047b7c9bd207c8d227ebaa9f78

[Jay Rainey]

Added new catalog scheme. Refs #7866.

Changeset: 4b5b88c79d75b97d91e49ea3a8b58e5bb339c1a9

[Jay Rainey]

Added outline for CatalogInfo class. Refs #7866.

Changeset: 8f2e1dabc985d4f1d7e0ce9eeaa4e787c761439f

[Jay Rainey]

Use catalog class in FacilityInfo instead. Refs #7866.

Changeset: af396d251e3a228c0c7cfa22ff54ef32cfd13ef3

[Jay Rainey]

Removed now unnecessary methods. Refs #7866.

Changeset: 8ad3e03c7d1d49ce5677a408e120059c97b05a70

[Jay Rainey]

Methods updated for CatalogInfo. Refs #7866.

Changeset: 8756f39739d95f149ca4edb52faaa8db3012a914

[Jay Rainey]

Updated regex in Facilities.xml. Refs #7866.

Changeset: 8f2ae56c5da003ffa964ea36dd6a1dca75149b89

[Jay Rainey]

Updated transform method and created helper methods.Refs #7866.

Changeset: ae10345c859c4f75d53ecea234eefee51f83b88d

[Jay Rainey]

Updated regex in Facilities.xml. Refs #7866.

• Now accounts for Instruments$ in the search path.

Changeset: 10618f59a3d6aac0436df374efda21fab8e5375e

[Jay Rainey]

Added unit test for CatalogInfo. Refs #7866.

Changeset: 8bd0f1169f6ff1110dfce4bb9d25299c719472bb

[Jay Rainey]

Silence compiler warning. Refs #7866.

Changeset: bb5bcce55272ce047b7c9bd207c8d227ebaa9f78

[Jay Rainey]

Updated transformArchivePath. Refs #7866.

• The boolean check was removed and related search is now windows specific as it's not needed to be performed on all operating systems.
• Updated replaceAllOccurences to return the modified string.
• Added "@return" documentation.

Changeset: 19106ea5cf6fa8bb9bce82630ce91a4a316f366f

[Jay Rainey]

Updated CatalogInfoTest windows replacement. Refs #7866.

• Added new path to test against when on windows (winPrefixPath)

Changeset: b8f5ba7c2496c2d7a17f359a580e20ba7b4acefe

[Jay Rainey]

Added CatalogInfo class to FacilityInfo. Refs #7866.

Changeset: 87485a4f9593658ef93ce842b8ad66b765d54ab8

[Jay Rainey]

Removed any reference to soapEndPoint and catalogName. Refs #7866.

• Updated algorithms to make use of catalogInfo within FacilityInfo.

Changeset: c9e998a71f60ffd25bb2152a551e27b930bc145c

[Jay Rainey]

Transform file location to user specific operating system. Refs #7866.

• Removed unnecessary backslash replacement in ICat3Helper.cpp as it made the transform invalid.

Changeset: 225b8ca8cdfdb19ca702d48fc881f5212399f2ce

[Jay Rainey]

Tidied up CatalogDownloadDataFiles. Refs #7866.

Changeset: ad8f0c4b7b6109f9e8eed3daf0e95845fbce07df

[Jay Rainey]

To test

In order to verify that the changes are correct ensure that:

• All tests pass on all platforms.
• The code changes I made are up to standard, and in-line with Martyn's recommendations in comment:2

To test via ICAT GUI

1. Enable debugging in Results log (right click -> Log level -> Debug).
2. Log into ICAT
3. Perform a search (You could type bob into Keywords then click Search).
4. Double click on an investigation result - a new window is open.
5. Click on the dropdown button >
6. Click on the Run range dropdown button (>), and select a datafile and click download.
7. If you have access to the archives then the following will be output:

   File(fileName) located in archives.

   Otherwise, the file will be downloaded over HTTP and the output will be:

   Unable to open file (fileName) from archive. Beginning to download over Internet.

If possible, verify that this works on various platforms when you have access to archives and when you do not.

[Jay Rainey]

Const correctness corrections. Refs #7866.

Changeset: 3fa4725e5ffb9026baf0853e058b40ca271df894

[Martyn Gigg]

I have tested out the interface on Windows and things seems to work as suggested. I will also test on Linux.

I have a few comments about the code:

• The file ICATSearch.cpp produces a Qt error in the console saying "Object::connect: No such slot ApplicationWindow::writeErrorToLogWindow(const QString&)". This needs to be addressed.
• On line 127 of CatalogDownloadDataFiles.cpp there is a variable called "isisfile". Can we change this to something like filehandle or basically something that doesn't have isis in the name. This gives more confidence that we are not ISIS centric.
• The isDataFile() method in CatalogDownloadDataFiles needs to be more flexible. It should not simply be checking for extensions it should be looking at whether the stream is binary or not. My suggestion here is to change the name of <code>isDataFile</code> to <code>isBinary</code> and have it use the class in Kernel called FileDescriptor. This has a static method called 'isAscii' that takes a stream. I would suggest <code>isBinary</code> simply return <code>!FileDescriptor::isAscii(strm)</code>
• There is a slight inconsistency in parameter names between <code>init</code> & <code>exec</code>. Can we make sure Filenames parameters are used consistently
• The code block

  ICatalog_sptr catalog;
  
        try
        {
          catalog = CatalogFactory::Instance().create(ConfigService::Instance().getFacility().catalogInfo().catalogName());
  
        }
        catch(Kernel::Exception::NotFoundError&)
        {
          throwCatalogError();
        }
  
        if(!catalog)
        {
          throwCatalogError();
        }
  

is repeated alot across the ICAT algorithms. Can we have a common place for this? Or was this put into the maintenance ticket?

[Martyn Gigg]

From my tests on Ubuntu it would seem that the Load button does not work. The download button correctly identifies the archive file location but the load button tries to use the windows path that is in the table.

[Jay Rainey]

Preventted QT slot errors from occuring. Refs #7866.

Changeset: 4888460b4e6403dd3774b544aaca7a7971590342

[Jay Rainey]

Tidied up CatalogDownloadDataFiles. Refs #7866.

• Removed reference to ISIS.
• Made use of isAscii and removed method to isBinary.

Changeset: 0bd01239fd2fcfeeaa7f73e02a5dd58922706cf7

[Jay Rainey]

Added ICAT3 load support. Refs #7866.

• Transformed the string to user-specific ICAT path, in order to directly load it from archives.

Changeset: 72f9466ff587a121dd970d26fb536c8b9f158e2d

[Jay Rainey]

Fixed failing test. Refs #7866.

• If filename is invalid or does not exist then we want to return false.

Changeset: de367a5acf5093403db3d2753ed3c23409fe0b3b

[Jay Rainey]

Merge branch 'feature/7866_icat_cross_platform_archive_download' into develop. Refs #7866.

Conflicts:

Code/Mantid/Framework/ICat/src/CatalogDownloadDataFiles.cpp

Changeset: 31b13d97232c45f568ba47873a3a3e77c1b9fa42

[Jay Rainey]

Shared functionality and error handling across algorithms will be dealt with in another ticket during maintenance and will be encapsulated into an ICatAlgorithmHelper class.

[Jay Rainey]

When testing this ticket can you verify/invalidate #7886 as it's related to the removal of writeErrorToLogWindow errors.

[Jay Rainey]

.

[Martyn Gigg]

The new implementation of isBinary is not correct. It needs to be changed to accept an istream object and then pass this to FileDescriptor::isAscii()

[Jay Rainey]

Improved isBinary method. Refs #7866.

Changeset: 5bca4fc028b2a7f0735a4417e50d56fb626739ba

[Martyn Gigg]

The loading functionality seems to work properly now on both Windows & Linux. The download function does not work but this has already been noted and is covered by #8144.

[Martyn Gigg]

Merge remote-tracking branch 'origin/feature/7866_icat_cross_platform_archive_download'

Full changeset: ​28aee853d2539e08a43ae17be9709701786f398a

[Stuart Campbell]

This ticket has been transferred to github issue ​8711