Repo. for instrument files

[Nick Draper]

Allow Mantid to download changed/new instrument files from github

• no upload
• still provide files with installer in case no network connection

Possibly use the Github API as long as it does not require additional 3rd party depedencies

[Nick Draper]

Bulk move of tickets out of triage (new) to assigned at the introduction of the triage state

[Nick Draper]

Design doc

​https://github.com/mantidproject/documents/blob/master/Design/InstrumentFetching.md

[Nick Draper]

Reassigned after a chat with Pete where I found out that Russell will be looking into this as well.

[Russell Taylor]

This is for your safekeeping until there's a new designee.

[Nick Draper]

Moved to Backlog at the code freeze of release 3.2

[Nick Draper]

Created ChecksumHelper

This allows sha-1 and git checksums to be calculated for strings and files. Includes unit tests.

re #8924

Changeset: fe446412f9a0653e127b68d691d0f07f484d12b2

[Nick Draper]

added ChecksumHelper

re #8924

Changeset: 65000e831fff4de3758407df987494992923b720

[Nick Draper]

re #8924 checkpointing work, but alg is functional

Changeset: e96dae8660f137ef651f5a568eb34413d2daddd9

[Nick Draper]

re #8924 first version complete with tests

Changeset: d704f3d8d8996e660afa416bebc1cc0bbee5f5da

[Nick Draper]

re #8924 Addded rate limit handling

Changeset: 70abecc41bfd767d045421e39f2beef10e0c4c15

[Nick Draper]

re #8924 improved error handling

Changeset: 65c165b1f16fb56db9d85064f8ef2e7fe7529082

[Nick Draper]

re #8924 merge changes

Merge branch 'feature/8924_Intrument_file_repo' into develop

Conflicts:

Code/Mantid/Framework/Kernel/inc/MantidKernel/ChecksumHelper.h Code/Mantid/Framework/Kernel/src/ChecksumHelper.cpp Code/Mantid/Framework/Kernel/test/ChecksumHelperTest.h

Changeset: 64d5d24912463a21b044ab5eaf0dad9cfd2299a6

[Nick Draper]

re #8924 cppcheck and doxygen fixes

Changeset: 94c8bac69dc2ac35ddb865c6d38309e22cb674f9

[Nick Draper]

re #8924 solve cmakelist merge issue

Changeset: 0de65d2cbc7cc4f984a94884080c5e579f86477e

[Nick Draper]

re #8924 solve cmakelist merge issue

Changeset: 32887a7a08a5b4ee5391b25f56a380d127fcab1b

[Martyn Gigg]

Use std::string rather than c-style buffers.

Refs #8924

Changeset: 1ee638c6eb202b75769fcae5c3851aeb737f20d3

[Martyn Gigg]

Convert all line endings to UNIX style for git SHA1.

Refs #8924

Changeset: b5d2884b8720ae28a76d8fcf8e3c01b1d7663757

[Nick Draper]

re #8924 linux warning

Changeset: 268ab7d06a53684e13bb7238577d0338a3e00c51

[Martyn Gigg]

Avoid a bug in Poco SecureSocket stack.

The session must be aborted manually before any destructors from the other objects are called.

It is hard to pin down but at some point between v1.36 & v1.46 the bug was fixed by ensuring that destructors don't throw exceptions. Refs #8924

Changeset: b9520fc80b20b402c7f43b7c89f02d775ffd40bc

[Peter Peterson]

(In #10559) Adding #8924 since Nick is putting together some useful utilities.

[Martyn Gigg]

A few tidyups for namespacing, and includes

Also removes hack that didn't work for getting around Poco bug. Refs #8924

Changeset: cc5c8c04bcb9dbf50bcd687c151f1f66ce5c7e05

[Martyn Gigg]

Use jsoncpp to parse JSON files

Also restructured the JSON that is written by us to look a bit more like github and standard JSON style. Refs #8924

Changeset: b5f510b4355fbe2dfea3ef9065fbf064ba409cd1

[Nick Draper]

re #8924 Add doc file

Changeset: 1f874d6517a3a2010d7d7fe5d48e0848614db193

[Nick Draper]

re #8924, merge in #10549 and #10550

Changeset: a34f1ef969a4aeaa10f593e7837bde4c159fde17

[Nick Draper]

re #8924 refactor to use InternetHelper

Changeset: 0facbc0d64bcc6a75f87e845cc48d571debbfc2a

[Nick Draper]

re #8924 prevent Instrument updates for system tests

Changeset: 2af1627dacead03f5de18fae95e7d7cf3d35c34d

[Nick Draper]

re #8924 resolve a proxy problem

Changeset: 8212a2aab27617211f209595138c4b969da253ec

[Nick Draper]

re #8924 changes to make use of #10550 reduction of wrapper logging

Changeset: 6825f8440d1d7ebe950025328dcaa9f3b7d5a828

[Nick Draper]

To Test

This branch has changes to the mantid and systemTest repos, make sure to merge them both. For Macro users, this just means doing git test start and git test pass on both repos

Note: This process is rather slow in debug ~12 secs per run of DownloadInstrument, but is nice and quick in release ~0.4sec.

These notes are long - but don't be put off they are just thorough.

• For these tests I will refer to the appdata directory
  • Win %APPDATA%/mantidproject/mantid/instrument
  • Linux/Mac ~/.mantid/instrument
• Install directory
  • This is the instrument directory wherever Mantid was installed

setup

1. Check on develop builds that the unit tests and system tests have passed
2. If you are using mantid from an installed package you don't need to do this: but if you are coming from a dev environment then you need to go to your Mantid.Properties file and set:

   UpdateInstrumentDefinitions.OnStartup = 0
   to 
   UpdateInstrumentDefinitions.OnStartup = 1
   

3. Before starting Mantid go and look for appdata, this directory may not exist (yet).

Check both ways of starting the instrument download

1. Build mantid and startup mantidplot
   1. You should get a single additional line (at notice) in the log after the welcome to mantidplot it should say one of:
      • All instrument definitions up to date
      • Downloading ?? files from the instrument repository
      • An error message, but that is a fail really.
   2. The appdata directory, should be created and gain at least a github.json file, and the xml file for any instruments downloaded.
2. Run the DownloadInstrument Algorithm. This will do the same thing, but nothing should downloaded this time, as you should be up to date.
3. This should be relatively quick is it checks if there have been changes since the date of the github.json file. From now on we will bypass this check for the rest of the testing by deleting the appdata/gihub.json file at various points.

Extra files present in local instrument dirs

1. Go to the install directory and take a copy of LOQ_Definition_20020226-.xml - do not change it other than the file name, which you should change to LOQCopy_Definition_20020226-.xml
2. delete appdata/github.json
3. Close and restart mantidplot. Nothing should be downloaded, no new .xml files should appear in appdata, the copy file should remain in the install directory.

Updates to instrument definitions - github is the master you will just fake changes to force a download

1. Without closing mantidplot. Go to the install directory and fiddle with an instrument definition, I suggest you use LOQ_Definition_20020226-.xml and change line 122 to look like this (just a change of rot from 90.0 to 0.0) - DO NOT commit this change to master - see tear down

         <location ><facing rot="0.0"/> </location> <!-- bottom module looking from source to sample-->
   

2. Without closing mantidplot. Load LOQ48098 (in autotest data). look at the instrument view, the bottom part of the high angle bank should be missing.
3. delete appdata/github.json
4. Close and restart mantidplot. You should get a message that at least one file has downloaded.
5. Load LOQ48098 (in autotest data). look at the instrument view, the bottom part of the high angle bank will have returned.
6. In the appdata directory you will have a nice clean copy of the LOQ_Definition_20020226-.xml file that will have downloaded from the instrument repository.
7. Got to that file in the appdata directory and edit to make a visible change (the same edit as before will do).
8. delete appdata/github.json
9. Close and restart mantidplot. You should get a message that at least one file has downloaded.
10. The version of the file in appdata should have been overwritten, and your change reverted.

New instrument definitions

1. Close Mantidplot, delete LOQ_Definition_20020226-.xml from both appdata and install directory.
2. delete appdata/github.json
3. Open mantidplot. You should be told that 1 file has been downloaded.
4. In the appdata directory you will have a nice clean copy of the LOQ_Definition_20020226-.xml file that will have downloaded from the instrument repository.
5. Load LOQ48098 and check the instrument view to prove that the instrument file was loaded.

Tear down

1. Go to the install directory change the name of LOQCopy_Definition_20020226-.xml back to LOQ_Definition_20020226-.xml.
2. run a git status, make sure you are not about to merge any changes here into master, if so revert those files.
3. set UpdateInstrumentDefinitions.OnStartup = 0 in your mantid.properties file
4. Clear out the appdata dir if you want to

[Nick Draper]

re #8924 mac fix for old std libraries

Changeset: dc0ef46e3a164a84b0f9a82ef70e7fef0ae59286

[Nick Draper]

re #8924 resolve warning

Changeset: 345129828f0e34ed96beb7cd104d0b3716796fba

[Martyn Gigg]

Fix fstrea::open call for older OS X library.

Refs #8924

Changeset: 707d11573dc873944e29463f662a7ad6d4b19505

[Harry Jeffery]

Working as expected on Ubuntu 14.04

[Owen Arnold]

Mantid on OSX is failing to start up and crashing as a result of these changes.

Here's the crash report.

0   com.apple.CoreFoundation      	0x00007fff8be2804a CFArrayGetCount + 10
1   libMantidKernel.dylib         	0x000000010212f559 Mantid::Kernel::proxyInformationFromPac(__CFDictionary const*, std::string const&, Mantid::Kernel::Logger&) + 585
2   libMantidKernel.dylib         	0x000000010212eea1 Mantid::Kernel::NetworkProxy::getHttpProxy(std::string const&) + 81
3   libMantidKernel.dylib         	0x0000000101ebf03d Mantid::Kernel::ConfigServiceImpl::getProxy(std::string const&) + 717
4   libMantidKernel.dylib         	0x0000000101f0d0bc Mantid::Kernel::InternetHelper::sendHTTPSRequest(std::string const&, std::ostream&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) + 1244
5   libMantidKernel.dylib         	0x0000000101f0c967 Mantid::Kernel::InternetHelper::sendRequest(std::string const&, std::ostream&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) + 71
6   libMantidKernel.dylib         	0x0000000101f0c8a5 Mantid::Kernel::InternetHelper::downloadFile(std::string const&, std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) + 165
7   libMantidDataHandling.dylib   	0x0000000111daad9e Mantid::DataHandling::DownloadInstrument::doDownloadFile(std::string const&, std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) + 46
8   libMantidDataHandling.dylib   	0x0000000111da9758 Mantid::DataHandling::DownloadInstrument::processRepository() + 680
9   libMantidDataHandling.dylib   	0x0000000111da899b Mantid::DataHandling::DownloadInstrument::exec() + 315
10  libMantidAPI.dylib            	0x0000000102b0e4c5 Mantid::API::Algorithm::execute() + 1109
11  libMantidAPI.dylib            	0x0000000102b1888c Mantid::API::Algorithm::executeAsyncImpl(Poco::Void const&) + 28
12  libMantidAPI.dylib            	0x0000000102b45095 Mantid::API::AlgorithmProxy::executeAsyncImpl(Poco::Void const&) + 37
13  libMantidAPI.dylib            	0x0000000102b45bf2 Poco::ActiveRunnable<bool, Poco::Void, Mantid::API::AlgorithmProxy>::run() + 98
14  libPocoFoundation.16.dylib    	0x0000000106ddf533 Poco::PooledThread::run() + 111
15  libPocoFoundation.16.dylib    	0x0000000106ddca40 Poco::ThreadImpl::runnableEntry(void*) + 100
16  libsystem_c.dylib             	0x00007fff88aae772 _pthread_start + 327
17  libsystem_c.dylib             	0x00007fff88a9b1a1 thread_start + 13

[Nick Draper]

re #8924 forsce a better proxy url to solve mac crashes

Changeset: 84c74490cbb0e6baec7cd44a2aefc0ec963f2e4d

[Nick Draper]

re #8924 forsce a better proxy url to solve mac crashes

Changeset: 3130053611a50abd1dac866f6c8ed2705b0c2f74

[Nick Draper]

Ready for retest on Mac

[Nick Draper]

Merge branch 'feature/8924_Intrument_file_repo' of github.com:mantidproject/mantid into feature/8924_Intrument_file_repo

Full changeset: ​9e2b70d0aa03b9ec63b6d65b88dc6e03412f1d0f

[Nick Draper]

Merge branch 'feature/10550_alg_startup_logging' of github.com:mantidproject/mantid into feature/8924_Intrument_file_repo

Full changeset: ​36a2cfe876e02c84a8fb4f797cbcecc2512b2668

[Nick Draper]

Merge branch 'master' of github.com:mantidproject/mantid into feature/8924_Intrument_file_repo

Full changeset: ​b21996104cfded93ce61f21b9f940968708fb3c1

[Owen Arnold]

Merge remote-tracking branch 'origin/feature/8924_Intrument_file_repo'

Full changeset: ​837047fad4164e48daef6aba57930767a757ecf8

[Owen Arnold]

Minor point. Your tests define and create a Fake not a Mock. It's a useful distinction and the name is confusing.

[Martyn Gigg]

Merge branch 'feature/8924_Intrument_file_repo'

Full changeset: ​f2a8fcba41a0f7cedb1abc4313ea9591ef4ff6cf

[Stuart Campbell]

This ticket has been transferred to github issue ​9767