Ticket #8924 (closed: fixed)

Opened 7 years ago

Last modified 5 years ago

Repo. for instrument files

Reported by: Nick Draper Owned by: Nick Draper
Priority: major Milestone: Release 3.3
Component: Framework Keywords: SSC,2014,All
Cc: Blocked By: #10489, #10549, #10550, #10573, #10574
Blocking: Tester: Owen Arnold

Description

Allow Mantid to download changed/new instrument files from github

  • no upload
  • still provide files with installer in case no network connection

Possibly use the Github API as long as it does not require additional 3rd party depedencies

Change History

comment:1 Changed 7 years ago by Anders Markvardsen

  • Owner set to Anders Markvardsen

comment:2 Changed 7 years ago by Nick Draper

  • Status changed from new to assigned

Bulk move of tickets out of triage (new) to assigned at the introduction of the triage state

comment:3 Changed 7 years ago by Nick Draper

  • Owner changed from Anders Markvardsen to Russell Taylor

comment:4 Changed 7 years ago by Nick Draper

Reassigned after a chat with Pete where I found out that Russell will be looking into this as well.

comment:5 Changed 6 years ago by Russell Taylor

  • Owner changed from Russell Taylor to Peter Peterson

This is for your safekeeping until there's a new designee.

comment:6 Changed 6 years ago by Nick Draper

  • Milestone changed from Release 3.2 to Backlog

Moved to Backlog at the code freeze of release 3.2

comment:7 Changed 6 years ago by Nick Draper

  • Milestone changed from Backlog to Release 3.3

comment:8 Changed 6 years ago by Nick Draper

  • Owner changed from Peter Peterson to Garrett Granroth

comment:9 Changed 6 years ago by Nick Draper

  • Owner changed from Garrett Granroth to Nick Draper

comment:10 Changed 6 years ago by Nick Draper

  • Blocked By 10489 added

comment:11 Changed 6 years ago by Nick Draper

  • Status changed from assigned to inprogress

Created ChecksumHelper

This allows sha-1 and git checksums to be calculated for strings and files. Includes unit tests.

re #8924

Changeset: fe446412f9a0653e127b68d691d0f07f484d12b2

comment:12 Changed 6 years ago by Nick Draper

added ChecksumHelper

re #8924

Changeset: 65000e831fff4de3758407df987494992923b720

comment:13 Changed 6 years ago by Nick Draper

re #8924 checkpointing work, but alg is functional

Changeset: e96dae8660f137ef651f5a568eb34413d2daddd9

comment:14 Changed 6 years ago by Nick Draper

  • Blocking 10549 added

comment:15 Changed 6 years ago by Nick Draper

  • Blocking 10550 added

comment:16 Changed 6 years ago by Nick Draper

re #8924 first version complete with tests

Changeset: d704f3d8d8996e660afa416bebc1cc0bbee5f5da

comment:17 Changed 6 years ago by Nick Draper

re #8924 Addded rate limit handling

Changeset: 70abecc41bfd767d045421e39f2beef10e0c4c15

comment:18 Changed 6 years ago by Nick Draper

re #8924 improved error handling

Changeset: 65c165b1f16fb56db9d85064f8ef2e7fe7529082

comment:19 Changed 6 years ago by Nick Draper

re #8924 merge changes

Merge branch 'feature/8924_Intrument_file_repo' into develop

Conflicts:

Code/Mantid/Framework/Kernel/inc/MantidKernel/ChecksumHelper.h Code/Mantid/Framework/Kernel/src/ChecksumHelper.cpp Code/Mantid/Framework/Kernel/test/ChecksumHelperTest.h

Changeset: 64d5d24912463a21b044ab5eaf0dad9cfd2299a6

comment:20 Changed 6 years ago by Nick Draper

re #8924 cppcheck and doxygen fixes

Changeset: 94c8bac69dc2ac35ddb865c6d38309e22cb674f9

comment:21 Changed 6 years ago by Nick Draper

re #8924 solve cmakelist merge issue

Changeset: 0de65d2cbc7cc4f984a94884080c5e579f86477e

comment:22 Changed 6 years ago by Nick Draper

re #8924 solve cmakelist merge issue

Changeset: 32887a7a08a5b4ee5391b25f56a380d127fcab1b

comment:23 Changed 6 years ago by Martyn Gigg

Use std::string rather than c-style buffers.

Refs #8924

Changeset: 1ee638c6eb202b75769fcae5c3851aeb737f20d3

comment:24 Changed 6 years ago by Martyn Gigg

Convert all line endings to UNIX style for git SHA1.

Refs #8924

Changeset: b5d2884b8720ae28a76d8fcf8e3c01b1d7663757

comment:25 Changed 6 years ago by Nick Draper

re #8924 linux warning

Changeset: 268ab7d06a53684e13bb7238577d0338a3e00c51

comment:26 Changed 6 years ago by Martyn Gigg

Avoid a bug in Poco SecureSocket stack.

The session must be aborted manually before any destructors from the other objects are called.

It is hard to pin down but at some point between v1.36 & v1.46 the bug was fixed by ensuring that destructors don't throw exceptions. Refs #8924

Changeset: b9520fc80b20b402c7f43b7c89f02d775ffd40bc

comment:27 Changed 6 years ago by Peter Peterson

  • Blocking 10559 added

(In #10559) Adding #8924 since Nick is putting together some useful utilities.

comment:28 Changed 6 years ago by Martyn Gigg

  • Blocked By 10573 added

comment:29 Changed 6 years ago by Martyn Gigg

  • Blocked By 10574 added

comment:30 Changed 6 years ago by Martyn Gigg

A few tidyups for namespacing, and includes

Also removes hack that didn't work for getting around Poco bug. Refs #8924

Changeset: cc5c8c04bcb9dbf50bcd687c151f1f66ce5c7e05

comment:31 Changed 6 years ago by Martyn Gigg

Use jsoncpp to parse JSON files

Also restructured the JSON that is written by us to look a bit more like github and standard JSON style. Refs #8924

Changeset: b5f510b4355fbe2dfea3ef9065fbf064ba409cd1

comment:32 Changed 6 years ago by Nick Draper

re #8924 Add doc file

Changeset: 1f874d6517a3a2010d7d7fe5d48e0848614db193

comment:33 Changed 6 years ago by Nick Draper

  • Blocking 10549 removed
  • Blocked By 10549 added

comment:34 Changed 6 years ago by Nick Draper

  • Blocking 10559 removed

comment:35 Changed 6 years ago by Nick Draper

  • Blocking 10550 removed
  • Blocked By 10550 added

comment:36 Changed 6 years ago by Nick Draper

comment:37 Changed 6 years ago by Nick Draper

re #8924 refactor to use InternetHelper

Changeset: 0facbc0d64bcc6a75f87e845cc48d571debbfc2a

comment:38 Changed 6 years ago by Nick Draper

re #8924 prevent Instrument updates for system tests

Changeset: 2af1627dacead03f5de18fae95e7d7cf3d35c34d

comment:39 Changed 6 years ago by Nick Draper

re #8924 resolve a proxy problem

Changeset: 8212a2aab27617211f209595138c4b969da253ec

comment:40 Changed 6 years ago by Nick Draper

re #8924 changes to make use of #10550 reduction of wrapper logging

Changeset: 6825f8440d1d7ebe950025328dcaa9f3b7d5a828

comment:41 Changed 6 years ago by Nick Draper

  • Status changed from inprogress to verify
  • Resolution set to fixed

To Test

This branch has changes to the mantid and systemTest repos, make sure to merge them both. For Macro users, this just means doing git test start and git test pass on both repos

Note: This process is rather slow in debug ~12 secs per run of DownloadInstrument, but is nice and quick in release ~0.4sec.

These notes are long - but don't be put off they are just thorough.

  • For these tests I will refer to the appdata directory
    • Win %APPDATA%/mantidproject/mantid/instrument
    • Linux/Mac ~/.mantid/instrument
  • Install directory
    • This is the instrument directory wherever Mantid was installed

setup

  1. Check on develop builds that the unit tests and system tests have passed
  2. If you are using mantid from an installed package you don't need to do this: but if you are coming from a dev environment then you need to go to your Mantid.Properties file and set:
    UpdateInstrumentDefinitions.OnStartup = 0
    to 
    UpdateInstrumentDefinitions.OnStartup = 1
    
  3. Before starting Mantid go and look for appdata, this directory may not exist (yet).

Check both ways of starting the instrument download

  1. Build mantid and startup mantidplot
    1. You should get a single additional line (at notice) in the log after the welcome to mantidplot it should say one of:
      • All instrument definitions up to date
      • Downloading ?? files from the instrument repository
      • An error message, but that is a fail really.
    2. The appdata directory, should be created and gain at least a github.json file, and the xml file for any instruments downloaded.
  2. Run the DownloadInstrument Algorithm. This will do the same thing, but nothing should downloaded this time, as you should be up to date.
  3. This should be relatively quick is it checks if there have been changes since the date of the github.json file. From now on we will bypass this check for the rest of the testing by deleting the appdata/gihub.json file at various points.

Extra files present in local instrument dirs

  1. Go to the install directory and take a copy of LOQ_Definition_20020226-.xml - do not change it other than the file name, which you should change to LOQCopy_Definition_20020226-.xml
  2. delete appdata/github.json
  3. Close and restart mantidplot. Nothing should be downloaded, no new .xml files should appear in appdata, the copy file should remain in the install directory.

Updates to instrument definitions - github is the master you will just fake changes to force a download

  1. Without closing mantidplot. Go to the install directory and fiddle with an instrument definition, I suggest you use LOQ_Definition_20020226-.xml and change line 122 to look like this (just a change of rot from 90.0 to 0.0) - DO NOT commit this change to master - see tear down
          <location ><facing rot="0.0"/> </location> <!-- bottom module looking from source to sample-->
    
  2. Without closing mantidplot. Load LOQ48098 (in autotest data). look at the instrument view, the bottom part of the high angle bank should be missing.
  3. delete appdata/github.json
  4. Close and restart mantidplot. You should get a message that at least one file has downloaded.
  5. Load LOQ48098 (in autotest data). look at the instrument view, the bottom part of the high angle bank will have returned.
  6. In the appdata directory you will have a nice clean copy of the LOQ_Definition_20020226-.xml file that will have downloaded from the instrument repository.
  7. Got to that file in the appdata directory and edit to make a visible change (the same edit as before will do).
  8. delete appdata/github.json
  9. Close and restart mantidplot. You should get a message that at least one file has downloaded.
  10. The version of the file in appdata should have been overwritten, and your change reverted.

New instrument definitions

  1. Close Mantidplot, delete LOQ_Definition_20020226-.xml from both appdata and install directory.
  2. delete appdata/github.json
  3. Open mantidplot. You should be told that 1 file has been downloaded.
  4. In the appdata directory you will have a nice clean copy of the LOQ_Definition_20020226-.xml file that will have downloaded from the instrument repository.
  5. Load LOQ48098 and check the instrument view to prove that the instrument file was loaded.

Tear down

  1. Go to the install directory change the name of LOQCopy_Definition_20020226-.xml back to LOQ_Definition_20020226-.xml.
  2. run a git status, make sure you are not about to merge any changes here into master, if so revert those files.
  3. set UpdateInstrumentDefinitions.OnStartup = 0 in your mantid.properties file
  4. Clear out the appdata dir if you want to

comment:42 Changed 6 years ago by Nick Draper

re #8924 mac fix for old std libraries

Changeset: dc0ef46e3a164a84b0f9a82ef70e7fef0ae59286

comment:43 Changed 6 years ago by Nick Draper

re #8924 resolve warning

Changeset: 345129828f0e34ed96beb7cd104d0b3716796fba

comment:44 Changed 6 years ago by Martyn Gigg

Fix fstrea::open call for older OS X library.

Refs #8924

Changeset: 707d11573dc873944e29463f662a7ad6d4b19505

comment:45 Changed 6 years ago by Harry Jeffery

Working as expected on Ubuntu 14.04

comment:46 Changed 6 years ago by Owen Arnold

Mantid on OSX is failing to start up and crashing as a result of these changes.

Here's the crash report.

0   com.apple.CoreFoundation      	0x00007fff8be2804a CFArrayGetCount + 10
1   libMantidKernel.dylib         	0x000000010212f559 Mantid::Kernel::proxyInformationFromPac(__CFDictionary const*, std::string const&, Mantid::Kernel::Logger&) + 585
2   libMantidKernel.dylib         	0x000000010212eea1 Mantid::Kernel::NetworkProxy::getHttpProxy(std::string const&) + 81
3   libMantidKernel.dylib         	0x0000000101ebf03d Mantid::Kernel::ConfigServiceImpl::getProxy(std::string const&) + 717
4   libMantidKernel.dylib         	0x0000000101f0d0bc Mantid::Kernel::InternetHelper::sendHTTPSRequest(std::string const&, std::ostream&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) + 1244
5   libMantidKernel.dylib         	0x0000000101f0c967 Mantid::Kernel::InternetHelper::sendRequest(std::string const&, std::ostream&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) + 71
6   libMantidKernel.dylib         	0x0000000101f0c8a5 Mantid::Kernel::InternetHelper::downloadFile(std::string const&, std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) + 165
7   libMantidDataHandling.dylib   	0x0000000111daad9e Mantid::DataHandling::DownloadInstrument::doDownloadFile(std::string const&, std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) + 46
8   libMantidDataHandling.dylib   	0x0000000111da9758 Mantid::DataHandling::DownloadInstrument::processRepository() + 680
9   libMantidDataHandling.dylib   	0x0000000111da899b Mantid::DataHandling::DownloadInstrument::exec() + 315
10  libMantidAPI.dylib            	0x0000000102b0e4c5 Mantid::API::Algorithm::execute() + 1109
11  libMantidAPI.dylib            	0x0000000102b1888c Mantid::API::Algorithm::executeAsyncImpl(Poco::Void const&) + 28
12  libMantidAPI.dylib            	0x0000000102b45095 Mantid::API::AlgorithmProxy::executeAsyncImpl(Poco::Void const&) + 37
13  libMantidAPI.dylib            	0x0000000102b45bf2 Poco::ActiveRunnable<bool, Poco::Void, Mantid::API::AlgorithmProxy>::run() + 98
14  libPocoFoundation.16.dylib    	0x0000000106ddf533 Poco::PooledThread::run() + 111
15  libPocoFoundation.16.dylib    	0x0000000106ddca40 Poco::ThreadImpl::runnableEntry(void*) + 100
16  libsystem_c.dylib             	0x00007fff88aae772 _pthread_start + 327
17  libsystem_c.dylib             	0x00007fff88a9b1a1 thread_start + 13

comment:47 Changed 6 years ago by Owen Arnold

  • Status changed from verify to reopened
  • Resolution fixed deleted

comment:48 Changed 6 years ago by Nick Draper

  • Status changed from reopened to inprogress

re #8924 forsce a better proxy url to solve mac crashes

Changeset: 84c74490cbb0e6baec7cd44a2aefc0ec963f2e4d

comment:49 Changed 6 years ago by Nick Draper

re #8924 forsce a better proxy url to solve mac crashes

Changeset: 3130053611a50abd1dac866f6c8ed2705b0c2f74

comment:50 Changed 6 years ago by Nick Draper

  • Status changed from inprogress to verify
  • Resolution set to fixed

Ready for retest on Mac

comment:51 Changed 6 years ago by Owen Arnold

  • Status changed from verify to verifying
  • Tester set to Owen Arnold

comment:52 Changed 6 years ago by Nick Draper

  • Status changed from verifying to closed

Merge branch 'feature/8924_Intrument_file_repo' of github.com:mantidproject/mantid into feature/8924_Intrument_file_repo

Full changeset: 9e2b70d0aa03b9ec63b6d65b88dc6e03412f1d0f

comment:53 Changed 6 years ago by Nick Draper

Merge branch 'feature/10550_alg_startup_logging' of github.com:mantidproject/mantid into feature/8924_Intrument_file_repo

Full changeset: 36a2cfe876e02c84a8fb4f797cbcecc2512b2668

comment:54 Changed 6 years ago by Nick Draper

Merge branch 'master' of github.com:mantidproject/mantid into feature/8924_Intrument_file_repo

Full changeset: b21996104cfded93ce61f21b9f940968708fb3c1

comment:55 Changed 6 years ago by Owen Arnold

Merge remote-tracking branch 'origin/feature/8924_Intrument_file_repo'

Full changeset: 837047fad4164e48daef6aba57930767a757ecf8

comment:56 Changed 6 years ago by Owen Arnold

Minor point. Your tests define and create a Fake not a Mock. It's a useful distinction and the name is confusing.

comment:57 Changed 6 years ago by Martyn Gigg

Merge branch 'feature/8924_Intrument_file_repo'

Full changeset: f2a8fcba41a0f7cedb1abc4313ea9591ef4ff6cf

comment:58 Changed 6 years ago by Nick Draper

  • Keywords SSC,2014,All added; SSC removed

comment:59 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 9767

Note: See TracTickets for help on using tickets.