Ticket #8924 (closed: fixed)
Repo. for instrument files
Reported by: | Nick Draper | Owned by: | Nick Draper |
---|---|---|---|
Priority: | major | Milestone: | Release 3.3 |
Component: | Framework | Keywords: | SSC,2014,All |
Cc: | Blocked By: | #10489, #10549, #10550, #10573, #10574 | |
Blocking: | Tester: | Owen Arnold |
Description
Allow Mantid to download changed/new instrument files from github
- no upload
- still provide files with installer in case no network connection
Possibly use the Github API as long as it does not require additional 3rd party depedencies
Change History
comment:2 Changed 7 years ago by Nick Draper
- Status changed from new to assigned
Bulk move of tickets out of triage (new) to assigned at the introduction of the triage state
comment:3 Changed 7 years ago by Nick Draper
- Owner changed from Anders Markvardsen to Russell Taylor
comment:4 Changed 7 years ago by Nick Draper
Reassigned after a chat with Pete where I found out that Russell will be looking into this as well.
comment:5 Changed 6 years ago by Russell Taylor
- Owner changed from Russell Taylor to Peter Peterson
This is for your safekeeping until there's a new designee.
comment:6 Changed 6 years ago by Nick Draper
- Milestone changed from Release 3.2 to Backlog
Moved to Backlog at the code freeze of release 3.2
comment:11 Changed 6 years ago by Nick Draper
- Status changed from assigned to inprogress
Created ChecksumHelper
This allows sha-1 and git checksums to be calculated for strings and files. Includes unit tests.
re #8924
Changeset: fe446412f9a0653e127b68d691d0f07f484d12b2
comment:12 Changed 6 years ago by Nick Draper
comment:13 Changed 6 years ago by Nick Draper
re #8924 checkpointing work, but alg is functional
Changeset: e96dae8660f137ef651f5a568eb34413d2daddd9
comment:16 Changed 6 years ago by Nick Draper
re #8924 first version complete with tests
Changeset: d704f3d8d8996e660afa416bebc1cc0bbee5f5da
comment:17 Changed 6 years ago by Nick Draper
re #8924 Addded rate limit handling
Changeset: 70abecc41bfd767d045421e39f2beef10e0c4c15
comment:18 Changed 6 years ago by Nick Draper
re #8924 improved error handling
Changeset: 65c165b1f16fb56db9d85064f8ef2e7fe7529082
comment:19 Changed 6 years ago by Nick Draper
re #8924 merge changes
Merge branch 'feature/8924_Intrument_file_repo' into develop
Conflicts:
Code/Mantid/Framework/Kernel/inc/MantidKernel/ChecksumHelper.h Code/Mantid/Framework/Kernel/src/ChecksumHelper.cpp Code/Mantid/Framework/Kernel/test/ChecksumHelperTest.h
Changeset: 64d5d24912463a21b044ab5eaf0dad9cfd2299a6
comment:20 Changed 6 years ago by Nick Draper
re #8924 cppcheck and doxygen fixes
Changeset: 94c8bac69dc2ac35ddb865c6d38309e22cb674f9
comment:21 Changed 6 years ago by Nick Draper
re #8924 solve cmakelist merge issue
Changeset: 0de65d2cbc7cc4f984a94884080c5e579f86477e
comment:22 Changed 6 years ago by Nick Draper
re #8924 solve cmakelist merge issue
Changeset: 32887a7a08a5b4ee5391b25f56a380d127fcab1b
comment:23 Changed 6 years ago by Martyn Gigg
Use std::string rather than c-style buffers.
Refs #8924
Changeset: 1ee638c6eb202b75769fcae5c3851aeb737f20d3
comment:24 Changed 6 years ago by Martyn Gigg
Convert all line endings to UNIX style for git SHA1.
Refs #8924
Changeset: b5d2884b8720ae28a76d8fcf8e3c01b1d7663757
comment:25 Changed 6 years ago by Nick Draper
re #8924 linux warning
Changeset: 268ab7d06a53684e13bb7238577d0338a3e00c51
comment:26 Changed 6 years ago by Martyn Gigg
Avoid a bug in Poco SecureSocket stack.
The session must be aborted manually before any destructors from the other objects are called.
It is hard to pin down but at some point between v1.36 & v1.46 the bug was fixed by ensuring that destructors don't throw exceptions. Refs #8924
Changeset: b9520fc80b20b402c7f43b7c89f02d775ffd40bc
comment:27 Changed 6 years ago by Peter Peterson
- Blocking 10559 added
comment:30 Changed 6 years ago by Martyn Gigg
A few tidyups for namespacing, and includes
Also removes hack that didn't work for getting around Poco bug. Refs #8924
Changeset: cc5c8c04bcb9dbf50bcd687c151f1f66ce5c7e05
comment:31 Changed 6 years ago by Martyn Gigg
Use jsoncpp to parse JSON files
Also restructured the JSON that is written by us to look a bit more like github and standard JSON style. Refs #8924
Changeset: b5f510b4355fbe2dfea3ef9065fbf064ba409cd1
comment:32 Changed 6 years ago by Nick Draper
re #8924 Add doc file
Changeset: 1f874d6517a3a2010d7d7fe5d48e0848614db193
comment:36 Changed 6 years ago by Nick Draper
re #8924, merge in #10549 and #10550
Changeset: a34f1ef969a4aeaa10f593e7837bde4c159fde17
comment:37 Changed 6 years ago by Nick Draper
re #8924 refactor to use InternetHelper
Changeset: 0facbc0d64bcc6a75f87e845cc48d571debbfc2a
comment:38 Changed 6 years ago by Nick Draper
re #8924 prevent Instrument updates for system tests
Changeset: 2af1627dacead03f5de18fae95e7d7cf3d35c34d
comment:39 Changed 6 years ago by Nick Draper
re #8924 resolve a proxy problem
Changeset: 8212a2aab27617211f209595138c4b969da253ec
comment:40 Changed 6 years ago by Nick Draper
re #8924 changes to make use of #10550 reduction of wrapper logging
Changeset: 6825f8440d1d7ebe950025328dcaa9f3b7d5a828
comment:41 Changed 6 years ago by Nick Draper
- Status changed from inprogress to verify
- Resolution set to fixed
To Test
This branch has changes to the mantid and systemTest repos, make sure to merge them both. For Macro users, this just means doing git test start and git test pass on both repos
Note: This process is rather slow in debug ~12 secs per run of DownloadInstrument, but is nice and quick in release ~0.4sec.
These notes are long - but don't be put off they are just thorough.
- For these tests I will refer to the appdata directory
- Win %APPDATA%/mantidproject/mantid/instrument
- Linux/Mac ~/.mantid/instrument
- Install directory
- This is the instrument directory wherever Mantid was installed
setup
- Check on develop builds that the unit tests and system tests have passed
- If you are using mantid from an installed package you don't need to do this: but if you are coming from a dev environment then you need to go to your Mantid.Properties file and set:
UpdateInstrumentDefinitions.OnStartup = 0 to UpdateInstrumentDefinitions.OnStartup = 1
- Before starting Mantid go and look for appdata, this directory may not exist (yet).
Check both ways of starting the instrument download
- Build mantid and startup mantidplot
- You should get a single additional line (at notice) in the log after the welcome to mantidplot it should say one of:
- All instrument definitions up to date
- Downloading ?? files from the instrument repository
- An error message, but that is a fail really.
- The appdata directory, should be created and gain at least a github.json file, and the xml file for any instruments downloaded.
- You should get a single additional line (at notice) in the log after the welcome to mantidplot it should say one of:
- Run the DownloadInstrument Algorithm. This will do the same thing, but nothing should downloaded this time, as you should be up to date.
- This should be relatively quick is it checks if there have been changes since the date of the github.json file. From now on we will bypass this check for the rest of the testing by deleting the appdata/gihub.json file at various points.
Extra files present in local instrument dirs
- Go to the install directory and take a copy of LOQ_Definition_20020226-.xml - do not change it other than the file name, which you should change to LOQCopy_Definition_20020226-.xml
- delete appdata/github.json
- Close and restart mantidplot. Nothing should be downloaded, no new .xml files should appear in appdata, the copy file should remain in the install directory.
Updates to instrument definitions - github is the master you will just fake changes to force a download
- Without closing mantidplot. Go to the install directory and fiddle with an instrument definition, I suggest you use LOQ_Definition_20020226-.xml and change line 122 to look like this (just a change of rot from 90.0 to 0.0) - DO NOT commit this change to master - see tear down
<location ><facing rot="0.0"/> </location> <!-- bottom module looking from source to sample-->
- Without closing mantidplot. Load LOQ48098 (in autotest data). look at the instrument view, the bottom part of the high angle bank should be missing.
- delete appdata/github.json
- Close and restart mantidplot. You should get a message that at least one file has downloaded.
- Load LOQ48098 (in autotest data). look at the instrument view, the bottom part of the high angle bank will have returned.
- In the appdata directory you will have a nice clean copy of the LOQ_Definition_20020226-.xml file that will have downloaded from the instrument repository.
- Got to that file in the appdata directory and edit to make a visible change (the same edit as before will do).
- delete appdata/github.json
- Close and restart mantidplot. You should get a message that at least one file has downloaded.
- The version of the file in appdata should have been overwritten, and your change reverted.
New instrument definitions
- Close Mantidplot, delete LOQ_Definition_20020226-.xml from both appdata and install directory.
- delete appdata/github.json
- Open mantidplot. You should be told that 1 file has been downloaded.
- In the appdata directory you will have a nice clean copy of the LOQ_Definition_20020226-.xml file that will have downloaded from the instrument repository.
- Load LOQ48098 and check the instrument view to prove that the instrument file was loaded.
Tear down
- Go to the install directory change the name of LOQCopy_Definition_20020226-.xml back to LOQ_Definition_20020226-.xml.
- run a git status, make sure you are not about to merge any changes here into master, if so revert those files.
- set UpdateInstrumentDefinitions.OnStartup = 0 in your mantid.properties file
- Clear out the appdata dir if you want to
comment:42 Changed 6 years ago by Nick Draper
re #8924 mac fix for old std libraries
Changeset: dc0ef46e3a164a84b0f9a82ef70e7fef0ae59286
comment:43 Changed 6 years ago by Nick Draper
re #8924 resolve warning
Changeset: 345129828f0e34ed96beb7cd104d0b3716796fba
comment:44 Changed 6 years ago by Martyn Gigg
Fix fstrea::open call for older OS X library.
Refs #8924
Changeset: 707d11573dc873944e29463f662a7ad6d4b19505
comment:45 Changed 6 years ago by Harry Jeffery
Working as expected on Ubuntu 14.04
comment:46 Changed 6 years ago by Owen Arnold
Mantid on OSX is failing to start up and crashing as a result of these changes.
Here's the crash report.
0 com.apple.CoreFoundation 0x00007fff8be2804a CFArrayGetCount + 10 1 libMantidKernel.dylib 0x000000010212f559 Mantid::Kernel::proxyInformationFromPac(__CFDictionary const*, std::string const&, Mantid::Kernel::Logger&) + 585 2 libMantidKernel.dylib 0x000000010212eea1 Mantid::Kernel::NetworkProxy::getHttpProxy(std::string const&) + 81 3 libMantidKernel.dylib 0x0000000101ebf03d Mantid::Kernel::ConfigServiceImpl::getProxy(std::string const&) + 717 4 libMantidKernel.dylib 0x0000000101f0d0bc Mantid::Kernel::InternetHelper::sendHTTPSRequest(std::string const&, std::ostream&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) + 1244 5 libMantidKernel.dylib 0x0000000101f0c967 Mantid::Kernel::InternetHelper::sendRequest(std::string const&, std::ostream&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) + 71 6 libMantidKernel.dylib 0x0000000101f0c8a5 Mantid::Kernel::InternetHelper::downloadFile(std::string const&, std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) + 165 7 libMantidDataHandling.dylib 0x0000000111daad9e Mantid::DataHandling::DownloadInstrument::doDownloadFile(std::string const&, std::string const&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&) + 46 8 libMantidDataHandling.dylib 0x0000000111da9758 Mantid::DataHandling::DownloadInstrument::processRepository() + 680 9 libMantidDataHandling.dylib 0x0000000111da899b Mantid::DataHandling::DownloadInstrument::exec() + 315 10 libMantidAPI.dylib 0x0000000102b0e4c5 Mantid::API::Algorithm::execute() + 1109 11 libMantidAPI.dylib 0x0000000102b1888c Mantid::API::Algorithm::executeAsyncImpl(Poco::Void const&) + 28 12 libMantidAPI.dylib 0x0000000102b45095 Mantid::API::AlgorithmProxy::executeAsyncImpl(Poco::Void const&) + 37 13 libMantidAPI.dylib 0x0000000102b45bf2 Poco::ActiveRunnable<bool, Poco::Void, Mantid::API::AlgorithmProxy>::run() + 98 14 libPocoFoundation.16.dylib 0x0000000106ddf533 Poco::PooledThread::run() + 111 15 libPocoFoundation.16.dylib 0x0000000106ddca40 Poco::ThreadImpl::runnableEntry(void*) + 100 16 libsystem_c.dylib 0x00007fff88aae772 _pthread_start + 327 17 libsystem_c.dylib 0x00007fff88a9b1a1 thread_start + 13
comment:47 Changed 6 years ago by Owen Arnold
- Status changed from verify to reopened
- Resolution fixed deleted
comment:48 Changed 6 years ago by Nick Draper
- Status changed from reopened to inprogress
re #8924 forsce a better proxy url to solve mac crashes
Changeset: 84c74490cbb0e6baec7cd44a2aefc0ec963f2e4d
comment:49 Changed 6 years ago by Nick Draper
re #8924 forsce a better proxy url to solve mac crashes
Changeset: 3130053611a50abd1dac866f6c8ed2705b0c2f74
comment:50 Changed 6 years ago by Nick Draper
- Status changed from inprogress to verify
- Resolution set to fixed
Ready for retest on Mac
comment:51 Changed 6 years ago by Owen Arnold
- Status changed from verify to verifying
- Tester set to Owen Arnold
comment:52 Changed 6 years ago by Nick Draper
- Status changed from verifying to closed
Merge branch 'feature/8924_Intrument_file_repo' of github.com:mantidproject/mantid into feature/8924_Intrument_file_repo
Full changeset: 9e2b70d0aa03b9ec63b6d65b88dc6e03412f1d0f
comment:53 Changed 6 years ago by Nick Draper
Merge branch 'feature/10550_alg_startup_logging' of github.com:mantidproject/mantid into feature/8924_Intrument_file_repo
Full changeset: 36a2cfe876e02c84a8fb4f797cbcecc2512b2668
comment:54 Changed 6 years ago by Nick Draper
Merge branch 'master' of github.com:mantidproject/mantid into feature/8924_Intrument_file_repo
Full changeset: b21996104cfded93ce61f21b9f940968708fb3c1
comment:55 Changed 6 years ago by Owen Arnold
Merge remote-tracking branch 'origin/feature/8924_Intrument_file_repo'
Full changeset: 837047fad4164e48daef6aba57930767a757ecf8
comment:56 Changed 6 years ago by Owen Arnold
Minor point. Your tests define and create a Fake not a Mock. It's a useful distinction and the name is confusing.
comment:57 Changed 6 years ago by Martyn Gigg
Merge branch 'feature/8924_Intrument_file_repo'
Full changeset: f2a8fcba41a0f7cedb1abc4313ea9591ef4ff6cf
comment:59 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 9767