Ticket #10870 (closed: fixed)
Snapshot and move system tests to mantid repository
Reported by: | Martyn Gigg | Owned by: | Martyn Gigg |
---|---|---|---|
Priority: | major | Milestone: | Release 3.4 |
Component: | Framework | Keywords: | Maintenance |
Cc: | Blocked By: | ||
Blocking: | #10868 | Tester: | Owen Arnold |
Description
Take a snapshot of the code portions of the system test repository and add it to the main mantid repository using the layout described in https://github.com/mantidproject/documents/blob/master/Design/SystemTestRepoMerge.md.
For the moment the data links will be invalid.
Change History
comment:2 Changed 6 years ago by Martyn Gigg
- Status changed from assigned to inprogress
Move performance test code to Testing directory.
The system tests will be added here so it makes sense to have all of this in one place. Refs #10870
Changeset: ffa2bbf490fc0d66d997252348285cf975acca4b
comment:3 Changed 6 years ago by Martyn Gigg
Initial dump of system test supporting framework
Some modules were dropped from the other repository as they were not used, namely
- emailreporter.py
- sqlresultreporter.py
- StressTests directory.
Fixes for path changes etc will still be required for this to work. Refs #10870
Changeset: 98041ca89fda676a096bd38e1a6e3a20961a52ba
comment:4 Changed 6 years ago by Martyn Gigg
Add an external data target for the system tests
This is not added as a dependency and must be explicitly invoked. Refs #10870
Changeset: 7b31f84ba183d238b29a13cb4716f74a655a0c17
comment:5 Changed 6 years ago by Martyn Gigg
Update module paths for new structure
CMake now writes out two text files containing the location of the data files and the requested default save directory. The save directory now defaults to outside the source tree making it easier to clear away. Refs #10870
Changeset: 110a9b33e514bb858563d24df295c8aef8327409
comment:6 Changed 6 years ago by Martyn Gigg
Update InstallerTests wrapper script for new layout.
It can now be passed a directory to search for packages rather than assuming it should look in the current directory. Refs #10870
Changeset: 54f0754b27412464fbe494d83117672e080be5b7
comment:7 Changed 6 years ago by Martyn Gigg
Create version_tested.log file in binary directory.
Refs #10870
Changeset: 1d77c360a9639df973b395bd627b92654fd62a68
comment:8 Changed 6 years ago by Martyn Gigg
Merge branch 'master' into 10870_fold_in_systemtest_code
Conflicts:
Code/Mantid/Build/CMake/CommonSetup.cmake
Refs #10870
Changeset: 5a4a6ec62697ceb171b53f38f943b0375de7afe6
comment:9 Changed 6 years ago by Martyn Gigg
Update buildscripts for new setup
Changes:
- any references to develop have been removed
- pull_request builds will also build packages
Refs #10870
Changeset: 2c47c9efb330b451e4ce7534fec9a400fec2e012
comment:10 Changed 6 years ago by Martyn Gigg
Run the system tests on RHEL6 pull request builds.
Refs #10870
Changeset: 316d13e2416fcf1cce09dbdb0ebcb31c917a2e82
comment:11 Changed 6 years ago by Martyn Gigg
Remove screenshots from MantidPlot tests
They proved confusing when running the build scripts as they looked like they had come from building the documentation and had gone to the wrong directory. Refs #10870
Changeset: 9ade93ddf9a7e4d419291d3bfe65e5b3c326abac
comment:12 Changed 6 years ago by Martyn Gigg
Move contents of TestingTools to Testing/Tools directory
Refs #10870
Changeset: e8354ad3233ca8cc5e5eff606c24c0d0ffcfbafd
comment:13 Changed 6 years ago by Martyn Gigg
Clean up a screenshot after the test
Refs #10870
Changeset: be9daee5756776a78759983629b2d8d2c9b43550
comment:14 Changed 6 years ago by Martyn Gigg
Use a driver script for the system tests
Avoids duplication of job configuration on the server. Refs #10870
Changeset: f1ba93a9240bf912a93d1b422e40be3f5e57197f
comment:15 Changed 6 years ago by Martyn Gigg
Add cmake option to only create data targets
This is useful for the standalone system tests that won't necessarily have all of the Mantid build dependencies installed. Refs #10870
Changeset: c1bc081d8715e77120ae39eaa1f178f814b70f3f
comment:16 Changed 6 years ago by Martyn Gigg
comment:17 Changed 6 years ago by Martyn Gigg
Reorganise the windows built script so the order matches the Unix one
Refs #10870
Changeset: 5db99b85456a8054564625b06025536770a344b1
comment:18 Changed 6 years ago by Martyn Gigg
Alter the mantidinstaller script to be able to run standalone.
It already contains the logic to install/uninstall Mantid so we don't want to duplicate it for the Windows build script Refs #10870
Changeset: 5c96dae1ac54cac86671b32e471c8ec2ae8ba81c
comment:19 Changed 6 years ago by Martyn Gigg
Merge parts of script that deal with the clean build
Refs #10870
Changeset: 2c544868a426d78d9b8f331f93fb4705db6af5f6
comment:20 Changed 6 years ago by Martyn Gigg
Add documentation tests to Windows pull request build
Refs #10870
Changeset: 007bf35428c070070286297c4aa4c0d45f71bd1c
comment:21 Changed 6 years ago by Martyn Gigg
- Status changed from inprogress to verify
- Resolution set to fixed
This is being verified as pull request #216.
comment:22 Changed 6 years ago by Martyn Gigg
Move default external data location to HOME
Refs #10870
Changeset: 759f9820901cce5155b687597cca0ec2e4700ed6
comment:23 Changed 6 years ago by Martyn Gigg
Temporary rule to remove build directory if old layout in use
Refs #10870
Changeset: d0c3785bb25d53d6e58aeff38b5da6cc94f7f627
comment:24 Changed 6 years ago by Martyn Gigg
comment:25 Changed 6 years ago by Martyn Gigg
Don't use readlink in buildscript.
The OS X version doesn't support the same options as GNU readline Refs #10870
Changeset: bb2726ec76c284b76f552a543c68155446d8e5ea
comment:26 Changed 6 years ago by Martyn Gigg
Jenkins, retest this please.
comment:27 Changed 6 years ago by Martyn Gigg
Ignore grep return code and compare the output
The build scripts are run such that any non-zero return code exits the script immediately. Refs #10870
Changeset: 716b62eda1327be30119a2de8a1b54e9455d87ca
comment:28 Changed 6 years ago by Martyn Gigg
Fix bugs with cleaning logic in Windows build script
Refs #10870
Changeset: 824839e0ec3a27bb154bbc8a7a06312432f498ea
comment:29 Changed 6 years ago by Martyn Gigg
Fix bug in setting PATH variable.
It needs to come after choosing the configuration. Refs #10870
Changeset: 39f854ad28f30d706831a8146d69756b0a89380a
comment:30 Changed 6 years ago by Martyn Gigg
Follow the standard convention for the system tests results filename.
Refs #10870
Changeset: 6dc3ae2935ee63f910173d777c6c72d334742467
comment:31 Changed 6 years ago by Martyn Gigg
Fix path to installer python script on Windows
Refs #10870
Changeset: 707889044120cc2b128adf863579d6887ef0b2b0
comment:32 Changed 6 years ago by Martyn Gigg
Apply OSX > 10.8 fix to system test scripts
Refs #10870
Changeset: dc21b4d134f726c91e615d25f93d9d4adfec6e9f
comment:33 Changed 6 years ago by Martyn Gigg
comment:34 Changed 6 years ago by Martyn Gigg
Jenkins, retest this please
comment:35 Changed 6 years ago by Martyn Gigg
Jenkins, retest this please.
comment:36 Changed 6 years ago by Martyn Gigg
Use delayed expansion to get error level behaviour correct.
Also make sure the data paths are setup correct for the doc tests. Refs #10870
Changeset: 64e9872f79f8c1a77986ed888b76ce88397587e2
comment:37 Changed 6 years ago by Martyn Gigg
Add a systemtests.bat script for driving system test jobs.
Refs #10870
Changeset: 7fdedb08f933f9cdf0fa2ab3e6b67d32847bafff
comment:38 Changed 6 years ago by Martyn Gigg
Fix a doctest output to work cross platform.
On windows the numpy shape array was printed as (100L, 100L), which caused the string comparison to fail. Refs #10870
Changeset: de6037d8fd8335b7cbe6a19a26063445ca43640b
comment:39 Changed 6 years ago by Martyn Gigg
Fix some issues with the Windows build script
- CMake path is set once
- Backslash chars are removed from the data search paths before addition to properties file.
- Package uninstall happens regardless of exit status.
Refs #10870
Changeset: 979b8125234fdbf88662ebfa464138ae938a240d
comment:40 Changed 6 years ago by Martyn Gigg
Fix some issues with the Windows build script
- CMake path is set once
- Backslash chars are removed from the data search paths before addition to properties file.
- Package uninstall happens regardless of exit status.
Refs #10870 [skip ci]
Changeset: 1f6235982a725df20f6954f429fd396049eb73c0
comment:41 Changed 6 years ago by Martyn Gigg
@peterfpeterson @rosswhitfield @sic @OwenArnold @Anders-Markvardsen - I wouldn't mind a few eyes on this one.
My intention here was to get the scripts in order to run the system tests and doctests on the pull request. For sake of ease in testing I have added a single system test: HRPD as it hasn't changed in a long time. I would like to push this through and verify everything is working before pulling in everything (which should be much simpler). Let me know what you think.
comment:42 Changed 6 years ago by Martyn Gigg
Jenkins, retest this please.
comment:43 Changed 6 years ago by Ross Whitfield
:sparkles: From what I can make of all the changes and what I tested it looks good. :+1:
comment:44 Changed 6 years ago by Pete Peterson
It works, but there is a fair amount of gymnastics to actually run the system tests as a normal developer. There should be a follow on bit of work to copy a templated version of runSystemTests.py into the CMAKE_BINARY_DIR which would also eliminate some of the command line arguments.
comment:45 Changed 6 years ago by Martyn Gigg
I agree but it felt like this would turn into a never ending list of improvements so I stopped at porting what we currently have.
comment:46 Changed 6 years ago by Pete Peterson
Can we have couple more up-votes from @sic @OwenArnold @Anders-Markvardsen so this can be merged?
comment:47 Changed 6 years ago by Alex Buts
- Status changed from verify to verifying
- Tester set to Alex Buts
I need this ticket so would happily merge disregarding voting meeting as the test provided as the nucleus for other tests runs fine for me, but I can not compile Mantid without error on this branch.
In addition to trivial conflict with cmake, the build fails with error (in addition to missing external source):
D:\Data\Mantid_GIT_test\Code\builds\br_10870_fold_in_systemtest_code\Vates\ParaviewPlugins\ParaViewWidgets\RebinningTransformObjectPanel\moc_RebinningTransformObjectPanel.cxx(10): fatal error C1083: Cannot open include file: '../../../../../../Mantid/Vates/ParaviewPlugins/ParaViewWidgets/RebinningTransformObjectPanel/RebinningTransformObjectPanel.h': No such file or directory
Despite I've successfully merged master into the branch to resolve cmake conflicts and merge was fine.
There are some more complicated merge problems here.
comment:48 Changed 6 years ago by Alex Buts
- Status changed from verifying to reopened
- Resolution fixed deleted
comment:49 Changed 6 years ago by Martyn Gigg
Any comments @sic @OwenArnold @Anders-Markvardsen?
comment:50 Changed 6 years ago by Pete Peterson
@martyngigg there is a small merge conflict in the main build script.
comment:51 Changed 6 years ago by Martyn Gigg
Thanks @peterfpeterson. I'll take a look what has happened
comment:52 Changed 6 years ago by Martyn Gigg
- Status changed from reopened to inprogress
Merge branch 'master' into 10870_fold_in_systemtest_code
Conflicts:
Code/Mantid/Build/Jenkins/buildscript
Refs #10870
Changeset: 49f750b43504b9d3df785b20f1d7a7d4dbdcc853
comment:53 Changed 6 years ago by Martyn Gigg
Jenkins, retest this please.
comment:54 Changed 6 years ago by Martyn Gigg
Jenkins, retest this please.
comment:55 Changed 6 years ago by Owen Arnold
- Status changed from inprogress to verifying
- Tester changed from Alex Buts to Owen Arnold
comment:56 Changed 6 years ago by Owen Arnold
Jenkins retest this please
comment:57 Changed 6 years ago by Owen Arnold
Worked as expected. Looks ready to go pending the update to the ORNL build servers.
comment:58 Changed 6 years ago by Owen Arnold
- Status changed from verifying to closed
Merge pull request #216 from mantidproject/10870_fold_in_systemtest_code
Pull in the system test code and tests
Full changeset: 8a297a169b7913761b21fdbfe135416fb9e55dca
comment:59 Changed 5 years ago by Nick Draper
- Resolution set to fixed
Somehow these slipped through without a resolution. Set to Fixed.
comment:60 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 11712