Snapshot and move system tests to mantid repository

[Martyn Gigg]

Take a snapshot of the code portions of the system test repository and add it to the main mantid repository using the layout described in ​https://github.com/mantidproject/documents/blob/master/Design/SystemTestRepoMerge.md.

For the moment the data links will be invalid.

[Martyn Gigg]

Move performance test code to Testing directory.

The system tests will be added here so it makes sense to have all of this in one place. Refs #10870

Changeset: ffa2bbf490fc0d66d997252348285cf975acca4b

[Martyn Gigg]

Initial dump of system test supporting framework

Some modules were dropped from the other repository as they were not used, namely

• emailreporter.py
• sqlresultreporter.py
• StressTests directory.

Fixes for path changes etc will still be required for this to work. Refs #10870

Changeset: 98041ca89fda676a096bd38e1a6e3a20961a52ba

[Martyn Gigg]

Add an external data target for the system tests

This is not added as a dependency and must be explicitly invoked. Refs #10870

Changeset: 7b31f84ba183d238b29a13cb4716f74a655a0c17

[Martyn Gigg]

Update module paths for new structure

CMake now writes out two text files containing the location of the data files and the requested default save directory. The save directory now defaults to outside the source tree making it easier to clear away. Refs #10870

Changeset: 110a9b33e514bb858563d24df295c8aef8327409

[Martyn Gigg]

Update InstallerTests wrapper script for new layout.

It can now be passed a directory to search for packages rather than assuming it should look in the current directory. Refs #10870

Changeset: 54f0754b27412464fbe494d83117672e080be5b7

[Martyn Gigg]

Create version_tested.log file in binary directory.

Refs #10870

Changeset: 1d77c360a9639df973b395bd627b92654fd62a68

[Martyn Gigg]

Merge branch 'master' into 10870_fold_in_systemtest_code

Conflicts:

Code/Mantid/Build/CMake/CommonSetup.cmake

Refs #10870

Changeset: 5a4a6ec62697ceb171b53f38f943b0375de7afe6

[Martyn Gigg]

Update buildscripts for new setup

Changes:

• any references to develop have been removed
• pull_request builds will also build packages

Refs #10870

Changeset: 2c47c9efb330b451e4ce7534fec9a400fec2e012

[Martyn Gigg]

Run the system tests on RHEL6 pull request builds.

Refs #10870

Changeset: 316d13e2416fcf1cce09dbdb0ebcb31c917a2e82

[Martyn Gigg]

Remove screenshots from MantidPlot tests

They proved confusing when running the build scripts as they looked like they had come from building the documentation and had gone to the wrong directory. Refs #10870

Changeset: 9ade93ddf9a7e4d419291d3bfe65e5b3c326abac

[Martyn Gigg]

Move contents of TestingTools to Testing/Tools directory

Refs #10870

Changeset: e8354ad3233ca8cc5e5eff606c24c0d0ffcfbafd

[Martyn Gigg]

Clean up a screenshot after the test

Refs #10870

Changeset: be9daee5756776a78759983629b2d8d2c9b43550

[Martyn Gigg]

Use a driver script for the system tests

Avoids duplication of job configuration on the server. Refs #10870

Changeset: f1ba93a9240bf912a93d1b422e40be3f5e57197f

[Martyn Gigg]

Add cmake option to only create data targets

This is useful for the standalone system tests that won't necessarily have all of the Mantid build dependencies installed. Refs #10870

Changeset: c1bc081d8715e77120ae39eaa1f178f814b70f3f

[Martyn Gigg]

Add first test to try things out.

Refs #10870

Changeset: 1752fc2fe186490921b7ead2a32be003dedcd4db

[Martyn Gigg]

Reorganise the windows built script so the order matches the Unix one

Refs #10870

Changeset: 5db99b85456a8054564625b06025536770a344b1

[Martyn Gigg]

Alter the mantidinstaller script to be able to run standalone.

It already contains the logic to install/uninstall Mantid so we don't want to duplicate it for the Windows build script Refs #10870

Changeset: 5c96dae1ac54cac86671b32e471c8ec2ae8ba81c

[Martyn Gigg]

Merge parts of script that deal with the clean build

Refs #10870

Changeset: 2c544868a426d78d9b8f331f93fb4705db6af5f6

[Martyn Gigg]

Add documentation tests to Windows pull request build

Refs #10870

Changeset: 007bf35428c070070286297c4aa4c0d45f71bd1c

[Martyn Gigg]

This is being verified as ​pull request #216.

[Martyn Gigg]

Move default external data location to HOME

Refs #10870

Changeset: 759f9820901cce5155b687597cca0ec2e4700ed6

[Martyn Gigg]

Temporary rule to remove build directory if old layout in use

Refs #10870

Changeset: d0c3785bb25d53d6e58aeff38b5da6cc94f7f627

[Martyn Gigg]

Fix errors in build script

Refs #10870

Changeset: efb1f91b6cf7c913f64acf4ae041aef22dab019a

[Martyn Gigg]

Don't use readlink in buildscript.

The OS X version doesn't support the same options as GNU readline Refs #10870

Changeset: bb2726ec76c284b76f552a543c68155446d8e5ea

[Martyn Gigg]

Jenkins, retest this please.

[Martyn Gigg]

Ignore grep return code and compare the output

The build scripts are run such that any non-zero return code exits the script immediately. Refs #10870

Changeset: 716b62eda1327be30119a2de8a1b54e9455d87ca

[Martyn Gigg]

Fix bugs with cleaning logic in Windows build script

Refs #10870

Changeset: 824839e0ec3a27bb154bbc8a7a06312432f498ea

[Martyn Gigg]

Fix bug in setting PATH variable.

It needs to come after choosing the configuration. Refs #10870

Changeset: 39f854ad28f30d706831a8146d69756b0a89380a

[Martyn Gigg]

Follow the standard convention for the system tests results filename.

Refs #10870

Changeset: 6dc3ae2935ee63f910173d777c6c72d334742467

[Martyn Gigg]

Fix path to installer python script on Windows

Refs #10870

Changeset: 707889044120cc2b128adf863579d6887ef0b2b0

[Martyn Gigg]

Apply OSX > 10.8 fix to system test scripts

Refs #10870

Changeset: dc21b4d134f726c91e615d25f93d9d4adfec6e9f

[Martyn Gigg]

Fix path to mantidinstaller script

Refs #10870

Changeset: 59d199fd5cce1f8675e46bdc949d03b7a168ee29

[Martyn Gigg]

Jenkins, retest this please

[Martyn Gigg]

Jenkins, retest this please.

[Martyn Gigg]

Use delayed expansion to get error level behaviour correct.

Also make sure the data paths are setup correct for the doc tests. Refs #10870

Changeset: 64e9872f79f8c1a77986ed888b76ce88397587e2

[Martyn Gigg]

Add a systemtests.bat script for driving system test jobs.

Refs #10870

Changeset: 7fdedb08f933f9cdf0fa2ab3e6b67d32847bafff

[Martyn Gigg]

Fix a doctest output to work cross platform.

On windows the numpy shape array was printed as (100L, 100L), which caused the string comparison to fail. Refs #10870

Changeset: de6037d8fd8335b7cbe6a19a26063445ca43640b

[Martyn Gigg]

Fix some issues with the Windows build script

• CMake path is set once
• Backslash chars are removed from the data search paths before addition to properties file.
• Package uninstall happens regardless of exit status.

Refs #10870

Changeset: 979b8125234fdbf88662ebfa464138ae938a240d

[Martyn Gigg]

Fix some issues with the Windows build script

• CMake path is set once
• Backslash chars are removed from the data search paths before addition to properties file.
• Package uninstall happens regardless of exit status.

Refs #10870 [skip ci]

Changeset: 1f6235982a725df20f6954f429fd396049eb73c0

[Martyn Gigg]

@peterfpeterson @rosswhitfield @sic @OwenArnold @Anders-Markvardsen - I wouldn't mind a few eyes on this one.

My intention here was to get the scripts in order to run the system tests and doctests on the pull request. For sake of ease in testing I have added a single system test: HRPD as it hasn't changed in a long time. I would like to push this through and verify everything is working before pulling in everything (which should be much simpler). Let me know what you think.

[Martyn Gigg]

Jenkins, retest this please.

[Ross Whitfield]

:sparkles: From what I can make of all the changes and what I tested it looks good. :+1:

[Pete Peterson]

It works, but there is a fair amount of gymnastics to actually run the system tests as a normal developer. There should be a follow on bit of work to copy a templated version of runSystemTests.py into the CMAKE_BINARY_DIR which would also eliminate some of the command line arguments.

[Martyn Gigg]

I agree but it felt like this would turn into a never ending list of improvements so I stopped at porting what we currently have.

[Pete Peterson]

Can we have couple more up-votes from @sic @OwenArnold @Anders-Markvardsen so this can be merged?

[Alex Buts]

I need this ticket so would happily merge disregarding voting meeting as the test provided as the nucleus for other tests runs fine for me, but I can not compile Mantid without error on this branch.

In addition to trivial conflict with cmake, the build fails with error (in addition to missing external source):

D:\Data\Mantid_GIT_test\Code\builds\br_10870_fold_in_systemtest_code\Vates\ParaviewPlugins\ParaViewWidgets\RebinningTransformObjectPanel\moc_RebinningTransformObjectPanel.cxx(10): fatal error C1083: Cannot open include file: '../../../../../../Mantid/Vates/ParaviewPlugins/ParaViewWidgets/RebinningTransformObjectPanel/RebinningTransformObjectPanel.h': No such file or directory

Despite I've successfully merged master into the branch to resolve cmake conflicts and merge was fine.

There are some more complicated merge problems here.

[Martyn Gigg]

Any comments @sic @OwenArnold @Anders-Markvardsen?

[Pete Peterson]

@martyngigg there is a small merge conflict in the main build script.

[Martyn Gigg]

Thanks @peterfpeterson. I'll take a look what has happened

[Martyn Gigg]

Merge branch 'master' into 10870_fold_in_systemtest_code

Conflicts:

Code/Mantid/Build/Jenkins/buildscript

Refs #10870

Changeset: 49f750b43504b9d3df785b20f1d7a7d4dbdcc853

[Martyn Gigg]

Jenkins, retest this please.

[Martyn Gigg]

Jenkins, retest this please.

[Owen Arnold]

Jenkins retest this please

[Owen Arnold]

Worked as expected. Looks ready to go pending the update to the ORNL build servers.

[Owen Arnold]

Merge pull request #216 from mantidproject/10870_fold_in_systemtest_code

Pull in the system test code and tests

Full changeset: ​8a297a169b7913761b21fdbfe135416fb9e55dca

[Nick Draper]

Somehow these slipped through without a resolution. Set to Fixed.

[Stuart Campbell]

This ticket has been transferred to github issue ​11712