Ticket #10870 (closed: fixed)

Opened 6 years ago

Last modified 5 years ago

Snapshot and move system tests to mantid repository

Reported by: Martyn Gigg Owned by: Martyn Gigg
Priority: major Milestone: Release 3.4
Component: Framework Keywords: Maintenance
Cc: Blocked By:
Blocking: #10868 Tester: Owen Arnold

Description

Take a snapshot of the code portions of the system test repository and add it to the main mantid repository using the layout described in https://github.com/mantidproject/documents/blob/master/Design/SystemTestRepoMerge.md.

For the moment the data links will be invalid.

Change History

comment:1 Changed 6 years ago by Martyn Gigg

  • Status changed from new to assigned

comment:2 Changed 6 years ago by Martyn Gigg

  • Status changed from assigned to inprogress

Move performance test code to Testing directory.

The system tests will be added here so it makes sense to have all of this in one place. Refs #10870

Changeset: ffa2bbf490fc0d66d997252348285cf975acca4b

comment:3 Changed 6 years ago by Martyn Gigg

Initial dump of system test supporting framework

Some modules were dropped from the other repository as they were not used, namely

  • emailreporter.py
  • sqlresultreporter.py
  • StressTests directory.

Fixes for path changes etc will still be required for this to work. Refs #10870

Changeset: 98041ca89fda676a096bd38e1a6e3a20961a52ba

comment:4 Changed 6 years ago by Martyn Gigg

Add an external data target for the system tests

This is not added as a dependency and must be explicitly invoked. Refs #10870

Changeset: 7b31f84ba183d238b29a13cb4716f74a655a0c17

comment:5 Changed 6 years ago by Martyn Gigg

Update module paths for new structure

CMake now writes out two text files containing the location of the data files and the requested default save directory. The save directory now defaults to outside the source tree making it easier to clear away. Refs #10870

Changeset: 110a9b33e514bb858563d24df295c8aef8327409

comment:6 Changed 6 years ago by Martyn Gigg

Update InstallerTests wrapper script for new layout.

It can now be passed a directory to search for packages rather than assuming it should look in the current directory. Refs #10870

Changeset: 54f0754b27412464fbe494d83117672e080be5b7

comment:7 Changed 6 years ago by Martyn Gigg

Create version_tested.log file in binary directory.

Refs #10870

Changeset: 1d77c360a9639df973b395bd627b92654fd62a68

comment:8 Changed 6 years ago by Martyn Gigg

Merge branch 'master' into 10870_fold_in_systemtest_code

Conflicts:

Code/Mantid/Build/CMake/CommonSetup.cmake

Refs #10870

Changeset: 5a4a6ec62697ceb171b53f38f943b0375de7afe6

comment:9 Changed 6 years ago by Martyn Gigg

Update buildscripts for new setup

Changes:

  • any references to develop have been removed
  • pull_request builds will also build packages

Refs #10870

Changeset: 2c47c9efb330b451e4ce7534fec9a400fec2e012

comment:10 Changed 6 years ago by Martyn Gigg

Run the system tests on RHEL6 pull request builds.

Refs #10870

Changeset: 316d13e2416fcf1cce09dbdb0ebcb31c917a2e82

comment:11 Changed 6 years ago by Martyn Gigg

Remove screenshots from MantidPlot tests

They proved confusing when running the build scripts as they looked like they had come from building the documentation and had gone to the wrong directory. Refs #10870

Changeset: 9ade93ddf9a7e4d419291d3bfe65e5b3c326abac

comment:12 Changed 6 years ago by Martyn Gigg

Move contents of TestingTools to Testing/Tools directory

Refs #10870

Changeset: e8354ad3233ca8cc5e5eff606c24c0d0ffcfbafd

comment:13 Changed 6 years ago by Martyn Gigg

Clean up a screenshot after the test

Refs #10870

Changeset: be9daee5756776a78759983629b2d8d2c9b43550

comment:14 Changed 6 years ago by Martyn Gigg

Use a driver script for the system tests

Avoids duplication of job configuration on the server. Refs #10870

Changeset: f1ba93a9240bf912a93d1b422e40be3f5e57197f

comment:15 Changed 6 years ago by Martyn Gigg

Add cmake option to only create data targets

This is useful for the standalone system tests that won't necessarily have all of the Mantid build dependencies installed. Refs #10870

Changeset: c1bc081d8715e77120ae39eaa1f178f814b70f3f

comment:16 Changed 6 years ago by Martyn Gigg

Add first test to try things out.

Refs #10870

Changeset: 1752fc2fe186490921b7ead2a32be003dedcd4db

comment:17 Changed 6 years ago by Martyn Gigg

Reorganise the windows built script so the order matches the Unix one

Refs #10870

Changeset: 5db99b85456a8054564625b06025536770a344b1

comment:18 Changed 6 years ago by Martyn Gigg

Alter the mantidinstaller script to be able to run standalone.

It already contains the logic to install/uninstall Mantid so we don't want to duplicate it for the Windows build script Refs #10870

Changeset: 5c96dae1ac54cac86671b32e471c8ec2ae8ba81c

comment:19 Changed 6 years ago by Martyn Gigg

Merge parts of script that deal with the clean build

Refs #10870

Changeset: 2c544868a426d78d9b8f331f93fb4705db6af5f6

comment:20 Changed 6 years ago by Martyn Gigg

Add documentation tests to Windows pull request build

Refs #10870

Changeset: 007bf35428c070070286297c4aa4c0d45f71bd1c

comment:21 Changed 6 years ago by Martyn Gigg

  • Status changed from inprogress to verify
  • Resolution set to fixed

This is being verified as pull request #216.

comment:22 Changed 6 years ago by Martyn Gigg

Move default external data location to HOME

Refs #10870

Changeset: 759f9820901cce5155b687597cca0ec2e4700ed6

comment:23 Changed 6 years ago by Martyn Gigg

Temporary rule to remove build directory if old layout in use

Refs #10870

Changeset: d0c3785bb25d53d6e58aeff38b5da6cc94f7f627

comment:24 Changed 6 years ago by Martyn Gigg

Fix errors in build script

Refs #10870

Changeset: efb1f91b6cf7c913f64acf4ae041aef22dab019a

comment:25 Changed 6 years ago by Martyn Gigg

Don't use readlink in buildscript.

The OS X version doesn't support the same options as GNU readline Refs #10870

Changeset: bb2726ec76c284b76f552a543c68155446d8e5ea

comment:26 Changed 6 years ago by Martyn Gigg

Jenkins, retest this please.

comment:27 Changed 6 years ago by Martyn Gigg

Ignore grep return code and compare the output

The build scripts are run such that any non-zero return code exits the script immediately. Refs #10870

Changeset: 716b62eda1327be30119a2de8a1b54e9455d87ca

comment:28 Changed 6 years ago by Martyn Gigg

Fix bugs with cleaning logic in Windows build script

Refs #10870

Changeset: 824839e0ec3a27bb154bbc8a7a06312432f498ea

comment:29 Changed 6 years ago by Martyn Gigg

Fix bug in setting PATH variable.

It needs to come after choosing the configuration. Refs #10870

Changeset: 39f854ad28f30d706831a8146d69756b0a89380a

comment:30 Changed 6 years ago by Martyn Gigg

Follow the standard convention for the system tests results filename.

Refs #10870

Changeset: 6dc3ae2935ee63f910173d777c6c72d334742467

comment:31 Changed 6 years ago by Martyn Gigg

Fix path to installer python script on Windows

Refs #10870

Changeset: 707889044120cc2b128adf863579d6887ef0b2b0

comment:32 Changed 6 years ago by Martyn Gigg

Apply OSX > 10.8 fix to system test scripts

Refs #10870

Changeset: dc21b4d134f726c91e615d25f93d9d4adfec6e9f

comment:33 Changed 6 years ago by Martyn Gigg

Fix path to mantidinstaller script

Refs #10870

Changeset: 59d199fd5cce1f8675e46bdc949d03b7a168ee29

comment:34 Changed 6 years ago by Martyn Gigg

Jenkins, retest this please

comment:35 Changed 6 years ago by Martyn Gigg

Jenkins, retest this please.

comment:36 Changed 6 years ago by Martyn Gigg

Use delayed expansion to get error level behaviour correct.

Also make sure the data paths are setup correct for the doc tests. Refs #10870

Changeset: 64e9872f79f8c1a77986ed888b76ce88397587e2

comment:37 Changed 6 years ago by Martyn Gigg

Add a systemtests.bat script for driving system test jobs.

Refs #10870

Changeset: 7fdedb08f933f9cdf0fa2ab3e6b67d32847bafff

comment:38 Changed 6 years ago by Martyn Gigg

Fix a doctest output to work cross platform.

On windows the numpy shape array was printed as (100L, 100L), which caused the string comparison to fail. Refs #10870

Changeset: de6037d8fd8335b7cbe6a19a26063445ca43640b

comment:39 Changed 6 years ago by Martyn Gigg

Fix some issues with the Windows build script

  • CMake path is set once
  • Backslash chars are removed from the data search paths before addition to properties file.
  • Package uninstall happens regardless of exit status.

Refs #10870

Changeset: 979b8125234fdbf88662ebfa464138ae938a240d

comment:40 Changed 6 years ago by Martyn Gigg

Fix some issues with the Windows build script

  • CMake path is set once
  • Backslash chars are removed from the data search paths before addition to properties file.
  • Package uninstall happens regardless of exit status.

Refs #10870 [skip ci]

Changeset: 1f6235982a725df20f6954f429fd396049eb73c0

comment:41 Changed 6 years ago by Martyn Gigg

@peterfpeterson @rosswhitfield @sic @OwenArnold @Anders-Markvardsen - I wouldn't mind a few eyes on this one.

My intention here was to get the scripts in order to run the system tests and doctests on the pull request. For sake of ease in testing I have added a single system test: HRPD as it hasn't changed in a long time. I would like to push this through and verify everything is working before pulling in everything (which should be much simpler). Let me know what you think.

comment:42 Changed 6 years ago by Martyn Gigg

Jenkins, retest this please.

comment:43 Changed 6 years ago by Ross Whitfield

:sparkles: From what I can make of all the changes and what I tested it looks good. :+1:

comment:44 Changed 6 years ago by Pete Peterson

It works, but there is a fair amount of gymnastics to actually run the system tests as a normal developer. There should be a follow on bit of work to copy a templated version of runSystemTests.py into the CMAKE_BINARY_DIR which would also eliminate some of the command line arguments.

comment:45 Changed 6 years ago by Martyn Gigg

I agree but it felt like this would turn into a never ending list of improvements so I stopped at porting what we currently have.

comment:46 Changed 6 years ago by Pete Peterson

Can we have couple more up-votes from @sic @OwenArnold @Anders-Markvardsen so this can be merged?

comment:47 Changed 6 years ago by Alex Buts

  • Status changed from verify to verifying
  • Tester set to Alex Buts

I need this ticket so would happily merge disregarding voting meeting as the test provided as the nucleus for other tests runs fine for me, but I can not compile Mantid without error on this branch.

In addition to trivial conflict with cmake, the build fails with error (in addition to missing external source):

D:\Data\Mantid_GIT_test\Code\builds\br_10870_fold_in_systemtest_code\Vates\ParaviewPlugins\ParaViewWidgets\RebinningTransformObjectPanel\moc_RebinningTransformObjectPanel.cxx(10): fatal error C1083: Cannot open include file: '../../../../../../Mantid/Vates/ParaviewPlugins/ParaViewWidgets/RebinningTransformObjectPanel/RebinningTransformObjectPanel.h': No such file or directory

Despite I've successfully merged master into the branch to resolve cmake conflicts and merge was fine.

There are some more complicated merge problems here.

comment:48 Changed 6 years ago by Alex Buts

  • Status changed from verifying to reopened
  • Resolution fixed deleted

comment:49 Changed 6 years ago by Martyn Gigg

Any comments @sic @OwenArnold @Anders-Markvardsen?

comment:50 Changed 6 years ago by Pete Peterson

@martyngigg there is a small merge conflict in the main build script.

comment:51 Changed 6 years ago by Martyn Gigg

Thanks @peterfpeterson. I'll take a look what has happened

comment:52 Changed 6 years ago by Martyn Gigg

  • Status changed from reopened to inprogress

Merge branch 'master' into 10870_fold_in_systemtest_code

Conflicts:

Code/Mantid/Build/Jenkins/buildscript

Refs #10870

Changeset: 49f750b43504b9d3df785b20f1d7a7d4dbdcc853

comment:53 Changed 6 years ago by Martyn Gigg

Jenkins, retest this please.

comment:54 Changed 6 years ago by Martyn Gigg

Jenkins, retest this please.

comment:55 Changed 6 years ago by Owen Arnold

  • Status changed from inprogress to verifying
  • Tester changed from Alex Buts to Owen Arnold

comment:56 Changed 6 years ago by Owen Arnold

Jenkins retest this please

comment:57 Changed 6 years ago by Owen Arnold

Worked as expected. Looks ready to go pending the update to the ORNL build servers.

comment:58 Changed 6 years ago by Owen Arnold

  • Status changed from verifying to closed

Merge pull request #216 from mantidproject/10870_fold_in_systemtest_code

Pull in the system test code and tests

Full changeset: 8a297a169b7913761b21fdbfe135416fb9e55dca

comment:59 Changed 5 years ago by Nick Draper

  • Resolution set to fixed

Somehow these slipped through without a resolution. Set to Fixed.

comment:60 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 11712

Note: See TracTickets for help on using tickets.