Syntax error in any systemtest stops the entire sset from running

[Martyn Gigg]

This is example output from a recent jenkins run of the system tests:

+ python InstallerTests.py -o
FrameworkManager-[Notice] Welcome to Mantid version 3.0.20131128.1550 - Manipulation and Analysis Toolkit for Instrument Data
FrameworkManager-[Notice] Please cite Mantid in your publications using: http://dx.doi.org/10.5286/Software/Mantid
Traceback (most recent call last):
  File "runSystemTests.py", line 73, in <module>
    testsInclude=options.testsInclude, testsExclude=options.testsExclude)
  File "../StressTestFramework/stresstesting.py", line 757, in __init__
    self._tests = self.loadTestsFromDir(test_dir)
  File "../StressTestFramework/stresstesting.py", line 824, in loadTestsFromDir
    tests.extend(self.loadTestsFromModule(os.path.join(test_dir,file)))
  File "../StressTestFramework/stresstesting.py", line 838, in loadTestsFromModule
    mod = imp.load_module(modname, pyfile, filename, ("","",imp.PY_SOURCE))
  File "/home/mantidlog/jenkins/workspace/is_test_rhel6_develop/SystemTests/AnalysisTests/ReflectometryQuickPointDetectorSURF.py", line 16
    try
      ^
SyntaxError: invalid syntax
Starting system tests
Using installer /home/mantidlog/jenkins/workspace/is_test_rhel6_develop/InstallerTesting/mantidnightly-3.0.20131128.1550-1.el6.x86_64.rpm
Installing package '/home/mantidlog/jenkins/workspace/is_test_rhel6_develop/InstallerTesting/mantidnightly-3.0.20131128.1550-1.el6.x86_64.rpm'
Application path /opt/mantidnightly/bin/MantidPlot
MantidPlot directory /opt/mantidnightly/bin
Pointing MANTIDPATH at MantidPlot directory /opt/mantidnightly/bin
3.0.20131128.1550 (Thu, 28 Nov 2013)

It hit a syntax error when loading a test and subsequently the whole thing fell over. We should catch these and report them as errors in the tests themselves.

[Martyn Gigg]

Catch syntax errors when loading tests and report them properly.

This allows the rest of the framework to continue and not come crashing down due to an error in one test. Refs #8533

Changeset: f8b6963a8d23c455d3402e5d05247029df9c6f91

[Martyn Gigg]

Branch: bugfix/8533_catch_syntax_errors in the systemtests repository.

Tester: To test this properly you will need to introduce a syntax error into one of the tests. I would use a package version of Mantid to test this:

• Merge the code
• Edit, for example, ReflectometryQuick.py, to introduce a syntax error
• Try and run the system tests using runSystemTests.py -R Reflectometry You should get a single failure in ReflectometryQuick relating to the syntax error and the other reflectometry tests should pass.

You could also check that the develop system tests are actually running correctly.

[Gesner Passos]

Merge remote-tracking branch 'origin/bugfix/8533_catch_syntax_errors'

Full changeset: ​8758e7da468fff81c6c892958ec370a1b0354863

[Gesner Passos]

It is clear that it is correct, even by checking the current state of the SystemTests. Before we had the whole system tests aborted because of the error in syntax of the ReflectometryQuickPointDetectorSURF.py, now, we have all the testing passing except the one failuring, and the message of the failed test is clear that it was a problem of syntax.

[Stuart Campbell]

This ticket has been transferred to github issue ​9377