Ticket #8533 (closed: fixed)

Opened 7 years ago

Last modified 5 years ago

Syntax error in any systemtest stops the entire sset from running

Reported by: Martyn Gigg Owned by: Martyn Gigg
Priority: major Milestone: Release 3.1
Component: Framework Keywords:
Cc: Blocked By:
Blocking: Tester:

Description

This is example output from a recent jenkins run of the system tests:

+ python InstallerTests.py -o
FrameworkManager-[Notice] Welcome to Mantid version 3.0.20131128.1550 - Manipulation and Analysis Toolkit for Instrument Data
FrameworkManager-[Notice] Please cite Mantid in your publications using: http://dx.doi.org/10.5286/Software/Mantid
Traceback (most recent call last):
  File "runSystemTests.py", line 73, in <module>
    testsInclude=options.testsInclude, testsExclude=options.testsExclude)
  File "../StressTestFramework/stresstesting.py", line 757, in __init__
    self._tests = self.loadTestsFromDir(test_dir)
  File "../StressTestFramework/stresstesting.py", line 824, in loadTestsFromDir
    tests.extend(self.loadTestsFromModule(os.path.join(test_dir,file)))
  File "../StressTestFramework/stresstesting.py", line 838, in loadTestsFromModule
    mod = imp.load_module(modname, pyfile, filename, ("","",imp.PY_SOURCE))
  File "/home/mantidlog/jenkins/workspace/is_test_rhel6_develop/SystemTests/AnalysisTests/ReflectometryQuickPointDetectorSURF.py", line 16
    try
      ^
SyntaxError: invalid syntax
Starting system tests
Using installer /home/mantidlog/jenkins/workspace/is_test_rhel6_develop/InstallerTesting/mantidnightly-3.0.20131128.1550-1.el6.x86_64.rpm
Installing package '/home/mantidlog/jenkins/workspace/is_test_rhel6_develop/InstallerTesting/mantidnightly-3.0.20131128.1550-1.el6.x86_64.rpm'
Application path /opt/mantidnightly/bin/MantidPlot
MantidPlot directory /opt/mantidnightly/bin
Pointing MANTIDPATH at MantidPlot directory /opt/mantidnightly/bin
3.0.20131128.1550 (Thu, 28 Nov 2013)

It hit a syntax error when loading a test and subsequently the whole thing fell over. We should catch these and report them as errors in the tests themselves.

Change History

comment:1 Changed 7 years ago by Martyn Gigg

  • Status changed from new to inprogress

comment:2 Changed 7 years ago by Martyn Gigg

Catch syntax errors when loading tests and report them properly.

This allows the rest of the framework to continue and not come crashing down due to an error in one test. Refs #8533

Changeset: f8b6963a8d23c455d3402e5d05247029df9c6f91

comment:3 Changed 7 years ago by Martyn Gigg

  • Status changed from inprogress to verify
  • Resolution set to fixed

Branch: bugfix/8533_catch_syntax_errors in the systemtests repository.

Tester: To test this properly you will need to introduce a syntax error into one of the tests. I would use a package version of Mantid to test this:

  • Merge the code
  • Edit, for example, ReflectometryQuick.py, to introduce a syntax error
  • Try and run the system tests using runSystemTests.py -R Reflectometry You should get a single failure in ReflectometryQuick relating to the syntax error and the other reflectometry tests should pass.

You could also check that the develop system tests are actually running correctly.

comment:4 Changed 7 years ago by Gesner Passos

  • Status changed from verify to closed

Merge remote-tracking branch 'origin/bugfix/8533_catch_syntax_errors'

Full changeset: 8758e7da468fff81c6c892958ec370a1b0354863

comment:5 Changed 7 years ago by Gesner Passos

It is clear that it is correct, even by checking the current state of the SystemTests. Before we had the whole system tests aborted because of the error in syntax of the ReflectometryQuickPointDetectorSURF.py, now, we have all the testing passing except the one failuring, and the message of the failed test is clear that it was a problem of syntax.

comment:6 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 9377

Note: See TracTickets for help on using tickets.