Ticket #8533 (closed: fixed)
Syntax error in any systemtest stops the entire sset from running
Reported by: | Martyn Gigg | Owned by: | Martyn Gigg |
---|---|---|---|
Priority: | major | Milestone: | Release 3.1 |
Component: | Framework | Keywords: | |
Cc: | Blocked By: | ||
Blocking: | Tester: |
Description
This is example output from a recent jenkins run of the system tests:
+ python InstallerTests.py -o FrameworkManager-[Notice] Welcome to Mantid version 3.0.20131128.1550 - Manipulation and Analysis Toolkit for Instrument Data FrameworkManager-[Notice] Please cite Mantid in your publications using: http://dx.doi.org/10.5286/Software/Mantid Traceback (most recent call last): File "runSystemTests.py", line 73, in <module> testsInclude=options.testsInclude, testsExclude=options.testsExclude) File "../StressTestFramework/stresstesting.py", line 757, in __init__ self._tests = self.loadTestsFromDir(test_dir) File "../StressTestFramework/stresstesting.py", line 824, in loadTestsFromDir tests.extend(self.loadTestsFromModule(os.path.join(test_dir,file))) File "../StressTestFramework/stresstesting.py", line 838, in loadTestsFromModule mod = imp.load_module(modname, pyfile, filename, ("","",imp.PY_SOURCE)) File "/home/mantidlog/jenkins/workspace/is_test_rhel6_develop/SystemTests/AnalysisTests/ReflectometryQuickPointDetectorSURF.py", line 16 try ^ SyntaxError: invalid syntax Starting system tests Using installer /home/mantidlog/jenkins/workspace/is_test_rhel6_develop/InstallerTesting/mantidnightly-3.0.20131128.1550-1.el6.x86_64.rpm Installing package '/home/mantidlog/jenkins/workspace/is_test_rhel6_develop/InstallerTesting/mantidnightly-3.0.20131128.1550-1.el6.x86_64.rpm' Application path /opt/mantidnightly/bin/MantidPlot MantidPlot directory /opt/mantidnightly/bin Pointing MANTIDPATH at MantidPlot directory /opt/mantidnightly/bin 3.0.20131128.1550 (Thu, 28 Nov 2013)
It hit a syntax error when loading a test and subsequently the whole thing fell over. We should catch these and report them as errors in the tests themselves.
Change History
comment:2 Changed 7 years ago by Martyn Gigg
Catch syntax errors when loading tests and report them properly.
This allows the rest of the framework to continue and not come crashing down due to an error in one test. Refs #8533
Changeset: f8b6963a8d23c455d3402e5d05247029df9c6f91
comment:3 Changed 7 years ago by Martyn Gigg
- Status changed from inprogress to verify
- Resolution set to fixed
Branch: bugfix/8533_catch_syntax_errors in the systemtests repository.
Tester: To test this properly you will need to introduce a syntax error into one of the tests. I would use a package version of Mantid to test this:
- Merge the code
- Edit, for example, ReflectometryQuick.py, to introduce a syntax error
- Try and run the system tests using runSystemTests.py -R Reflectometry You should get a single failure in ReflectometryQuick relating to the syntax error and the other reflectometry tests should pass.
You could also check that the develop system tests are actually running correctly.
comment:4 Changed 7 years ago by Gesner Passos
- Status changed from verify to closed
Merge remote-tracking branch 'origin/bugfix/8533_catch_syntax_errors'
Full changeset: 8758e7da468fff81c6c892958ec370a1b0354863
comment:5 Changed 7 years ago by Gesner Passos
It is clear that it is correct, even by checking the current state of the SystemTests. Before we had the whole system tests aborted because of the error in syntax of the ReflectometryQuickPointDetectorSURF.py, now, we have all the testing passing except the one failuring, and the message of the failed test is clear that it was a problem of syntax.