Ticket #839 (closed: fixed)

Opened 11 years ago

Last modified 5 years ago

Automatic install and execution of performance tests

Reported by: Nick Draper Owned by: Janik Zikovsky
Priority: critical Milestone: Iteration 30
Component: Tools Keywords:
Cc: Blocked By:
Blocking: Tester: Russell Taylor

Description (last modified by Nick Draper) (diff)

This can use the same approach as the system tests for automatic install of the msi (for windows at least).

We could even chain the two together such that we do a fresh install once and then run the system tests and the performance tests. We should still report them both out seperately.

It may be good idea to make both types of tests record the results in an xml file that Jenkins can read.

This may remove the need to track the history of the performance tests ourselves so much, and give us a way of plotting trends.

However we do need a reasonable way of defining what a significant change in performance is. One possibility would be that in the test you specify an accepted variance (default 10%), and that each run would be compared to the average of the last 5 days runs (on this environment), if it varies by more than the acceptable variance then the test should fail. Of course that means you will need to keep track of the last 5 days of runs of the test, perhaps a simple text file would do.

Change History

comment:1 Changed 11 years ago by Nick Draper

  • Milestone changed from Iteration 19 to Iteration 20

Moved as part of the end of Iteration 19

comment:2 Changed 11 years ago by Nick Draper

  • Owner set to Martyn Gigg

comment:3 Changed 11 years ago by Nick Draper

  • Status changed from new to assigned
  • Owner changed from Martyn Gigg to Russell Taylor

comment:4 Changed 11 years ago by Nick Draper

  • Priority changed from major to minor

comment:5 Changed 10 years ago by Nick Draper

  • Milestone changed from Iteration 26 to Iteration 27

Bulk move of tickets to iteration 27, if your ticket is essential for Iteration 26 then move it back.

comment:6 Changed 10 years ago by Nick Draper

  • Milestone changed from Iteration 27 to Iteration 28

Bulk move of tickets at the end of iteration 27

comment:7 Changed 10 years ago by Nick Draper

  • Owner changed from Russell Taylor to Janik Zikovsky
  • Priority changed from minor to critical
  • Component set to Tools
  • Description modified (diff)
  • Summary changed from Automatic install and execution of stress tests to Automatic install and execution of performance tests

comment:8 Changed 10 years ago by Janik Zikovsky

  • Status changed from assigned to accepted

comment:9 Changed 10 years ago by Janik Zikovsky

(In [9803]) Refs #839: Creating a separate folder for combined system tests and performance tests; calling it SystemTests2 for now.

comment:10 Changed 10 years ago by Janik Zikovsky

(In [9806]) Refs #839: Renamed module systemtesting; cmd line arguments

comment:11 Changed 10 years ago by Janik Zikovsky

(In [9860]) Refs #839: Checkpointing system tests

comment:12 Changed 10 years ago by Janik Zikovsky

(In [9865]) Refs #839: More progress on auto performance testing

comment:13 Changed 10 years ago by Janik Zikovsky

(In [9866]) Refs #839: Added junit XML test reporter.

comment:14 Changed 10 years ago by Janik Zikovsky

(In [9867]) Refs #839: Support missing SQL python module

comment:15 Changed 10 years ago by Janik Zikovsky

(In [9868]) Refs #839: Support missing python libs

comment:16 Changed 10 years ago by Janik Zikovsky

(In [9869]) Refs #839: XML fix

comment:17 Changed 10 years ago by Janik Zikovsky

(In [9870]) Refs #839: XML fix

comment:18 Changed 10 years ago by Janik Zikovsky

(In [9871]) Refs #839: XML fix

comment:19 Changed 10 years ago by Janik Zikovsky

(In [9872]) Refs #839: XML fix

comment:20 Changed 10 years ago by Janik Zikovsky

(In [9873]) Refs #839: XML fix

comment:21 Changed 10 years ago by Janik Zikovsky

(In [9874]) Refs #839: XML give the log

comment:22 Changed 10 years ago by Janik Zikovsky

(In [9885]) Refs #839: Create a Data folder at the root of the SVN to put system test data.

comment:23 Changed 10 years ago by Janik Zikovsky

(In [9886]) Refs #839: Started svn copying system test data files

comment:24 Changed 10 years ago by Janik Zikovsky

(In [9887]) Refs #839: Make a Mantid.properties file when running; checking in missing file.

comment:25 Changed 10 years ago by Janik Zikovsky

(In [9888]) Refs #839: System tests data files

comment:26 Changed 10 years ago by Janik Zikovsky

(In [9893]) Refs #839: Switched to sqlite3 to use a local file DB instead.

comment:27 Changed 10 years ago by Janik Zikovsky

(In [9895]) Refs #839: Automatic creation of SQLite database if it does not exist

comment:28 Changed 10 years ago by Janik Zikovsky

(In [9897]) Refs #839: Try to make the mantid properties point only to one plugins dir

comment:29 Changed 10 years ago by Janik Zikovsky

(In [9898]) Refs #839: Point to the instrument directory

comment:30 Changed 10 years ago by Janik Zikovsky

(In [9899]) Refs #839: System tests data files

comment:31 Changed 10 years ago by Janik Zikovsky

(In [9900]) Refs #839: more system tests data files

comment:32 Changed 10 years ago by Janik Zikovsky

(In [9901]) Refs #839: Fixing up paths in the SANS system tests

comment:33 Changed 10 years ago by Janik Zikovsky

(In [9903]) Refs #839: HFIRReduction test fix

comment:34 Changed 10 years ago by Janik Zikovsky

(In [9904]) Refs #839: more system tests fix

comment:35 Changed 10 years ago by Janik Zikovsky

(In [9905]) Refs #839: more system tests fix

comment:36 Changed 10 years ago by Janik Zikovsky

(In [9906]) Refs #839: more reporting

comment:37 Changed 10 years ago by Janik Zikovsky

(In [9908]) Refs #839: more reporting

comment:38 Changed 10 years ago by Janik Zikovsky

(In [9910]) Refs #839: HTML report for system tests started

comment:39 Changed 10 years ago by Janik Zikovsky

(In [9911]) Refs #839: More detailed HTML reporting.

comment:40 Changed 10 years ago by Janik Zikovsky

(In [9914]) Refs #839: More plots and better looking HTML reports.

comment:41 Changed 10 years ago by Janik Zikovsky

(In [9915]) Refs #839: Detect failed plotting and do the rest of reporting anyway

comment:42 Changed 10 years ago by Janik Zikovsky

(In [9916]) Refs #839: fix to makeReport.py

comment:43 Changed 10 years ago by Janik Zikovsky

(In [9917]) Refs #839: reverse order of test results

comment:44 Changed 10 years ago by Janik Zikovsky

(In [9918]) Refs #839: html reporting

comment:45 Changed 10 years ago by Janik Zikovsky

(In [9919]) Refs #839: Do-nothing change; I am testing SVN revision number updating in MantidVersion.h

comment:46 Changed 10 years ago by Janik Zikovsky

(In [9923]) Refs #839: Use MantidPlot --version to determine SVN version

comment:47 Changed 10 years ago by Janik Zikovsky

(In [9924]) Refs #839: Use MantidPlot --version to determine SVN version

comment:48 Changed 10 years ago by Janik Zikovsky

(In [9926]) Refs #839: Summary table in HTML report

comment:49 Changed 10 years ago by Janik Zikovsky

(In [9928]) Refs #839: Option to choose which DB file to use. Archiving log files to a folder instead of in the database because the DB would get too huge.

comment:50 Changed 10 years ago by Janik Zikovsky

(In [9930]) Refs #839: Save the type of the runner in db.

comment:51 Changed 10 years ago by Janik Zikovsky

(In [9954]) Refs #839: Arguments to runTests

comment:52 Changed 10 years ago by Janik Zikovsky

(In [9958]) Refs #839: HTML report

comment:53 Changed 10 years ago by Janik Zikovsky

(In [10003]) Refs #839: Path finding for runTests (performance tests)

comment:54 Changed 10 years ago by Janik Zikovsky

(In [10005]) Refs #839: Paths for runTests.py

comment:55 Changed 10 years ago by Janik Zikovsky

(In [10025]) Refs #839: More reporting options.

comment:56 Changed 10 years ago by Janik Zikovsky

(In [10040]) Refs #839: report for system tests.

comment:57 Changed 10 years ago by Janik Zikovsky

(In [10041]) Refs #839: Fix for runTests

comment:58 Changed 10 years ago by Janik Zikovsky

(In [10042]) Refs #839: Fix for runTests

comment:59 Changed 10 years ago by Janik Zikovsky

(In [10043]) Refs #839: Fix for runTests

comment:60 Changed 10 years ago by Janik Zikovsky

(In [10704]) Refs #2804, #839: Added support for Performance tests in C++ by defining an extra test suite in a test .h file. File is added to ctest list if CXXTEST_ADD_PERFORMANCE cmake variable is TRUE. The tests are always built though.

comment:61 Changed 9 years ago by Janik Zikovsky

  • Milestone changed from Iteration 28 to Iteration 29

comment:62 Changed 9 years ago by Janik Zikovsky

(In [11163]) Refs #839: Added a svn folder for holding performance test results databases.

comment:63 Changed 9 years ago by Janik Zikovsky

(In [11276]) Refs #839: Fix TOPAZ grouping performance test.

comment:64 Changed 9 years ago by Janik Zikovsky

(In [11384]) Refs #2998, #839: Made DiffractionFocussing2TestPerformance a lot faster.

comment:65 Changed 9 years ago by Janik Zikovsky

(In [12148]) Refs #839, Refs #2830: Performance test for reciprocal space integration. About 10000 peaks per second on my system.

comment:66 Changed 9 years ago by Janik Zikovsky

  • Milestone changed from Iteration 29 to Iteration 30

Moved my tickets that won't get finished by iteration 29

comment:67 Changed 9 years ago by Janik Zikovsky

(In [12891]) Refs #839: wikimaker

comment:68 Changed 9 years ago by Janik Zikovsky

In [13237]:

Refs #839: Adjusted a performance test to slow it down enough that it gives consistent timings.

comment:69 Changed 9 years ago by Janik Zikovsky

  • Status changed from accepted to verify
  • Resolution set to fixed

OK, this ticket has been pretty much fixed for ages, and the performance tests are running on a regular basis. Time for fresh tickets instead of always using this one.

Fixed!

comment:70 Changed 9 years ago by Janik Zikovsky

In [13707]:

Refs #839: Disabled too-fast and irrelevant performance tests.

comment:71 Changed 9 years ago by Russell Taylor

  • Status changed from verify to verifying
  • Tester set to Russell Taylor

comment:72 Changed 9 years ago by Russell Taylor

  • Status changed from verifying to closed

We have performance testing in production via the ORNL Jenkins! It doesn't work quite as set out in the ticket description, in that it is integrated into cxxtest/ctest rather than running off an installed version. We have it running on rhel, but there's no reason why it shouldn't work anywhere. The automatic emails are almost always spurious (maybe we should up the threshold), but the plots accessible via Jenkins mean it's easy to periodically scan all the test results and spot slowdowns.

Now we need more performance tests!

comment:73 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue ​1687

Note: See TracTickets for help on using tickets.