Add memory checking for system tests

[Nick Draper]

Allow a test to check memory before an after (with some tolerance) to check for (large) memory leaks.

[Nick Draper]

Moved at the code freeze for release 2.4

[Russell Taylor]

Re #6169. Report the memory footprint increase during a test.

First attempt to show the increase in the memory footprint due to a test (after any created workspaces etc. have been deleted). Added to the XML report for use by Jenkins & the performance test framework.

Changeset: 97a597968a93d170d4ce86669629de2379219b18

[Russell Taylor]

Re #6169. Make memory reporting field in xml report more compact.

I was hoping that the Jenkins Measurement Plots plugin would pick it up and formatted it for that, but it seems that will need more messing with the Jenkins config than I'm willing to do when the main way of viewing this will be via the performances tests infrastructure.

Changeset: 6d15528be767a0e3323f6b565709d6f2380ec792

[Russell Taylor]

Re #6169. Use new API export of MemoryStats.

I was thinking of using psutil (​http://code.google.com/p/psutil/) but then realised that this would require it to be installed on any machine that was going to run system tests. Also, it doesn't seem to work on the latest Mac OS's. However, I was pleased to see that its report of the available memory on Linux & Windows matches exactly that given by Mantid.

Changeset: 264b981411240e8a46b83557e635619f38849b67

[Russell Taylor]

Move the tickets I'm definitely not going to do this iteration so that I can better see the ones that I might.

[Russell Taylor]

Re #6169. Don't report timing information for skipped or failed tests.

Changeset: 7e21b1d29600e18b65ead7bd140ce93905895621

[Nick Draper]

Moved to the Backlog after R2.6

[Russell Taylor]

Re #6169. Don't report timing information for skipped or failed tests.

Changeset: a02f49c795b920f1fd00b132d42d5fbd832f9712

[Russell Taylor]

Re #6169. Add memory reporting to the performance tests.

Added a column to the database and then took the value from there into the HTML reports (detailed and summary).

Changeset: b53c44369e0b5c9d8d62367fef55515c082356e9

[Russell Taylor]

Re #6169. Add memory info to main summary page.

Changeset: c1751e61a6f78f7de8e16d70e4499129b6f5f48b

[Russell Taylor]

Re #6169. Don't bomb out if a date is formatted incorrectly.

For some reason, one of the dates in the first test I did on Windows was missing the milliseconds, which stopped the script. It really isn't a big deal if it's missing from the summary page, so just catch the exception and continue.

Changeset: 4778776c3e08a08c218b3facaa3b863b24b22b92

[Russell Taylor]

Re #6169. Clear the FrameworkManager at the start.

I don't think this should make any difference, but I just want to check.

Changeset: 9216a39645532dfed168affa09dd312dcaa09455

[Russell Taylor]

Re #6169. Try getting the initial memory in init instead of execute.

Some tests load data in their init so I can't clear things out in execute.

Changeset: 6cd4e4d42c517cd5f5c74476d020c95b34f6da41

[Russell Taylor]

Tester: The branch is feature/6169_performance_monitoring in the systemtests repository (N.B. only the commits from comment:9 onwards are in this branch; the earlier ones are already in master).

The memory reporting that this produces can be seen ​here and ​here. I'll be honest - I'm a little sceptical of what's coming out of here, but I think that the code I've put in is right. It gets the free memory of the system before and after the test and the subtracts the latter from the former, where all workspaces etc. are cleared before the second check. The approximation is that nothing else on the system changes its memory use while the test is running.

[Martyn Gigg]

I think the reason the results are so different is down to things we don't yet fully understand about tcmalloc etc on linux. Nevertheless the changes are sensible and the I believe it to be reporting what is actually happening.

[Martyn Gigg]

Merge remote-tracking branch 'origin/feature/6169_performance_monitoring'

[Stuart Campbell]

This ticket has been transferred to github issue ​7015