Ticket #9874 (closed: fixed)

Opened 6 years ago

Last modified 5 years ago

Have doctest output JUnit style output that Jenkins can parse

Reported by: Martyn Gigg Owned by: Martyn Gigg
Priority: critical Milestone: Release 3.3
Component: Framework Keywords: Maintenance
Cc: Blocked By:
Blocking: Tester: Alex Buts

Description

That way the failures and timings are more visible and easier to access.

Attachments

output.txt​ (122.2 KB) - added by Martyn Gigg 6 years ago.

Change History

comment:1 Changed 6 years ago by Nick Draper

  • Status changed from new to assigned

comment:2 Changed 6 years ago by Martyn Gigg

  • Status changed from assigned to inprogress

Add doctest module & TestCase, TestSuite & DocTestOutput classes

Refs #9874

Changeset: 7de8a8cb99c06d566289a75044a696055e4da253

comment:3 Changed 6 years ago by Martyn Gigg

Parses documents where all tests passed.

Refs #9874

Changeset: 9ed63acf383cb922f8965e5472e6cbb7ee3600d3

comment:4 Changed 6 years ago by Martyn Gigg

Add unit test for all passed case and fixed problems unit test found.

Refs #9874

Changeset: 87a17c625a5394d125b5da82a9ae0d977b4cf617

comment:5 Changed 6 years ago by Martyn Gigg

Parser and tests for all pass and all failure cases.

Refs #9874

Changeset: 720388979c9b641b73a1c390fae50584fb92de8b

comment:6 Changed 6 years ago by Martyn Gigg

Finish parsing of mixed pass/fail document.

Refs #9874

Changeset: 3f44170dcf0a1461397a028c70439bb3b7f3a737

comment:7 Changed 6 years ago by Martyn Gigg

Write out xunit XML when Sphinx build has finished.

This only happens if the builder type is set to doctest. Refs #9874

Changeset: 75df64452cc79114cceb6ee8990ccbcd532f701c

comment:8 Changed 6 years ago by Martyn Gigg

Add a test for multi-document output.

Refs #9874

Changeset: 90c90438101112dfae2200f14aa9d1fa21db087b

comment:9 Changed 6 years ago by Martyn Gigg

Change xunit output file to match those produced by ctest

Refs #9874

Changeset: 16a3c0f03985009ca1465876985409b4503ac9b3

comment:10 Changed 6 years ago by Martyn Gigg

Add in some debug statements.

Refs #9874

Changeset: 858c77529624febc5e3fe3cd5b6c11570d0c08d1

comment:11 Changed 6 years ago by Martyn Gigg

Activate the mantiddoc.doctest extension

Refs #9874

Changeset: 674756b8f52d3a8425ba5277c7237e94ed0f33e2

comment:12 Changed 6 years ago by Martyn Gigg

Use lxml for xml if it is available.

Refs #9874

Changeset: d900fcf539180412c678f13e320284a4fbbba567

comment:13 Changed 6 years ago by Martyn Gigg

Add package information to XML doctest output.

Refs #9874

Changeset: a98b7221c36bbd36988e163dbfa22eeacdb4ccfe

comment:14 Changed 6 years ago by Martyn Gigg

Fix classname for tests part of fail suite.

Refs #9874

Changeset: 1708cb2a7afd1b4d0f2a26f0d3c42597cba71250

comment:15 Changed 6 years ago by Martyn Gigg

Fix unittest after classname change.

Refs #9874

Changeset: aa219b9da112437c8a91c0b24a235477f9b5c90a

comment:16 Changed 6 years ago by Martyn Gigg

Branch: ​feature/9874_doctest_xunit_output

Tester: A final step to the doctest build has been added to parse the output and convert it to junit-style output. The results can be seen here:

​http://builds.mantidproject.org/job/develop_incremental/1277/label=rhel6-build/testReport/docs/

Jenkins seems particularly slow at loading those pages, which is most probably a configuration issue and will be looked at.

There should be a break down of each file and then clicking on an individual file should give a break down of the tests in that file.

Last edited 6 years ago by Martyn Gigg (previous) (diff)

comment:17 Changed 6 years ago by Martyn Gigg

  • Status changed from inprogress to verify
  • Resolution set to fixed

comment:18 Changed 6 years ago by Martyn Gigg

  • Status changed from verify to reopened
  • Resolution fixed deleted

It seems there is an issue parsing a failure case where the cleanup code passes. This is demonstrated in the attached file.

Changed 6 years ago by Martyn Gigg

comment:19 Changed 6 years ago by Martyn Gigg

  • Status changed from reopened to inprogress

Fix parser when encountering testcleanup/testsetup failures.

Refs #9874

Changeset: 2e3742c50867832f0b7d286080d3bff7210cf95f

comment:20 Changed 6 years ago by Martyn Gigg

  • Status changed from inprogress to verify
  • Resolution set to fixed

comment:21 Changed 6 years ago by Martyn Gigg

  • Status changed from verify to reopened
  • Resolution fixed deleted

comment:22 Changed 6 years ago by Martyn Gigg

  • Status changed from reopened to inprogress

Raise exception doctest build-finished handler if there was an error.

Refs #9874

Changeset: 6927464c9a55c8107a62600ed49c30cde8a2d4ba

comment:23 Changed 6 years ago by Martyn Gigg

  • Status changed from inprogress to verify
  • Resolution set to fixed

comment:24 Changed 6 years ago by Alex Buts

  • Status changed from verify to verifying
  • Tester set to Alex Buts

comment:25 Changed 6 years ago by Martyn Gigg

  • Status changed from verifying to reopened
  • Resolution fixed deleted

comment:26 Changed 6 years ago by Martyn Gigg

  • Status changed from reopened to inprogress

Fix regex for finding failed test name.

Refs #9874

Changeset: bcd95955feb3a05e3fe680df0939d7e182e5b3cf

comment:27 Changed 6 years ago by Martyn Gigg

  • Status changed from inprogress to verify
  • Resolution set to fixed

The regex fix should be the final fix.

This can be seen at: ​http://builds.mantidproject.org/job/doctest_develop/, where build #5 and #6 parsed the errors correctly and when the build passed in #7, the results confirmed that everything passes.

comment:28 Changed 6 years ago by Alex Buts

  • Status changed from verify to verifying

comment:29 Changed 6 years ago by Alex Buts

  • Status changed from verifying to closed

Merge remote-tracking branch 'origin/feature/9874_doctest_xunit_output'

Full changeset: ​4b13f01765d884c8e36529d2ec5751e7acffdf18

comment:30 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue ​10716

Note: See TracTickets for help on using tickets.