Ticket #8665 (closed: wontfix)

Opened 7 years ago

Last modified 5 years ago

Make CompressEvents work on a regular grid of the tolerance

Reported by: Russell Taylor Owned by: Russell Taylor
Priority: major Milestone: Release 3.3
Component: Framework Keywords:
Cc: Blocked By:
Blocking: Tester: Vickie Lynch

Description (last modified by Russell Taylor) (diff)

Event compression currently works by finding all the events within a tolerance window of the first event, and then shifting the minimum time of the next 'group' to the time of the first event beyond the window (and then repeating the procedure). This introduces a bias towards larger gaps in the times between (weighted) events and can lead to artefacts (e.g. dips) in the rebinned distribution, which have been a cause of concern to one of our users.
Admittedly, these are only really striking if the rebinning is at a comparable scale to the tolerance (which is the wrong thing to do - it should be much larger), but still it seems like it would be better to eliminate the bias and use a regular grid of tolerance instead. This will eliminate the artefacts at fine binning while having a statistically negligible effect at wider binning. It will lead to slightly less memory being saved for a given tolerance, but only at the few percent level.

Change History

comment:1 Changed 7 years ago by Russell Taylor

  • Description modified (diff)

comment:2 Changed 7 years ago by Russell Taylor

  • Status changed from new to inprogress

Re #8665. Refactor by pulling the first event out of the loop.

Dealing with the first event ahead of the loop allows some simplifications of the code within the loop. It does require returning early if the list is empty (which is a good thing anyway).

Changeset: 13cf9b252281641d09ad56fe0229cf38ceddb34f

comment:3 Changed 7 years ago by Russell Taylor

Re #8665. Move up the compression 'window' by a multiple of tolerance.

This is instead of moving it up to the time of the first event outside the current window, which introduces a bias. The previous behaviour is maintained for a zero tolerance (which is a valid input) to avoid an infinite loop.

Changeset: e21827e24d710b36352b219ca7a604aa86e4c32f

comment:4 Changed 7 years ago by Russell Taylor

Re #8665. Update expected number of points.

It's change by a fraction of a percent because of the change to event compression. Also relax the tolerance - we don't need to care about the smallest changes.

Changeset: c3355a9ac169b3487651c1f05cfec11b85775511

comment:5 Changed 7 years ago by Russell Taylor

Re #8665. Relax tolerance to get test passing after compress events change.

Changeset: fea12b7d324aa089f29579e89ac893e46f15bbc2

comment:6 Changed 7 years ago by Russell Taylor

This had a noticeable negative effect on performance - it was something like 50% slower for the two compressEvents performance tests and there was a system test (PG3CCCalibration) that took a 10s (30%) hit as well. So I pulled the changes out of develop pending further discussions.

comment:7 Changed 7 years ago by Russell Taylor

Actually, I just noticed that the SNSPowderRedux.SeriesAndConjoinFilesTest system test went from 37 -> 94s on this change!

comment:8 Changed 7 years ago by Russell Taylor

  • Milestone changed from Release 3.1 to Backlog

comment:9 Changed 6 years ago by Russell Taylor

  • Status changed from inprogress to verify
  • Resolution set to wontfix

We don't want to take the big performance hit for a change that isn't any more correct that what's in place at present.

comment:10 Changed 6 years ago by Vickie Lynch

  • Status changed from verify to verifying
  • Tester set to Vickie Lynch

comment:11 Changed 6 years ago by Vickie Lynch

  • Status changed from verifying to closed

comment:12 Changed 6 years ago by Nick Draper

  • Milestone changed from Backlog to Release 3.3

comment:13 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 9509

Note: See TracTickets for help on using tickets.