ManagedWorkspace forget the spectra/detector lookups

[Martyn Gigg]

When moving to the ISpectrum implementation, the detector IDs moved to be contained within the ISpectrum objects. A managed workspace saves and reads ManagedDataBlocks to a file and this means that when it saves the data block it only knows about the data.

The SpectraDetectorMap was always held in memory and so this was not lost when saving the blocks to disk.

This needs to be fixed urgently as it renders ManagedWorkspaces effectively useless.

[Martyn Gigg]

In [13325]:

Refs #3400. Very temporary fix so that ManagedWorkspaces function correctly and at least the system tests should pass. Needs a better solution, details in ticket.

[Russell Taylor]

In [13332]:

Add test of spectrum and detector numbers. Re #3400.

[Russell Taylor]

My first thought was to write out the spectrum number and detector IDs to the file. However, that won't work because we rely on the data relating to a spectrum being of fixed length (to seek to the right point in the file) and the number of detector IDs is not fixed.

The two options I see are:

• Make sure that a managed workspace always keeps a full set (i.e. one per spectrum) of ISpectrum/Histogram1D instances in existance, and just fills and collapses the data vectors as things go back and forth to file. Currently, Histogram1D instances (actually ManagedDataBlock instances, which hold 1 or more Histogram1Ds) are created and destroyed as they enter and leave the mru list.
• Have a ManagedWorkspace hold a separate copy of all the spectrum/detector information and just push it into the ISpectrum/Histogram1D object on its creation.

The second option seems like a bit of a hack (with maintenance issues) but the first seems like a lot more work.

A further issue is that we need to make sure that any calls that in principle go through all the spectra looking for certain detector IDs, for example, don't lead to all the data being churned on and off the disk.

[Russell Taylor]

In [13500]:

Performance tests for managed workspaces. Re #3400.

[Russell Taylor]

In [13501]:

Try and get performance test to find data on build server. Re #3400.

[Russell Taylor]

In [13502]:

Try again. Re #3400.

[Mathieu Doucet]

In [13523]:

Fix warnings. Re #3400

[Janik Zikovsky]

Also: Make Workspace2D's data[] vector shared with the data in ManagedWorkspace2D for more robustness.

[Janik Zikovsky]

In [13534]:

Refs #3400, Refs #3203. Fix for ManagedWorkspace2D and its variants. Workspace2D now holds a vector of pointers that should keep things consistent between it and its managed variants. ManagedHistogram1D is now the replacement in the managed cases.

[Janik Zikovsky]

In [13535]:

Refs #3400: Forgot to add the files (as usual :P )

[Mathieu Doucet]

In [13556]:

Fix warning Re #3400

[Russell Taylor]

Some tests on a lousy win32 box:

Run ConvertUnits on a ManagedRawFileWorkspace in place: Last release ~43s, now perhaps ~47s

Clone a GEM ManagedRawFileWorkspace into a new ManagedWorkspace: Last release ~45s, now ~95s

Tests on a Mac showed similar behaviour.

[Martyn Gigg]

Last night the system tests took ~5 hours rather than the usual 1.5 hours. I've attached the log. A cursory glance seems to suggest that the loading parts are the worst.

[Russell Taylor]

Actually, it's MaskDetectorsInShape that accounted for 3.75 hours of the 4.5 total time spent running algorithms.

[Russell Taylor]

ManagedWorkspaces should now work. A ticket ([3740]) has been created for the next iteration to look into the suggestion that performance has been degraded.

[Martyn Gigg]

Tested with ManagedRawFileWorkspace and standard ManagedWorkspace from raw and NeXus files. The detector/spectrum tables look okay. As mentioned above #3740 has been created to examine the performance implact.

[Stuart Campbell]

This ticket has been transferred to github issue ​4247