Multiperiod reflectometry processing

[Owen Arnold]

• Use INTER00027729
• Process it through RRO auto.
• Check memory utilisation. Users are reporting crashes.

[Owen Arnold]

This is simply a memory size issue on a huge multiperiod group workspace. #9888 will be the start of a proper fix for this.

[Owen Arnold]

The RRO works provided that you have sufficient memory, but, the saving performance is horrible.

With 511 periods, it took 0.14 seconds to load on my machine. RRO take 0.01 seconds per call. That's 0.5 seconds to process, that's 5 seconds for all runs. However, saving is taking 0.7 seconds per call, that's about 360 seconds!

I suspect that what's happening is that SaveNexus is being applied recursively to the same file on disk. Open-write-close, open-write-close.... This would explain slow operation. Lets see....

[Owen Arnold]

I can now confirm that the above is happening with GroupWorkspaces. We need to angles of approach here, in the specified order.

1) Fix the current implementations of SaveNexus and SaveProcessedNexus so that they override processGroups and then write the entire file out in one shot. This will improve performance, but ultimately will not change the file size.

2) Handle the multiperiod case of SaveNexus and SaveProcessedNexus, which will actually demand a new file format, since the guarantees of the multiperiod workspace group make a large portion of the data that is currently being stored, redundant.

[Owen Arnold]

We have mitigated the need for a new workspace or file format (at least in the short term) Blocker tickets of this umbrella have all been sorted out. Release 3.4 sees much better performance around loading.

[Stuart Campbell]

This ticket has been transferred to github issue ​10690