Ticket #9848 (verify: fixed)

Opened 6 years ago

Last modified 5 years ago

Multiperiod reflectometry processing

Reported by: Owen Arnold Owned by: Owen Arnold
Priority: critical Milestone: Release 3.5
Component: Framework Keywords:
Cc: Blocked By: #9888, #10187, #10188
Blocking: #10385 Tester:


  • Use INTER00027729
  • Process it through RRO auto.
  • Check memory utilisation. Users are reporting crashes.


multiperiod_crash.zip (178.8 KB) - added by Owen Arnold 6 years ago.

Change History

comment:1 Changed 6 years ago by Nick Draper

  • Status changed from new to assigned

comment:2 Changed 6 years ago by Owen Arnold

  • Blocked By 9888 added

This is simply a memory size issue on a huge multiperiod group workspace. #9888 will be the start of a proper fix for this.

Changed 6 years ago by Owen Arnold

comment:3 Changed 6 years ago by Owen Arnold

The RRO works provided that you have sufficient memory, but, the saving performance is horrible.

With 511 periods, it took 0.14 seconds to load on my machine. RRO take 0.01 seconds per call. That's 0.5 seconds to process, that's 5 seconds for all runs. However, saving is taking 0.7 seconds per call, that's about 360 seconds!

I suspect that what's happening is that SaveNexus is being applied recursively to the same file on disk. Open-write-close, open-write-close.... This would explain slow operation. Lets see....

comment:4 Changed 6 years ago by Owen Arnold

I can now confirm that the above is happening with GroupWorkspaces. We need to angles of approach here, in the specified order.

1) Fix the current implementations of SaveNexus and SaveProcessedNexus so that they override processGroups and then write the entire file out in one shot. This will improve performance, but ultimately will not change the file size.

2) Handle the multiperiod case of SaveNexus and SaveProcessedNexus, which will actually demand a new file format, since the guarantees of the multiperiod workspace group make a large portion of the data that is currently being stored, redundant.

comment:5 Changed 6 years ago by Owen Arnold

  • Blocked By 10187 added

comment:6 Changed 6 years ago by Owen Arnold

  • Blocked By 10188 added

comment:7 Changed 6 years ago by Owen Arnold

  • Blocking 10385 added

comment:8 Changed 6 years ago by Owen Arnold

  • Milestone changed from Release 3.3 to Release 3.4

comment:9 Changed 5 years ago by Owen Arnold

  • Status changed from assigned to verify
  • Resolution set to fixed

We have mitigated the need for a new workspace or file format (at least in the short term) Blocker tickets of this umbrella have all been sorted out. Release 3.4 sees much better performance around loading.

comment:10 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 10690

Note: See TracTickets for help on using tickets.