Ticket #805 (closed: fixed)

Opened 11 years ago

Last modified 5 years ago

Speed up the loading of nexus files and reduce memory usage

Reported by: Russell Taylor Owned by: Martyn Gigg
Priority: critical Milestone: Iteration 20
Component: Keywords:
Cc: Blocked By:
Blocking: Tester:

Description

It's really slow at the moment. One example showed ~50 seconds as against ~3 seconds for the corresponding raw file.

Change History

comment:1 Changed 11 years ago by Russell Taylor

It turns out that LoadISISNexus v2 is 6 times slower than LoadISISNexus v1!

Here are the benchmarks for SANS2D run 808 on Windows, running off a local disk:

  • LoadRaw v3 - 2.0s - 286MB
  • LoadISISNexus v1 - 5.5s - 469MB
  • LoadISISNexus v2 - 28.6s - 560MB

Even LoadISISNexus version 1 is too slow, but the first step should be to see what caused the big slowdown between versions. Note also the big difference in memory footprint after running LoadNexus as opposed to LoadRaw. We need to find out what that is. (The difference between v1 & v2 is because v2 isn't sharing X vectors at present - I'll fix that shortly.)

comment:2 Changed 11 years ago by Russell Taylor

(In [2943]) Restore sharing of X vectors for LoadISISNexus. Re #805.

comment:3 Changed 11 years ago by Russell Taylor

  • Summary changed from Speed up the loading of nexus files to Speed up the loading of nexus files and reduce memory usage

comment:4 Changed 11 years ago by Russell Taylor

Just also noticed the number of I/O Reads in the Windows Task Manager:

  • LoadRaw v3 - ~3,600
  • LoadNexus v1&2 - ~76,000

comment:5 Changed 11 years ago by Nick Draper

  • Milestone changed from Iteration 19 to Iteration 20

Moved to iteration 20 to make room for stability work

comment:6 Changed 11 years ago by Nick Draper

  • Priority changed from critical to major

comment:7 Changed 11 years ago by Nick Draper

  • Owner changed from Russell Taylor to Martyn Gigg

comment:8 Changed 11 years ago by Nick Draper

  • Priority changed from major to critical

comment:9 Changed 11 years ago by Martyn Gigg

(In [3289]) Updated Nexus classes and isis nexus loading to use variable blocksizes to improve speed. Also fixed a problem when there are only monitor blocks defined. Re #805

comment:10 Changed 11 years ago by Martyn Gigg

(In [3290]) Fix for workspace groups from nexus files if LoadISISNexus was called as a sub algorithm. Re #805

comment:11 Changed 11 years ago by Martyn Gigg

(In [3310]) Fix for failing test. Re #805

comment:12 Changed 11 years ago by Martyn Gigg

(In [3327]) Updated Nexus processed loading routine to use variable blocksize and newer NexusClasses reading routines. Re #805

comment:13 Changed 11 years ago by Martyn Gigg

(In [3328]) Added conversion from string to emum for Property Direction. Re #805

comment:14 Changed 11 years ago by Martyn Gigg

(In [3329]) Fix for windows build. Re #805

comment:15 Changed 11 years ago by Martyn Gigg

(In [3330]) Algorithm history also optinally stores direction. Re #805

comment:16 Changed 11 years ago by Martyn Gigg

(In [3332]) Documentation fix. Re #805

comment:17 Changed 11 years ago by Martyn Gigg

  • Status changed from new to accepted

comment:18 Changed 11 years ago by Martyn Gigg

  • Status changed from accepted to testing
  • Resolution set to fixed

comment:19 Changed 11 years ago by Russell Taylor

  • Status changed from testing to closed

Looks good. Time to load on my machine now down to ~3s. Still a slightly larger memory footprint after a load and 'Clear all memory' that after a LoadRaw and clear (~10MB). It should be noted that LoadRaw isn't 'free' in this regard either.

comment:20 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue ​1653

Note: See TracTickets for help on using tickets.