Ticket #11614 (closed: fixed)
SaveNXSPE: Write speeds very slow on ceph file system
Reported by: | Martyn Gigg | Owned by: | Martyn Gigg |
---|---|---|---|
Priority: | critical | Milestone: | Release 3.4 |
Component: | Framework | Keywords: | |
Cc: | Blocked By: | ||
Blocking: | Tester: | Andrei Savici |
Description
The excitations group have noticed that a 300Mb file can take ~20mins to write to the ceph file system used for autoreduction!
Change History
comment:2 Changed 5 years ago by Martyn Gigg
- Status changed from new to inprogress
Improve tests around SaveNXSPE
The main test now actually verifies the data in the file using HDF. Refs #11614
Changeset: dd3a8634edd87669c069adb38af76c128ce28fb6
comment:3 Changed 5 years ago by Martyn Gigg
Improve SaveNXSPE test to check different sizes.
Refs #11614
Changeset: 4e9598dd01329c6a8406c69548c4a5650d751561
comment:4 Changed 5 years ago by Martyn Gigg
Write the data to the file in larger chunks
Avoids excessive disk writes that slow things down. Refs #11614
Changeset: b2b0434faa29d22aab6115f40e6e95be0686f0ef
comment:5 Changed 5 years ago by Martyn Gigg
Improve tests around SaveNXSPE
The main test now actually verifies the data in the file using HDF. Refs #11614
Changeset: 40fd11d02dc5352791a0c8d2bc527d713fac9d79
comment:6 Changed 5 years ago by Martyn Gigg
Improve SaveNXSPE test to check different sizes.
Refs #11614
Changeset: a91025ca80192cc68bc661dd7f00a1b8a518960f
comment:7 Changed 5 years ago by Martyn Gigg
Write the data to the file in larger chunks
Avoids excessive disk writes that slow things down. Refs #11614
Changeset: 813aee83845fe04e768682a8f96b87664872f978
comment:8 Changed 5 years ago by Martyn Gigg
- Status changed from inprogress to verify
- Resolution set to fixed
This is being verified as pull request #632.
comment:10 Changed 5 years ago by Andrei Savici
- Status changed from verify to verifying
- Tester set to Andrei Savici
comment:11 Changed 5 years ago by Andrei Savici
Works with DAVE Mslice and Horace
comment:12 Changed 5 years ago by Martyn Gigg
Improve tests around SaveNXSPE
The main test now actually verifies the data in the file using HDF. Refs #11614
Changeset: 40fd11d02dc5352791a0c8d2bc527d713fac9d79
comment:13 Changed 5 years ago by Martyn Gigg
Improve SaveNXSPE test to check different sizes.
Refs #11614
Changeset: a91025ca80192cc68bc661dd7f00a1b8a518960f
comment:14 Changed 5 years ago by Martyn Gigg
Write the data to the file in larger chunks
Avoids excessive disk writes that slow things down. Refs #11614
Changeset: 813aee83845fe04e768682a8f96b87664872f978
comment:15 Changed 5 years ago by Martyn Gigg
comment:16 Changed 5 years ago by Andrei Savici
- Status changed from verifying to closed
Merge pull request #632 from mantidproject/11614_improve_savenxspe_write_speed
Improve speed of SaveNXSPE
Full changeset: d31c0d839c8821ef8ac2b6061c4013fb0b6dc566
comment:17 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 12452
The following script uses h5py to verify the problem:
If you uncomment the slab_size = 8*num_cols line then you will observe a large difference in the write speed. This is because more calls to putslab mean more disk accesses and some parts of ceph must be waiting on previous chunks to be transferred to remote locations before allowing the next to be written.
A solution is to write the data in larger chunks.