Ticket #2342 (closed: wontfix)
Nexus: Can you compress (large) chunks using multiple processors
| Reported by: | Janik Zikovsky | Owned by: | Janik Zikovsky |
|---|---|---|---|
| Priority: | major | Milestone: | Iteration 30 |
| Component: | Mantid | Keywords: | |
| Cc: | Blocked By: | ||
| Blocking: | Tester: | Nick Draper |
Description
... is this possible under HDF? The compression step seems to be the limiting step in, e.g. SaveSNSNexus.
Another option: use szip compression: http://www.hdfgroup.org/doc_resource/SZIP/ which may be 4x faster?
Change History
comment:1 Changed 10 years ago by Janik Zikovsky
- Milestone changed from Iteration 27 to Iteration 28
comment:2 Changed 10 years ago by Janik Zikovsky
Speed comparison depending on the type of compression in NEXUS format. The file is TOPAZ_2170_events, loaded then saved as SaveNexusProcessed.
- Uncompressed: 38 seconds to write 2.2 GB.
- LZW compression (default): 308 seconds to write 1.3 GB.
- SZIP compression: 56 seconds to write 1.7 GB. SZIP requires HDF with SZIP enabled, available from the HDF site but not from repository (license issues).
see #2355.
comment:4 Changed 9 years ago by Janik Zikovsky
- Milestone changed from Iteration 29 to Iteration 30
Moved my tickets that won't get finished by iteration 29
comment:5 Changed 9 years ago by Janik Zikovsky
- Status changed from new to verify
- Resolution set to wontfix
Compression of these data sets is so small as to make it silly. So I guess I'll forget about compression because of the huge drag on performance.
Note: See
TracTickets for help on using
tickets.
