Ticket #4790 (closed: fixed)
Reduction of chunks of events
| Reported by: | Vickie Lynch | Owned by: | Vickie Lynch |
|---|---|---|---|
| Priority: | major | Milestone: | Release 2.1 |
| Component: | Mantid | Keywords: | |
| Cc: | Blocked By: | ||
| Blocking: | Tester: | Shelly Ren |
Description
The focus workflow in SNSPowderReduction should done for one chunk of events at a time for Nomad data.
Change History
comment:2 Changed 9 years ago by Vickie Lynch
Refs #4790 SNSPowderReduction2 focus with chunks
Changeset: c976748c5810a219e3f885336003fce2fb8e3670
comment:3 Changed 9 years ago by Vickie Lynch
Refs #4790 same updates as SNSPowderReduction
Changeset: f264d7f55d3947bc79e3118dc7e81e9f4ebac743
comment:4 Changed 9 years ago by Vickie Lynch
Refs #4790 determine number of chunks by filesize
Changeset: 9c553d2a40ad7136b05852010109ee09c2e08c93
comment:5 Changed 9 years ago by Vickie Lynch
Refs #4790 return workspace not string
Changeset: 0c19511a07f6b3368fcd0f801ee74ecea808e209
comment:6 Changed 9 years ago by Vickie Lynch
Refs #4790 SNSPowderReduction2 focus with chunks
Changeset: c976748c5810a219e3f885336003fce2fb8e3670
comment:7 Changed 9 years ago by Vickie Lynch
Refs #4790 same updates as SNSPowderReduction
Changeset: f264d7f55d3947bc79e3118dc7e81e9f4ebac743
comment:8 Changed 9 years ago by Vickie Lynch
Refs #4790 determine number of chunks by filesize
Changeset: 9c553d2a40ad7136b05852010109ee09c2e08c93
comment:9 Changed 9 years ago by Vickie Lynch
Refs #4790 return workspace not string
Changeset: 0c19511a07f6b3368fcd0f801ee74ecea808e209
comment:10 Changed 9 years ago by Vickie Lynch
Refs #4790 No chunks yet for event nexus
Changeset: 9bab98078f5b087b22b1f0aa59d77197c1c80d61
comment:11 Changed 9 years ago by Vickie Lynch
Refs #4790 chunks for LoadEventNexus
Changeset: 87662587012ac81473cc46aab84deaca49cd41bb
comment:12 Changed 9 years ago by Vickie Lynch
Refs #4790 use chunks in reduction for event nexus
Changeset: 11512fa9fcaa3f382eb717e744ed526923f79935
comment:13 Changed 9 years ago by Vickie Lynch
Refs #4790 use chunks and time start and stop
Changeset: 94c0e1b596944d7fb9ea9f33a80bccff8fe749e0
comment:14 Changed 9 years ago by Vickie Lynch
Refs #4790 faster chunking by banks than events/bank
Changeset: 998d99f8d5da961ff4688f70d75e4de8e77b32fa
comment:15 Changed 9 years ago by Vickie Lynch
Refs #4790 events/bank chunks are faster
Changeset: 312eea775709b7d7b5effcb46b907266672f7177
comment:16 Changed 9 years ago by Vickie Lynch
Refs #4790 sum not max of multiple files
Changeset: 8fd15ad40ef00a344b14de89f4377e1ce4b1e268
comment:17 Changed 9 years ago by Vickie Lynch
REfs #4790 more like chunking sketch
Changeset: 7a54874afcc80595930eae8778251a3c64625e71
comment:18 Changed 9 years ago by Vickie Lynch
Refs #4790 fix rename
Changeset: 009ba4bc0e070f4f7ba5ab7d37dccf61e0b61f5e
comment:19 Changed 9 years ago by Vickie Lynch
Refs #4790 max chunk size in Gbytes
Changeset: e9f66170c5f1fb46eb81c04628f0429778580ff9
comment:20 Changed 9 years ago by Vickie Lynch
Refs #4790 chunk strategy in table workspace
Changeset: 6e44cb0f5168a59c687ca6c7a15ff7a17d889942
comment:21 Changed 9 years ago by Vickie Lynch
Refs #4790 added text for MaxChunkSize property
Changeset: c2ecf589d16264a11e42c19c9576314c08265253
comment:22 Changed 9 years ago by Vickie Lynch
Refs #4790 determineChunking returns TableWorkspace
Changeset: c323f97fbdf63e83c863cc875d851fcc2fe27fe0
comment:23 Changed 9 years ago by Vickie Lynch
Refs #4790 strategy output from LoadEventNexus
Changeset: ca0c42c0b8b0ed0cd3a7aaae379779af59a17615
comment:24 Changed 9 years ago by Vickie Lynch
Refs #4790 estimate Gbytes of chunk
Changeset: f8cbcb798af833acf674fadf3c6a540d23b7eea7
comment:25 Changed 9 years ago by Vickie Lynch
Refs #4790 CNCS does not have total_counts
Changeset: 24a9a56b91633de9c4621a28e69042930f54735b
comment:26 Changed 9 years ago by Martyn Gigg
Refs #4790. Update EQSANSLoad due to property type change.
Changeset: 671a2dba666aff7e35a2ca34a2d5bb47e30317cf
comment:27 Changed 9 years ago by Russell Taylor
Is this the best way to implement this 'chunking detemination' business? Changing a property type carries risk because a 'getProperty' to the wrong type is only uncovered at runtime - as seen above, and we have to be sure there's nowhere else that isn't touched by automated tests.
Moreover, I'm not convinced by the idea of having the output workspace be entirely different depending on the inputs - to me that says that this is a different algorithm and should be implemented as such. Any common code could, of course, go in a helper class.
comment:28 Changed 9 years ago by Vickie Lynch
Refs #4790 separate algorithm for determineChunks
Changeset: 7da399f1d1e18a4c34b5290122c0cf2933fddf0e
comment:29 Changed 9 years ago by Vickie Lynch
Refs #4790 reverting Martyn's change
Changeset: 31fd1492e734eb22630069556e48ac43c6c6c8d8
comment:30 Changed 9 years ago by Vickie Lynch
Refs #4790 using new python table workspace features
Changeset: 13bec6697f4c0ef75efc97eaa8da05a6b8aa2352
comment:31 Changed 9 years ago by Vickie Lynch
Refs #4790 use table workspace iterator
Changeset: 6fa2cc11cd6d7229bddfe5b280be6f9d64f9a557
comment:32 Changed 9 years ago by Vickie Lynch
Refs #4790 using new python table workspace features
Changeset: 13bec6697f4c0ef75efc97eaa8da05a6b8aa2352
comment:33 Changed 9 years ago by Vickie Lynch
Refs #4790 use table workspace iterator
Changeset: 6fa2cc11cd6d7229bddfe5b280be6f9d64f9a557
comment:34 Changed 9 years ago by Vickie Lynch
Refs #4790 do not sum proton_charge for chunks
Changeset: 026a5e1f1938c6715a1482b21a4caa325003bde1
comment:35 Changed 9 years ago by Vickie Lynch
Refs #4790 option to run chunks using mpi
Changeset: 08294d7b02dd78cb70ef028893d1a9c7b698a927
comment:36 Changed 9 years ago by Vickie Lynch
Refs #4790 SumSpectra not needed
Changeset: 6f406be659612e5b83407f041ce205ef91f3a757
comment:37 Changed 9 years ago by Vickie Lynch
Refs #4790 use Tableworkspace features from old api
Changeset: 0554ff71ea40f212e6d51cddcb18e46976e957d1
comment:38 Changed 9 years ago by Vickie Lynch
Refs #4790 Pete's suggestion to recalculate from log
Changeset: 55dfd56c820c08183b592fec6347e186893f491e
comment:39 Changed 9 years ago by Vickie Lynch
Refs #4790 hide mpi commands for non-mpi compile
Changeset: 0c44190ab83e99c939bcf88a0093c89cc6c181b8
comment:40 Changed 9 years ago by Vickie Lynch
Refs #4790 do not sum proton_charge for chunks
Changeset: 026a5e1f1938c6715a1482b21a4caa325003bde1
comment:41 Changed 9 years ago by Vickie Lynch
Refs #4790 option to run chunks using mpi
Changeset: 08294d7b02dd78cb70ef028893d1a9c7b698a927
comment:42 Changed 9 years ago by Vickie Lynch
Refs #4790 SumSpectra not needed
Changeset: 6f406be659612e5b83407f041ce205ef91f3a757
comment:43 Changed 9 years ago by Vickie Lynch
Refs #4790 use Tableworkspace features from old api
Changeset: 0554ff71ea40f212e6d51cddcb18e46976e957d1
comment:44 Changed 9 years ago by Vickie Lynch
Refs #4790 Pete's suggestion to recalculate from log
Changeset: 55dfd56c820c08183b592fec6347e186893f491e
comment:45 Changed 9 years ago by Vickie Lynch
Refs #4790 hide mpi commands for non-mpi compile
Changeset: 0c44190ab83e99c939bcf88a0093c89cc6c181b8
comment:46 Changed 9 years ago by Vickie Lynch
Refs #4790 remove not needed import lines
Changeset: 813e7c687db98ec78f3d2e91b401be4db547116b
comment:47 Changed 9 years ago by Vickie Lynch
Refs #4790 remove unneeded line
Changeset: 605e8228767a966185123d7948e28aaeac2d3d10
comment:48 Changed 9 years ago by Vickie Lynch
Refs #4790 normalize after integrateProtonCharge
Changeset: a108fbd7d02ddec3751a6fe21d142595c3a3a853
comment:49 Changed 9 years ago by Vickie Lynch
Refs #4790 move normalization
Changeset: bb5ba90241ed468bd1282833e030b8f24a102257
comment:50 Changed 9 years ago by Vickie Lynch
Refs #4790 only rank=0 after gather
Changeset: 42b032c6e67c236ec46e8ad5fe15149426ed6d0e
comment:51 Changed 9 years ago by Vickie Lynch
Refs #4790 use total events for prenexus; added test
Changeset: 9113f490812290855c61781fda756f35cb602870
comment:52 Changed 9 years ago by Vickie Lynch
Refs #4790 do mpi tasks in chunks
Changeset: 71eb6e9c3e92b8203f431f9f19e186797b370964
comment:53 Changed 9 years ago by Vickie Lynch
Refs #4790 output workspace needs proton charge
Changeset: 7bf5721bfae5fb3ec9106fc49eeab43b128ee0a3
comment:54 Changed 8 years ago by Vickie Lynch
Refs #4790 factor of 2 for compression
Changeset: 666af3853618dc636890e6880cbf439929340c20
comment:55 Changed 8 years ago by Vickie Lynch
Refs #4790 factor of 2 for compression
Changeset: 666af3853618dc636890e6880cbf439929340c20
comment:56 Changed 8 years ago by Vickie Lynch
Refs #4790 do part skipped by serial run
Changeset: c8bff0a50a25085e1462e1bfab8ce3284179bfec
comment:57 Changed 8 years ago by Vickie Lynch
Refs #4790 fix filterBadPulse for vnoise
Changeset: 2a9a179c2f7e0c7cca921c836a1f7fd1a7b5aa5a
comment:58 Changed 8 years ago by Vickie Lynch
Refs #4790 do part skipped by serial run
Changeset: c8bff0a50a25085e1462e1bfab8ce3284179bfec
comment:59 Changed 8 years ago by Vickie Lynch
Refs #4790 fix filterBadPulse for vnoise
Changeset: 2a9a179c2f7e0c7cca921c836a1f7fd1a7b5aa5a
comment:60 Changed 8 years ago by Vickie Lynch
- Status changed from accepted to verify
- Resolution set to fixed
SNSPowderReduction and SNSPowderReduction2 with chunking have the same results from the system test of PG3 data.
comment:61 Changed 8 years ago by Shelly Ren
- Status changed from verify to verifying
- Tester set to Shelly Ren
comment:62 Changed 8 years ago by Shelly Ren
- Status changed from verifying to closed
Tested SNSPowderReduction w/ and w/o chunking.
PG3 files were small, whether we set MaxChunkSize to 0 or 16(GB), it should work. So we tested PG3 3676 w/ and w/o chunking, both ran successfully.
NOMAD datasets were too large to fit into the memory. By setting MaxChunkSize = 16, NOMAD users could reduce their data utilizing 5 chunks for run 3778.
comment:63 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 5637
