Ticket #9509 (closed: fixed)
IntegratePeaksHybrid aka IntegratePeaksUsingClusters (Hybrid mode)
Reported by: | Owen Arnold | Owned by: | Owen Arnold |
---|---|---|---|
Priority: | major | Milestone: | Release 3.2 |
Component: | Diffraction | Keywords: | |
Cc: | Blocked By: | #9360, #9523 | |
Blocking: | #9863, #9864 | Tester: | Samuel Jackson |
Description
I think I have a slightly more sophisticated, but ultimately all-round better variant of this algorithm.
The current version of this algorithm works really nicely, but it has two problems:
- You are forced to apply a fine binning over the entire range in q. This is very memory expensive and given that you don't care about ~98% of the image.
- The choice of background has to be provided, and this is global to all regions of the workspace. This again, is not very sensible, and results in a mixture conservative, and extremely liberal cluster sizes.
Here is my outline of a new algorithm which will solve both of these problems. It could (for arguments sake) mostly be implemented in Python, as it glues together a few existing processing units including the existing IntegratePeaksUsingClusters, although I will implement it in C++.
- The user provides an estimate of the Peak Radius, equivalent to the OuterBackgroundRadius in IntegratePeaksMD. The user optionally provides the Q extents.
- If the user has not provided the Q, extents, these are calculated. The Q extents are used to calculate the minimum bin size in each dimension.
- The algorithm looks at each peak in the PeaksWorkspace, and calculates a local region for BinMD based on the information above, including the outer radius.
- An MDHistoWorkspace is produced, this is iterated over to establish the average SignalValue (ASV).
- IntegratePeaksUsingClusters is run on that single PeaksWorkspace using the MDHistoWorkspace as the source and the ASV as the Threshold.
- The Output single peaks workspace is appended onto a global output PeaksWorkspace.
- The process continues until all unintegrated peaks have been integrated.
Change History
comment:2 Changed 6 years ago by Owen Arnold
- Summary changed from IntegratePeaksUsingClusters (Hybrid) to IntegratePeaksHybrid aka IntegratePeaksUsingClusters (Hybrid mode)
comment:3 Changed 6 years ago by Owen Arnold
- Status changed from assigned to inprogress
refs #9509. Skeleton algorithm.
Changeset: 2ed733e62cf610e84f21d754fe1702fa429b2cd9
comment:4 Changed 6 years ago by Owen Arnold
refs #9509. Works for single peak.
Simple implementation and testing for a single peak. Lots of test and development work to do yet.
Changeset: 602dc6d77aaa767b0c7fa3c4a9bc33f37d74579e
comment:5 Changed 6 years ago by Owen Arnold
refs #9509. Additional testing of functionality.
Changeset: 34071dadb47e488c871e23592dc5e7442c1f943e
comment:6 Changed 6 years ago by Owen Arnold
refs #9509. Fix logic error and perf tests.
Add performance tests. Currently running at 2.2 seconds on my OSX debug build.
Changeset: 8dcc86eeb08d4feda593839998becdcdd0823820
comment:7 Changed 6 years ago by Owen Arnold
refs #9509. Single threaded CCL works faster.
Small adjustment to the code to use single threaded CCL. Results in a 4-5 times speed up. This is because we have n * small images being processed rather than one large one. The overhead of the parallel execution completely dominates the speed up acheived from the asynchronous thread execution in this case.
The next step will be to make the algorithm parallel with respect to the individual peaks in the peak list. This should result in a further speedup.
Changeset: db3708206a0be77fd6673c62c23c239b99ba67bc
comment:8 Changed 6 years ago by Owen Arnold
refs #9509. Parallel over peaks.
Each peak is integrated in parallel. Execution time for the performance tests is now 0.07 seconds. This is nearly an order of magnitude faster than the last modification, and about 40 times faster than the original implementation.
Changeset: bf81434474e267ae7c8dfe919432d3bfbffe3649
comment:9 Changed 6 years ago by Owen Arnold
refs #9509. More code documentation.
Changeset: 74fb79eab60bb33af970ee6d9082909203b09769
comment:10 Changed 6 years ago by Owen Arnold
refs #9509. Fix GCC warnings and errors.
Changeset: a19e2d3e609d535e46fff2530164fdb315273afd
comment:11 Changed 6 years ago by Owen Arnold
refs #9509. Initialize frameworkmanager.
Changeset: 9e2b97baec7c1121a76dc6d6c4c30d5e1180e7fd
comment:12 Changed 6 years ago by Owen Arnold
refs #9509. Documentation parsing error.
Changeset: adb111d2f620d2c573daff4b74e0b2a70a4cb7e9
comment:13 Changed 6 years ago by Owen Arnold
Need to avoid reporting every time CCL is run.
comment:15 Changed 6 years ago by Owen Arnold
refs #9509. Regression test for CCL problem.
Changeset: fec47ff7d6fe9b899b84836d57f6d38eeffc4c73
comment:16 Changed 6 years ago by Owen Arnold
refs #9509. Fix for CCL problem.
Changeset: 8cbfa4c566f668ca6a02871abd8d815d41e358cc
comment:17 Changed 6 years ago by Owen Arnold
refs #9509. Sigma values reported corrected.
Changeset: 9d544c76ca8c8f6bc9ff2fd3835b71a7b3900042
comment:19 Changed 6 years ago by Owen Arnold
refs #9509. Remove unnecessary code. Fix neighbour issue.
Changeset: 065fdcfdd1e63a1c93c2a2ae4b617c1eec511ec3
comment:20 Changed 6 years ago by Owen Arnold
refs #9509. Master progress reporting.
Changeset: d4ccecfb4eb22dbf5e847c5e5b61ea0036da6c8b
comment:21 Changed 6 years ago by Owen Arnold
Tester: Run the following and look at the ordered PeaksWorkspaces generated. You should see very good agreement between the ordering of the rows, and the Intens and SigIntens columns between the existing (IntegratePeaksMD) and newer (IntegratePeaksHybrid) algorithm.
IntegratePeaksHybrid runs in 0.2 seconds on my laptop. Which is much faster than the 1.8 seconds it takes for IntegratePeaksMD to run.
# Basic parameters for Triphylite Crystal #Name of the workspaces to create ws_name = "TOPAZ_3132" filename = ws_name +"_event.nxs" ws = LoadEventNexus(Filename=filename,FilterByTofMin=3000, FilterByTofMax=16000) # ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- # Part 1. Basic Reduction # Spherical Absorption and Lorentz Corrections ws = AnvredCorrection(InputWorkspace=ws, LinearScatteringCoef=0.451, LinearAbsorptionCoef=0.993, Radius=0.14) # Convert to Q space LabQ = ConvertToDiffractionMDWorkspace(InputWorkspace=ws, LorentzCorrection='0', OutputDimensions='Q (lab frame)', SplitInto=2, SplitThreshold=150) # Find peaks PeaksLattice = FindPeaksMD(InputWorkspace=LabQ,MaxPeaks=100) # 3d integration to centroid peaks PeaksLattice = CentroidPeaksMD(InputWorkspace=LabQ, PeakRadius=0.12, PeaksWorkspace=PeaksLattice) # Find the UB matrix using the peaks and known lattice parameters FindUBUsingLatticeParameters(PeaksWorkspace=PeaksLattice, a=10.3522, b=6.0768, c=4.7276, alpha=90, beta=90, gamma=90, NumInitial=20, Tolerance=0.12) # And index to HKL IndexPeaks(PeaksWorkspace=PeaksLattice, Tolerance=0.12) PeaksLattice_Integrated_Hybrid, ClusterImages = IntegratePeaksHybrid(InputWorkspace=LabQ, PeaksWorkspace=PeaksLattice, BackgroundOuterRadius=0.4) PeaksLattice_Integrated_MD = IntegratePeaksMD(InputWorkspace=LabQ,PeakRadius=0.12,PeaksWorkspace=PeaksLattice) PeaksLattice_Integrated_Hybrid = SortPeaksWorkspace(PeaksLattice_Integrated_Hybrid, 'Intens', False) PeaksLattice_Integrated_MD = SortPeaksWorkspace(PeaksLattice_Integrated_MD, 'Intens', False)
comment:21 Changed 6 years ago by Owen Arnold
refs #9509. Remove old API methods calls.
Changeset: 4d498b8ac5b0bd296cbdf5cfc7403268bd135bcb
comment:22 Changed 6 years ago by Owen Arnold
refs #9509. Fix new API usage.
Changeset: acc29325fc9b270cdecf70daec2fe39bcf45a978
comment:23 Changed 6 years ago by Owen Arnold
refs #9509. Remove unused var.
Changeset: b2448540e75c33e3df4468f472295e84f6e853c2
comment:24 Changed 6 years ago by Owen Arnold
- Status changed from inprogress to verify
- Resolution set to fixed
comment:25 Changed 6 years ago by Owen Arnold
Resolve merge conflicts refs #9509.
Merge branch 'master' into feature/9509_integrate_peaks_hybrid
Conflicts:
Code/Mantid/Framework/Crystal/src/IntegratePeaksUsingClusters.cpp
Changeset: 4ddb7c961ca401e4226fae3ee1191fcb08043282
comment:26 Changed 6 years ago by Samuel Jackson
- Status changed from verify to verifying
- Tester set to Samuel Jackson
comment:27 Changed 6 years ago by Samuel Jackson
- Status changed from verifying to closed
Merge remote-tracking branch 'origin/feature/9509_integrate_peaks_hybrid'
Full changeset: be6b59999ecbc29707ef006c8a77b9d7c8bb87cd
comment:28 Changed 6 years ago by Samuel Jackson
There were some minor issues with documentation. I created ticket #9661 for this so we can get this into master for the release.
comment:31 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 10352