IntegratePeaksHybrid aka IntegratePeaksUsingClusters (Hybrid mode)

[Owen Arnold]

I think I have a slightly more sophisticated, but ultimately all-round better variant of this algorithm.

The current version of this algorithm works really nicely, but it has two problems:

• You are forced to apply a fine binning over the entire range in q. This is very memory expensive and given that you don't care about ~98% of the image.
• The choice of background has to be provided, and this is global to all regions of the workspace. This again, is not very sensible, and results in a mixture conservative, and extremely liberal cluster sizes.

Here is my outline of a new algorithm which will solve both of these problems. It could (for arguments sake) mostly be implemented in Python, as it glues together a few existing processing units including the existing IntegratePeaksUsingClusters, although I will implement it in C++.

• The user provides an estimate of the Peak Radius, equivalent to the OuterBackgroundRadius in IntegratePeaksMD. The user optionally provides the Q extents.
• If the user has not provided the Q, extents, these are calculated. The Q extents are used to calculate the minimum bin size in each dimension.
• The algorithm looks at each peak in the PeaksWorkspace, and calculates a local region for BinMD based on the information above, including the outer radius.
• An MDHistoWorkspace is produced, this is iterated over to establish the average SignalValue (ASV).
• IntegratePeaksUsingClusters is run on that single PeaksWorkspace using the MDHistoWorkspace as the source and the ASV as the Threshold.
• The Output single peaks workspace is appended onto a global output PeaksWorkspace.
• The process continues until all unintegrated peaks have been integrated.

[Owen Arnold]

refs #9509. Skeleton algorithm.

Changeset: 2ed733e62cf610e84f21d754fe1702fa429b2cd9

[Owen Arnold]

refs #9509. Works for single peak.

Simple implementation and testing for a single peak. Lots of test and development work to do yet.

Changeset: 602dc6d77aaa767b0c7fa3c4a9bc33f37d74579e

[Owen Arnold]

refs #9509. Additional testing of functionality.

Changeset: 34071dadb47e488c871e23592dc5e7442c1f943e

[Owen Arnold]

refs #9509. Fix logic error and perf tests.

Add performance tests. Currently running at 2.2 seconds on my OSX debug build.

Changeset: 8dcc86eeb08d4feda593839998becdcdd0823820

[Owen Arnold]

refs #9509. Single threaded CCL works faster.

Small adjustment to the code to use single threaded CCL. Results in a 4-5 times speed up. This is because we have n * small images being processed rather than one large one. The overhead of the parallel execution completely dominates the speed up acheived from the asynchronous thread execution in this case.

The next step will be to make the algorithm parallel with respect to the individual peaks in the peak list. This should result in a further speedup.

Changeset: db3708206a0be77fd6673c62c23c239b99ba67bc

[Owen Arnold]

refs #9509. Parallel over peaks.

Each peak is integrated in parallel. Execution time for the performance tests is now 0.07 seconds. This is nearly an order of magnitude faster than the last modification, and about 40 times faster than the original implementation.

Changeset: bf81434474e267ae7c8dfe919432d3bfbffe3649

[Owen Arnold]

refs #9509. More code documentation.

Changeset: 74fb79eab60bb33af970ee6d9082909203b09769

[Owen Arnold]

refs #9509. Fix GCC warnings and errors.

Changeset: a19e2d3e609d535e46fff2530164fdb315273afd

[Owen Arnold]

refs #9509. Initialize frameworkmanager.

Changeset: 9e2b97baec7c1121a76dc6d6c4c30d5e1180e7fd

[Owen Arnold]

refs #9509. Documentation parsing error.

Changeset: adb111d2f620d2c573daff4b74e0b2a70a4cb7e9

[Owen Arnold]

Need to avoid reporting every time CCL is run.

[Owen Arnold]

refs #9509. Regression test for CCL problem.

Changeset: fec47ff7d6fe9b899b84836d57f6d38eeffc4c73

[Owen Arnold]

refs #9509. Fix for CCL problem.

Changeset: 8cbfa4c566f668ca6a02871abd8d815d41e358cc

[Owen Arnold]

refs #9509. Sigma values reported corrected.

Changeset: 9d544c76ca8c8f6bc9ff2fd3835b71a7b3900042

[Owen Arnold]

refs #9509. Remove unnecessary code. Fix neighbour issue.

Changeset: 065fdcfdd1e63a1c93c2a2ae4b617c1eec511ec3

[Owen Arnold]

refs #9509. Master progress reporting.

Changeset: d4ccecfb4eb22dbf5e847c5e5b61ea0036da6c8b

[Owen Arnold]

Tester: Run the following and look at the ordered PeaksWorkspaces generated. You should see very good agreement between the ordering of the rows, and the Intens and SigIntens columns between the existing (IntegratePeaksMD) and newer (IntegratePeaksHybrid) algorithm.

IntegratePeaksHybrid runs in 0.2 seconds on my laptop. Which is much faster than the 1.8 seconds it takes for IntegratePeaksMD to run.

# Basic parameters  for  Triphylite Crystal
#Name of the workspaces to create
ws_name = "TOPAZ_3132"
filename = ws_name +"_event.nxs"
ws = LoadEventNexus(Filename=filename,FilterByTofMin=3000, FilterByTofMax=16000)

# -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
# Part 1. Basic Reduction

# Spherical Absorption and Lorentz Corrections
ws = AnvredCorrection(InputWorkspace=ws, LinearScatteringCoef=0.451, LinearAbsorptionCoef=0.993, Radius=0.14)

# Convert to Q space
LabQ = ConvertToDiffractionMDWorkspace(InputWorkspace=ws, LorentzCorrection='0',
        OutputDimensions='Q (lab frame)', SplitInto=2, SplitThreshold=150)
        
# Find peaks
PeaksLattice = FindPeaksMD(InputWorkspace=LabQ,MaxPeaks=100)

# 3d integration to centroid peaks
PeaksLattice = CentroidPeaksMD(InputWorkspace=LabQ,
        PeakRadius=0.12, PeaksWorkspace=PeaksLattice)
        
# Find the UB matrix using the peaks and known lattice parameters
FindUBUsingLatticeParameters(PeaksWorkspace=PeaksLattice, a=10.3522, b=6.0768, c=4.7276,
                alpha=90, beta=90, gamma=90, NumInitial=20, Tolerance=0.12)
                
# And index to HKL            
IndexPeaks(PeaksWorkspace=PeaksLattice, Tolerance=0.12)


PeaksLattice_Integrated_Hybrid, ClusterImages = IntegratePeaksHybrid(InputWorkspace=LabQ, PeaksWorkspace=PeaksLattice, BackgroundOuterRadius=0.4)

PeaksLattice_Integrated_MD = IntegratePeaksMD(InputWorkspace=LabQ,PeakRadius=0.12,PeaksWorkspace=PeaksLattice)

PeaksLattice_Integrated_Hybrid = SortPeaksWorkspace(PeaksLattice_Integrated_Hybrid, 'Intens', False)
PeaksLattice_Integrated_MD = SortPeaksWorkspace(PeaksLattice_Integrated_MD, 'Intens', False)

[Owen Arnold]

refs #9509. Remove old API methods calls.

Changeset: 4d498b8ac5b0bd296cbdf5cfc7403268bd135bcb

[Owen Arnold]

refs #9509. Fix new API usage.

Changeset: acc29325fc9b270cdecf70daec2fe39bcf45a978

[Owen Arnold]

refs #9509. Remove unused var.

Changeset: b2448540e75c33e3df4468f472295e84f6e853c2

[Owen Arnold]

Resolve merge conflicts refs #9509.

Merge branch 'master' into feature/9509_integrate_peaks_hybrid

Conflicts:

Code/Mantid/Framework/Crystal/src/IntegratePeaksUsingClusters.cpp

Changeset: 4ddb7c961ca401e4226fae3ee1191fcb08043282

[Samuel Jackson]

Merge remote-tracking branch 'origin/feature/9509_integrate_peaks_hybrid'

Full changeset: ​be6b59999ecbc29707ef006c8a77b9d7c8bb87cd

[Samuel Jackson]

There were some minor issues with documentation. I created ticket #9661 for this so we can get this into master for the release.

[Stuart Campbell]

This ticket has been transferred to github issue ​10352