Ticket #9509 (closed: fixed)

Opened 6 years ago

Last modified 5 years ago

IntegratePeaksHybrid aka IntegratePeaksUsingClusters (Hybrid mode)

Reported by: Owen Arnold Owned by: Owen Arnold
Priority: major Milestone: Release 3.2
Component: Diffraction Keywords:
Cc: Blocked By: #9360, #9523
Blocking: #9863, #9864 Tester: Samuel Jackson

Description

I think I have a slightly more sophisticated, but ultimately all-round better variant of this algorithm.

The current version of this algorithm works really nicely, but it has two problems:

  • You are forced to apply a fine binning over the entire range in q. This is very memory expensive and given that you don't care about ~98% of the image.
  • The choice of background has to be provided, and this is global to all regions of the workspace. This again, is not very sensible, and results in a mixture conservative, and extremely liberal cluster sizes.

Here is my outline of a new algorithm which will solve both of these problems. It could (for arguments sake) mostly be implemented in Python, as it glues together a few existing processing units including the existing IntegratePeaksUsingClusters, although I will implement it in C++.

  • The user provides an estimate of the Peak Radius, equivalent to the OuterBackgroundRadius in IntegratePeaksMD. The user optionally provides the Q extents.
  • If the user has not provided the Q, extents, these are calculated. The Q extents are used to calculate the minimum bin size in each dimension.
  • The algorithm looks at each peak in the PeaksWorkspace, and calculates a local region for BinMD based on the information above, including the outer radius.
  • An MDHistoWorkspace is produced, this is iterated over to establish the average SignalValue (ASV).
  • IntegratePeaksUsingClusters is run on that single PeaksWorkspace using the MDHistoWorkspace as the source and the ASV as the Threshold.
  • The Output single peaks workspace is appended onto a global output PeaksWorkspace.
  • The process continues until all unintegrated peaks have been integrated.

Change History

comment:1 Changed 6 years ago by Nick Draper

  • Status changed from new to assigned

comment:2 Changed 6 years ago by Owen Arnold

  • Summary changed from IntegratePeaksUsingClusters (Hybrid) to IntegratePeaksHybrid aka IntegratePeaksUsingClusters (Hybrid mode)

comment:3 Changed 6 years ago by Owen Arnold

  • Status changed from assigned to inprogress

refs #9509. Skeleton algorithm.

Changeset: 2ed733e62cf610e84f21d754fe1702fa429b2cd9

comment:4 Changed 6 years ago by Owen Arnold

refs #9509. Works for single peak.

Simple implementation and testing for a single peak. Lots of test and development work to do yet.

Changeset: 602dc6d77aaa767b0c7fa3c4a9bc33f37d74579e

comment:5 Changed 6 years ago by Owen Arnold

refs #9509. Additional testing of functionality.

Changeset: 34071dadb47e488c871e23592dc5e7442c1f943e

comment:6 Changed 6 years ago by Owen Arnold

refs #9509. Fix logic error and perf tests.

Add performance tests. Currently running at 2.2 seconds on my OSX debug build.

Changeset: 8dcc86eeb08d4feda593839998becdcdd0823820

comment:7 Changed 6 years ago by Owen Arnold

refs #9509. Single threaded CCL works faster.

Small adjustment to the code to use single threaded CCL. Results in a 4-5 times speed up. This is because we have n * small images being processed rather than one large one. The overhead of the parallel execution completely dominates the speed up acheived from the asynchronous thread execution in this case.

The next step will be to make the algorithm parallel with respect to the individual peaks in the peak list. This should result in a further speedup.

Changeset: db3708206a0be77fd6673c62c23c239b99ba67bc

comment:8 Changed 6 years ago by Owen Arnold

refs #9509. Parallel over peaks.

Each peak is integrated in parallel. Execution time for the performance tests is now 0.07 seconds. This is nearly an order of magnitude faster than the last modification, and about 40 times faster than the original implementation.

Changeset: bf81434474e267ae7c8dfe919432d3bfbffe3649

comment:9 Changed 6 years ago by Owen Arnold

refs #9509. More code documentation.

Changeset: 74fb79eab60bb33af970ee6d9082909203b09769

comment:10 Changed 6 years ago by Owen Arnold

refs #9509. Fix GCC warnings and errors.

Changeset: a19e2d3e609d535e46fff2530164fdb315273afd

comment:11 Changed 6 years ago by Owen Arnold

refs #9509. Initialize frameworkmanager.

Changeset: 9e2b97baec7c1121a76dc6d6c4c30d5e1180e7fd

comment:12 Changed 6 years ago by Owen Arnold

refs #9509. Documentation parsing error.

Changeset: adb111d2f620d2c573daff4b74e0b2a70a4cb7e9

comment:13 Changed 6 years ago by Owen Arnold

Need to avoid reporting every time CCL is run.

comment:15 Changed 6 years ago by Owen Arnold

refs #9509. Regression test for CCL problem.

Changeset: fec47ff7d6fe9b899b84836d57f6d38eeffc4c73

comment:16 Changed 6 years ago by Owen Arnold

refs #9509. Fix for CCL problem.

Changeset: 8cbfa4c566f668ca6a02871abd8d815d41e358cc

comment:17 Changed 6 years ago by Owen Arnold

refs #9509. Sigma values reported corrected.

Changeset: 9d544c76ca8c8f6bc9ff2fd3835b71a7b3900042

comment:18 Changed 6 years ago by Jay Rainey

  • Blocked By 9523 added

comment:19 Changed 6 years ago by Owen Arnold

refs #9509. Remove unnecessary code. Fix neighbour issue.

Changeset: 065fdcfdd1e63a1c93c2a2ae4b617c1eec511ec3

comment:20 Changed 6 years ago by Owen Arnold

refs #9509. Master progress reporting.

Changeset: d4ccecfb4eb22dbf5e847c5e5b61ea0036da6c8b

comment:21 Changed 6 years ago by Owen Arnold

Tester: Run the following and look at the ordered PeaksWorkspaces generated. You should see very good agreement between the ordering of the rows, and the Intens and SigIntens columns between the existing (IntegratePeaksMD) and newer (IntegratePeaksHybrid) algorithm.

IntegratePeaksHybrid runs in 0.2 seconds on my laptop. Which is much faster than the 1.8 seconds it takes for IntegratePeaksMD to run.

# Basic parameters  for  Triphylite Crystal
#Name of the workspaces to create
ws_name = "TOPAZ_3132"
filename = ws_name +"_event.nxs"
ws = LoadEventNexus(Filename=filename,FilterByTofMin=3000, FilterByTofMax=16000)

# -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
# Part 1. Basic Reduction

# Spherical Absorption and Lorentz Corrections
ws = AnvredCorrection(InputWorkspace=ws, LinearScatteringCoef=0.451, LinearAbsorptionCoef=0.993, Radius=0.14)

# Convert to Q space
LabQ = ConvertToDiffractionMDWorkspace(InputWorkspace=ws, LorentzCorrection='0',
        OutputDimensions='Q (lab frame)', SplitInto=2, SplitThreshold=150)
        
# Find peaks
PeaksLattice = FindPeaksMD(InputWorkspace=LabQ,MaxPeaks=100)

# 3d integration to centroid peaks
PeaksLattice = CentroidPeaksMD(InputWorkspace=LabQ,
        PeakRadius=0.12, PeaksWorkspace=PeaksLattice)
        
# Find the UB matrix using the peaks and known lattice parameters
FindUBUsingLatticeParameters(PeaksWorkspace=PeaksLattice, a=10.3522, b=6.0768, c=4.7276,
                alpha=90, beta=90, gamma=90, NumInitial=20, Tolerance=0.12)
                
# And index to HKL            
IndexPeaks(PeaksWorkspace=PeaksLattice, Tolerance=0.12)


PeaksLattice_Integrated_Hybrid, ClusterImages = IntegratePeaksHybrid(InputWorkspace=LabQ, PeaksWorkspace=PeaksLattice, BackgroundOuterRadius=0.4)

PeaksLattice_Integrated_MD = IntegratePeaksMD(InputWorkspace=LabQ,PeakRadius=0.12,PeaksWorkspace=PeaksLattice)

PeaksLattice_Integrated_Hybrid = SortPeaksWorkspace(PeaksLattice_Integrated_Hybrid, 'Intens', False)
PeaksLattice_Integrated_MD = SortPeaksWorkspace(PeaksLattice_Integrated_MD, 'Intens', False)

comment:21 Changed 6 years ago by Owen Arnold

refs #9509. Remove old API methods calls.

Changeset: 4d498b8ac5b0bd296cbdf5cfc7403268bd135bcb

comment:22 Changed 6 years ago by Owen Arnold

refs #9509. Fix new API usage.

Changeset: acc29325fc9b270cdecf70daec2fe39bcf45a978

comment:23 Changed 6 years ago by Owen Arnold

refs #9509. Remove unused var.

Changeset: b2448540e75c33e3df4468f472295e84f6e853c2

comment:24 Changed 6 years ago by Owen Arnold

  • Status changed from inprogress to verify
  • Resolution set to fixed

comment:25 Changed 6 years ago by Owen Arnold

Resolve merge conflicts refs #9509.

Merge branch 'master' into feature/9509_integrate_peaks_hybrid

Conflicts:

Code/Mantid/Framework/Crystal/src/IntegratePeaksUsingClusters.cpp

Changeset: 4ddb7c961ca401e4226fae3ee1191fcb08043282

comment:26 Changed 6 years ago by Samuel Jackson

  • Status changed from verify to verifying
  • Tester set to Samuel Jackson

comment:27 Changed 6 years ago by Samuel Jackson

  • Status changed from verifying to closed

Merge remote-tracking branch 'origin/feature/9509_integrate_peaks_hybrid'

Full changeset: be6b59999ecbc29707ef006c8a77b9d7c8bb87cd

comment:28 Changed 6 years ago by Samuel Jackson

There were some minor issues with documentation. I created ticket #9661 for this so we can get this into master for the release.

Last edited 6 years ago by Samuel Jackson (previous) (diff)

comment:29 Changed 6 years ago by Owen Arnold

  • Blocking 9863 added

comment:30 Changed 6 years ago by Owen Arnold

  • Blocking 9864 added

comment:31 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 10352

Note: See TracTickets for help on using tickets.