Ticket #2023 (closed: fixed)

Opened 10 years ago

Last modified 5 years ago

Create gui for correcting sns Topaz data for ISAW peak integration

Reported by: Vickie Lynch Owned by: Vickie Lynch
Priority: major Milestone: Iteration 27
Component: Mantid Keywords:
Cc: Blocked By:
Blocking: Tester: Andrei Savici

Description

Pixel by pixel background subtraction and division by vanadium. Use SaveSNSNexus algorithm to save data for ISAW.

Change History

comment:1 Changed 10 years ago by Vickie Lynch

(In [7943]) Refs #2023 Saving corrected TOPAZ data for ISAW

comment:2 Changed 10 years ago by Vickie Lynch

(In [7947]) Refs#2023 Defaults for input changed for SNSSingleCrystalReduction

comment:3 Changed 10 years ago by Vickie Lynch

(In [8019]) Refs #2023 Reduced memory required for SNSSingleCrystalReduction

comment:4 Changed 10 years ago by Nick Draper

  • Milestone changed from Iteration 26 to Iteration 27

Bulk move of tickets to iteration 27, if your ticket is essential for Iteration 26 then move it back.

comment:5 Changed 10 years ago by Vickie Lynch

  • Status changed from new to accepted

comment:6 Changed 10 years ago by Vickie Lynch

(In [8624]) Refs #2023 Vanadium data focussed for each detector before division in SNSSingleCrystalReduction

comment:7 Changed 10 years ago by Vickie Lynch

(In [8660]) Refs #2023 Created DivideVanadium algorithm because python code for division was too slow in SNSSingleCrystalReduction

comment:8 Changed 10 years ago by Vickie Lynch

(In [8846]) Refs #2023 Testing loading events from only one detector bank for large TOPAZ event data

comment:9 Changed 10 years ago by Janik Zikovsky

(In [8913]) Refs #2023: Minor changes to SNSSingleCrystalReduction

comment:10 Changed 10 years ago by Janik Zikovsky

(In [8939]) Refs #2023: SCD script minor changes.

comment:11 Changed 10 years ago by Janik Zikovsky

---------------------------------------------------------------------------
Tue Jan 25 18:07:47 2011: Script execution started.
---------------------------------------------------------------------------
sample loaded. Time =  106.625463963
background loaded. Time =  273.816184044
subtraction. Time =  370.734992027
vanadium loaded. Time =  1095.36750889
vanadium sorted. Time =  1097.07427907
sample/vanadium. Time =  4787.487849
---------------------------------------------------------------------------
Tue Jan 25 19:27:37 2011: Script execution completed successfully.
---------------------------------------------------------------------------

Seems that the division is by far the slowest step.

comment:12 Changed 10 years ago by Janik Zikovsky

(In [8946]) Refs #2277: Divide algorithm has new parameter to perform division by grouped spectra, where workspace indices do not match. e.g. sample / vanadium (where vanadium was averaged per bank). Refs #2023.

comment:13 Changed 10 years ago by Janik Zikovsky

Better if you convert vanadium to Workspace2D and delete the Vanadium event workspace, but still slow.

---------------------------------------------------------------------------
Wed Jan 26 12:07:56 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  172.51343894
bank1,bank2,bank3,bank4,bank5,bank6,bank7,bank8,bank9,bank10,bank11,bank12,bank13,bank14,bank15
vanadium binned and focussed. Time =  370.558913946
sample loaded. Time =  536.524903059
background loaded. Time =  800.194772959
subtraction. Time =  1032.45306206
sample/vanadium. Time =  2651.39089489
---------------------------------------------------------------------------
Wed Jan 26 12:52:08 2011: Script execution completed successfully.
---------------------------------------------------------------------------

comment:14 Changed 10 years ago by Janik Zikovsky

Same as the previous one, but without LD_PRELOAD=tcmalloc.

---------------------------------------------------------------------------
Wed Jan 26 14:36:08 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  186.312375069
bank1,bank2,bank3,bank4,bank5,bank6,bank7,bank8,bank9,bank10,bank11,bank12,bank13,bank14,bank15
vanadium binned and focussed. Time =  309.687546968
sample loaded. Time =  423.059288025
background loaded. Time =  627.371264935
subtraction. Time =  703.581522942
sample/vanadium. Time =  1165.15904808
---------------------------------------------------------------------------
Wed Jan 26 14:55:34 2011: Script execution completed successfully.
---------------------------------------------------------------------------

comment:15 Changed 10 years ago by Janik Zikovsky

Linked to tcmalloc:

---------------------------------------------------------------------------
Wed Jan 26 15:05:57 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  174.915987015
bank1,bank2,bank3,bank4,bank5,bank6,bank7,bank8,bank9,bank10,bank11,bank12,bank13,bank14,bank15
vanadium binned and focussed. Time =  319.589958191
sample loaded. Time =  447.941715002
background loaded. Time =  750.023055077
subtraction. Time =  1083.41578007
approx 15 more minutes for divide.

tcmalloc seems slower! Perhaps the cache per thread is pushing memory use up until it reaches the physical limit?

comment:16 Changed 10 years ago by Janik Zikovsky

With tcmalloc, and using a call to release memory back to the system

MallocExtension::instance()->ReleaseFreeMemory();

, total time was 965 seconds. This might be significantly faster with more of the memory release calls? One more release call dropped the memory from full (14 gb) to 9 gb = the actually used memory.

comment:17 Changed 10 years ago by Janik Zikovsky

Added more calls to release free memory:

---------------------------------------------------------------------------
Wed Jan 26 16:53:59 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  173.489802122
bank1,bank2,bank3,bank4,bank5,bank6,bank7,bank8,bank9,bank10,bank11,bank12,bank13,bank14,bank15
vanadium binned and focussed. Time =  354.407647133
sample loaded. Time =  469.410380125
background loaded. Time =  637.114018917
subtraction. Time =  680.437106133
sorting (I was not sorting before - makes a big speed difference!)
sample/vanadium. Time =  about 728

Best result yet! Swap use is limited to the minus operation!

comment:18 Changed 10 years ago by Janik Zikovsky

(In [8961]) Refs #2023.

comment:19 Changed 10 years ago by Janik Zikovsky

  • TCmalloc
  • Calls to release free memory before every algorithm
  • Minus operation clears the RHS workspace. <- This looks to have saved 100 seconds.
---------------------------------------------------------------------------
Thu Jan 27 12:19:14 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  168.479260921
vanadium binned and focussed. Time =  280.20638895
sample loaded. Time =  398.196058989
background loaded. Time =  573.853898048
subtraction. Time =  584.281800032
sample/vanadium. Time =  637.521209002
---------------------------------------------------------------------------
Thu Jan 27 12:29:51 2011: Script execution completed successfully.
---------------------------------------------------------------------------

comment:20 Changed 10 years ago by Janik Zikovsky

Same, but with the use of the stock TCMalloc available by ubuntu synaptic/apt-get .

---------------------------------------------------------------------------
Thu Jan 27 13:29:44 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  173.589309931
vanadium binned and focussed. Time =  288.00959301
sample loaded. Time =  395.241351128
background loaded. Time =  565.24237895
subtraction. Time =  576.450856924
sample/vanadium. Time =  625.486242056
---------------------------------------------------------------------------
Thu Jan 27 13:40:09 2011: Script execution completed successfully.
---------------------------------------------------------------------------

About the same as the custom-built one, no segfaults or errors.

comment:21 Changed 10 years ago by Janik Zikovsky

Without TCmalloc:

---------------------------------------------------------------------------
Thu Jan 27 14:16:55 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  202.329317808
vanadium binned and focussed. Time =  332.317056894
sample loaded. Time =  446.396880865
background loaded. Time =  825.023164988
subtraction. Time =  997.560561895
(gave up - physical memory is full, lots of swapping)

comment:22 Changed 10 years ago by Vickie Lynch

(In [9134]) Refs #2023 Output from this script produces good peaks in ISAW for TOPAZ_1817

comment:23 Changed 10 years ago by Vickie Lynch

(In [9149]) Refs #2023 DivideVanadium algorithm no longer needed

comment:24 Changed 10 years ago by Vickie Lynch

(In [9150]) Refs #2023 Removed DivideVanadium from CMakeLists.txt

comment:25 Changed 10 years ago by Vickie Lynch

(In [9153]) Refs #2023 Changed argument to CreateCalFileByNames in SNSSingleCrystalReduction to match Janik's changes to algorithm

comment:26 Changed 10 years ago by Vickie Lynch

(In [9155]) Refs #2023 Uncommented FindSNSNexus in SNSSingleCrystalReduction

comment:27 Changed 10 years ago by Vickie Lynch

  • Status changed from accepted to verify
  • Resolution set to fixed

Results from ISAW look OK and it runs in 12 minutes for RunNumber=1817, BackgroundNumber=1748, VanadiumNumber=1789, and defaults for other input.

comment:28 Changed 10 years ago by Vickie Lynch

(In [9290]) Refs #2023 Comment reload statement to prevent SegFault

comment:29 Changed 10 years ago by Vickie Lynch

(In [9395]) Refs #2023 reload needed for FindSNSNexus to work; Exception at end prevents Segfault

comment:30 Changed 10 years ago by Anders Markvardsen

  • Status changed from verify to verifying
  • Tester set to Anders Markvardsen

comment:31 Changed 10 years ago by Anders Markvardsen

  • Status changed from verifying to verify
  • Tester Anders Markvardsen deleted

Have somewhat blindfolded tried to test this tickets. Tried renaming the TOPAZ files TOPAZ_1249_neutron_event.dat to TOPAZ_1249_event.nxs from Test\Data\sns_event_prenexus, and then in GUI setting RunNumber=1249. However get the message:

Error in execution of algorithm LoadLogsFromSNSNexus: NXopen(C:\MantidInstall\bin\TOPAZ_1249_event.nxs, 1) failed Error while loading Logs from SNS Nexus. Some sample logs may be missing. Error in execution of algorithm LoadSNSEventNexus: NXopen(C:\MantidInstall\bin\TOPAZ_1249_event.nxs, 1) failed Error in execution of algorithm SNSSingleCrystalReduction: Python error: An error occurred while running LoadSNSEventNexus. See results log for details.

possibly because I am not suppose to do the above...?

Note could not find wiki doc for this algorithm, at least no wiki page at http://www.mantidproject.org/SNSSingleCrystalReduction.

Since no other TOPAZ files appears to be in SVN probably best if the ticket is passed on to someone on the other side of the Atlantic ocean

comment:32 Changed 10 years ago by Andrei Savici

  • Status changed from verify to verifying
  • Tester set to Andrei Savici

comment:33 Changed 10 years ago by Vickie Lynch

(In [9551]) Refs #2023 Look for file first in working directory so it can be tested

comment:34 Changed 10 years ago by Andrei Savici

  • Status changed from verifying to closed

Seems to be working. The output workspace has 0 events, so it should be deleted. Wiki is till missing

comment:35 Changed 10 years ago by Vickie Lynch

  • Status changed from closed to reopened
  • Resolution fixed deleted

comment:36 Changed 10 years ago by Vickie Lynch

(In [9552]) Refs #2023 Deleted workspace after saved in NeXus file as suggested by tester

comment:37 Changed 10 years ago by Vickie Lynch

  • Status changed from reopened to accepted

comment:38 Changed 10 years ago by Vickie Lynch

  • Status changed from accepted to verify
  • Resolution set to fixed

comment:39 Changed 10 years ago by Andrei Savici

  • Status changed from verify to verifying

comment:40 Changed 10 years ago by Andrei Savici

  • Status changed from verifying to closed

comment:41 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 2870

Note: See TracTickets for help on using tickets.