Create gui for correcting sns Topaz data for ISAW peak integration

[Vickie Lynch]

Pixel by pixel background subtraction and division by vanadium. Use SaveSNSNexus algorithm to save data for ISAW.

[Vickie Lynch]

(In [7943]) Refs #2023 Saving corrected TOPAZ data for ISAW

[Vickie Lynch]

(In [7947]) Refs#2023 Defaults for input changed for SNSSingleCrystalReduction

[Vickie Lynch]

(In [8019]) Refs #2023 Reduced memory required for SNSSingleCrystalReduction

[Nick Draper]

Bulk move of tickets to iteration 27, if your ticket is essential for Iteration 26 then move it back.

[Vickie Lynch]

(In [8624]) Refs #2023 Vanadium data focussed for each detector before division in SNSSingleCrystalReduction

[Vickie Lynch]

(In [8660]) Refs #2023 Created DivideVanadium algorithm because python code for division was too slow in SNSSingleCrystalReduction

[Vickie Lynch]

(In [8846]) Refs #2023 Testing loading events from only one detector bank for large TOPAZ event data

[Janik Zikovsky]

(In [8913]) Refs #2023: Minor changes to SNSSingleCrystalReduction

[Janik Zikovsky]

(In [8939]) Refs #2023: SCD script minor changes.

[Janik Zikovsky]

---------------------------------------------------------------------------
Tue Jan 25 18:07:47 2011: Script execution started.
---------------------------------------------------------------------------
sample loaded. Time =  106.625463963
background loaded. Time =  273.816184044
subtraction. Time =  370.734992027
vanadium loaded. Time =  1095.36750889
vanadium sorted. Time =  1097.07427907
sample/vanadium. Time =  4787.487849
---------------------------------------------------------------------------
Tue Jan 25 19:27:37 2011: Script execution completed successfully.
---------------------------------------------------------------------------

Seems that the division is by far the slowest step.

[Janik Zikovsky]

(In [8946]) Refs #2277: Divide algorithm has new parameter to perform division by grouped spectra, where workspace indices do not match. e.g. sample / vanadium (where vanadium was averaged per bank). Refs #2023.

[Janik Zikovsky]

Better if you convert vanadium to Workspace2D and delete the Vanadium event workspace, but still slow.

---------------------------------------------------------------------------
Wed Jan 26 12:07:56 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  172.51343894
bank1,bank2,bank3,bank4,bank5,bank6,bank7,bank8,bank9,bank10,bank11,bank12,bank13,bank14,bank15
vanadium binned and focussed. Time =  370.558913946
sample loaded. Time =  536.524903059
background loaded. Time =  800.194772959
subtraction. Time =  1032.45306206
sample/vanadium. Time =  2651.39089489
---------------------------------------------------------------------------
Wed Jan 26 12:52:08 2011: Script execution completed successfully.
---------------------------------------------------------------------------

[Janik Zikovsky]

Same as the previous one, but without LD_PRELOAD=tcmalloc.

---------------------------------------------------------------------------
Wed Jan 26 14:36:08 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  186.312375069
bank1,bank2,bank3,bank4,bank5,bank6,bank7,bank8,bank9,bank10,bank11,bank12,bank13,bank14,bank15
vanadium binned and focussed. Time =  309.687546968
sample loaded. Time =  423.059288025
background loaded. Time =  627.371264935
subtraction. Time =  703.581522942
sample/vanadium. Time =  1165.15904808
---------------------------------------------------------------------------
Wed Jan 26 14:55:34 2011: Script execution completed successfully.
---------------------------------------------------------------------------

[Janik Zikovsky]

Linked to tcmalloc:

---------------------------------------------------------------------------
Wed Jan 26 15:05:57 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  174.915987015
bank1,bank2,bank3,bank4,bank5,bank6,bank7,bank8,bank9,bank10,bank11,bank12,bank13,bank14,bank15
vanadium binned and focussed. Time =  319.589958191
sample loaded. Time =  447.941715002
background loaded. Time =  750.023055077
subtraction. Time =  1083.41578007
approx 15 more minutes for divide.

tcmalloc seems slower! Perhaps the cache per thread is pushing memory use up until it reaches the physical limit?

[Janik Zikovsky]

With tcmalloc, and using a call to release memory back to the system

MallocExtension::instance()->ReleaseFreeMemory();

, total time was 965 seconds. This might be significantly faster with more of the memory release calls? One more release call dropped the memory from full (14 gb) to 9 gb = the actually used memory.

[Janik Zikovsky]

Added more calls to release free memory:

---------------------------------------------------------------------------
Wed Jan 26 16:53:59 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  173.489802122
bank1,bank2,bank3,bank4,bank5,bank6,bank7,bank8,bank9,bank10,bank11,bank12,bank13,bank14,bank15
vanadium binned and focussed. Time =  354.407647133
sample loaded. Time =  469.410380125
background loaded. Time =  637.114018917
subtraction. Time =  680.437106133
sorting (I was not sorting before - makes a big speed difference!)
sample/vanadium. Time =  about 728

Best result yet! Swap use is limited to the minus operation!

[Janik Zikovsky]

(In [8961]) Refs #2023.

[Janik Zikovsky]

• TCmalloc
• Calls to release free memory before every algorithm
• Minus operation clears the RHS workspace. <- This looks to have saved 100 seconds.

---------------------------------------------------------------------------
Thu Jan 27 12:19:14 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  168.479260921
vanadium binned and focussed. Time =  280.20638895
sample loaded. Time =  398.196058989
background loaded. Time =  573.853898048
subtraction. Time =  584.281800032
sample/vanadium. Time =  637.521209002
---------------------------------------------------------------------------
Thu Jan 27 12:29:51 2011: Script execution completed successfully.
---------------------------------------------------------------------------

[Janik Zikovsky]

Same, but with the use of the stock TCMalloc available by ubuntu synaptic/apt-get .

---------------------------------------------------------------------------
Thu Jan 27 13:29:44 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  173.589309931
vanadium binned and focussed. Time =  288.00959301
sample loaded. Time =  395.241351128
background loaded. Time =  565.24237895
subtraction. Time =  576.450856924
sample/vanadium. Time =  625.486242056
---------------------------------------------------------------------------
Thu Jan 27 13:40:09 2011: Script execution completed successfully.
---------------------------------------------------------------------------

About the same as the custom-built one, no segfaults or errors.

[Janik Zikovsky]

Without TCmalloc:

---------------------------------------------------------------------------
Thu Jan 27 14:16:55 2011: Script execution started.
---------------------------------------------------------------------------
vanadium loaded. Time =  202.329317808
vanadium binned and focussed. Time =  332.317056894
sample loaded. Time =  446.396880865
background loaded. Time =  825.023164988
subtraction. Time =  997.560561895
(gave up - physical memory is full, lots of swapping)

[Vickie Lynch]

(In [9134]) Refs #2023 Output from this script produces good peaks in ISAW for TOPAZ_1817

[Vickie Lynch]

(In [9149]) Refs #2023 DivideVanadium algorithm no longer needed

[Vickie Lynch]

(In [9150]) Refs #2023 Removed DivideVanadium from CMakeLists.txt

[Vickie Lynch]

(In [9153]) Refs #2023 Changed argument to CreateCalFileByNames in SNSSingleCrystalReduction to match Janik's changes to algorithm

[Vickie Lynch]

(In [9155]) Refs #2023 Uncommented FindSNSNexus in SNSSingleCrystalReduction

[Vickie Lynch]

Results from ISAW look OK and it runs in 12 minutes for RunNumber=1817, BackgroundNumber=1748, VanadiumNumber=1789, and defaults for other input.

[Vickie Lynch]

(In [9290]) Refs #2023 Comment reload statement to prevent SegFault

[Vickie Lynch]

(In [9395]) Refs #2023 reload needed for FindSNSNexus to work; Exception at end prevents Segfault

[Anders Markvardsen]

Have somewhat blindfolded tried to test this tickets. Tried renaming the TOPAZ files TOPAZ_1249_neutron_event.dat to TOPAZ_1249_event.nxs from Test\Data\sns_event_prenexus, and then in GUI setting RunNumber=1249. However get the message:

Error in execution of algorithm LoadLogsFromSNSNexus: NXopen(C:\MantidInstall\bin\TOPAZ_1249_event.nxs, 1) failed Error while loading Logs from SNS Nexus. Some sample logs may be missing. Error in execution of algorithm LoadSNSEventNexus: NXopen(C:\MantidInstall\bin\TOPAZ_1249_event.nxs, 1) failed Error in execution of algorithm SNSSingleCrystalReduction: Python error: An error occurred while running LoadSNSEventNexus. See results log for details.

possibly because I am not suppose to do the above...?

Note could not find wiki doc for this algorithm, at least no wiki page at http://www.mantidproject.org/SNSSingleCrystalReduction.

Since no other TOPAZ files appears to be in SVN probably best if the ticket is passed on to someone on the other side of the Atlantic ocean

[Vickie Lynch]

(In [9551]) Refs #2023 Look for file first in working directory so it can be tested

[Andrei Savici]

Seems to be working. The output workspace has 0 events, so it should be deleted. Wiki is till missing

[Vickie Lynch]

(In [9552]) Refs #2023 Deleted workspace after saved in NeXus file as suggested by tester

[Stuart Campbell]

This ticket has been transferred to github issue ​2870