Ticket #2023 (closed: fixed)
Create gui for correcting sns Topaz data for ISAW peak integration
Reported by: | Vickie Lynch | Owned by: | Vickie Lynch |
---|---|---|---|
Priority: | major | Milestone: | Iteration 27 |
Component: | Mantid | Keywords: | |
Cc: | Blocked By: | ||
Blocking: | Tester: | Andrei Savici |
Description
Pixel by pixel background subtraction and division by vanadium. Use SaveSNSNexus algorithm to save data for ISAW.
Change History
comment:4 Changed 10 years ago by Nick Draper
- Milestone changed from Iteration 26 to Iteration 27
Bulk move of tickets to iteration 27, if your ticket is essential for Iteration 26 then move it back.
comment:10 Changed 10 years ago by Janik Zikovsky
comment:11 Changed 10 years ago by Janik Zikovsky
--------------------------------------------------------------------------- Tue Jan 25 18:07:47 2011: Script execution started. --------------------------------------------------------------------------- sample loaded. Time = 106.625463963 background loaded. Time = 273.816184044 subtraction. Time = 370.734992027 vanadium loaded. Time = 1095.36750889 vanadium sorted. Time = 1097.07427907 sample/vanadium. Time = 4787.487849 --------------------------------------------------------------------------- Tue Jan 25 19:27:37 2011: Script execution completed successfully. ---------------------------------------------------------------------------
Seems that the division is by far the slowest step.
comment:12 Changed 10 years ago by Janik Zikovsky
comment:13 Changed 10 years ago by Janik Zikovsky
Better if you convert vanadium to Workspace2D and delete the Vanadium event workspace, but still slow.
--------------------------------------------------------------------------- Wed Jan 26 12:07:56 2011: Script execution started. --------------------------------------------------------------------------- vanadium loaded. Time = 172.51343894 bank1,bank2,bank3,bank4,bank5,bank6,bank7,bank8,bank9,bank10,bank11,bank12,bank13,bank14,bank15 vanadium binned and focussed. Time = 370.558913946 sample loaded. Time = 536.524903059 background loaded. Time = 800.194772959 subtraction. Time = 1032.45306206 sample/vanadium. Time = 2651.39089489 --------------------------------------------------------------------------- Wed Jan 26 12:52:08 2011: Script execution completed successfully. ---------------------------------------------------------------------------
comment:14 Changed 10 years ago by Janik Zikovsky
Same as the previous one, but without LD_PRELOAD=tcmalloc.
--------------------------------------------------------------------------- Wed Jan 26 14:36:08 2011: Script execution started. --------------------------------------------------------------------------- vanadium loaded. Time = 186.312375069 bank1,bank2,bank3,bank4,bank5,bank6,bank7,bank8,bank9,bank10,bank11,bank12,bank13,bank14,bank15 vanadium binned and focussed. Time = 309.687546968 sample loaded. Time = 423.059288025 background loaded. Time = 627.371264935 subtraction. Time = 703.581522942 sample/vanadium. Time = 1165.15904808 --------------------------------------------------------------------------- Wed Jan 26 14:55:34 2011: Script execution completed successfully. ---------------------------------------------------------------------------
comment:15 Changed 10 years ago by Janik Zikovsky
Linked to tcmalloc:
--------------------------------------------------------------------------- Wed Jan 26 15:05:57 2011: Script execution started. --------------------------------------------------------------------------- vanadium loaded. Time = 174.915987015 bank1,bank2,bank3,bank4,bank5,bank6,bank7,bank8,bank9,bank10,bank11,bank12,bank13,bank14,bank15 vanadium binned and focussed. Time = 319.589958191 sample loaded. Time = 447.941715002 background loaded. Time = 750.023055077 subtraction. Time = 1083.41578007 approx 15 more minutes for divide.
tcmalloc seems slower! Perhaps the cache per thread is pushing memory use up until it reaches the physical limit?
comment:16 Changed 10 years ago by Janik Zikovsky
With tcmalloc, and using a call to release memory back to the system
MallocExtension::instance()->ReleaseFreeMemory();
, total time was 965 seconds. This might be significantly faster with more of the memory release calls? One more release call dropped the memory from full (14 gb) to 9 gb = the actually used memory.
comment:17 Changed 10 years ago by Janik Zikovsky
Added more calls to release free memory:
--------------------------------------------------------------------------- Wed Jan 26 16:53:59 2011: Script execution started. --------------------------------------------------------------------------- vanadium loaded. Time = 173.489802122 bank1,bank2,bank3,bank4,bank5,bank6,bank7,bank8,bank9,bank10,bank11,bank12,bank13,bank14,bank15 vanadium binned and focussed. Time = 354.407647133 sample loaded. Time = 469.410380125 background loaded. Time = 637.114018917 subtraction. Time = 680.437106133 sorting (I was not sorting before - makes a big speed difference!) sample/vanadium. Time = about 728
Best result yet! Swap use is limited to the minus operation!
comment:18 Changed 10 years ago by Janik Zikovsky
comment:19 Changed 10 years ago by Janik Zikovsky
- TCmalloc
- Calls to release free memory before every algorithm
- Minus operation clears the RHS workspace. <- This looks to have saved 100 seconds.
--------------------------------------------------------------------------- Thu Jan 27 12:19:14 2011: Script execution started. --------------------------------------------------------------------------- vanadium loaded. Time = 168.479260921 vanadium binned and focussed. Time = 280.20638895 sample loaded. Time = 398.196058989 background loaded. Time = 573.853898048 subtraction. Time = 584.281800032 sample/vanadium. Time = 637.521209002 --------------------------------------------------------------------------- Thu Jan 27 12:29:51 2011: Script execution completed successfully. ---------------------------------------------------------------------------
comment:20 Changed 10 years ago by Janik Zikovsky
Same, but with the use of the stock TCMalloc available by ubuntu synaptic/apt-get .
--------------------------------------------------------------------------- Thu Jan 27 13:29:44 2011: Script execution started. --------------------------------------------------------------------------- vanadium loaded. Time = 173.589309931 vanadium binned and focussed. Time = 288.00959301 sample loaded. Time = 395.241351128 background loaded. Time = 565.24237895 subtraction. Time = 576.450856924 sample/vanadium. Time = 625.486242056 --------------------------------------------------------------------------- Thu Jan 27 13:40:09 2011: Script execution completed successfully. ---------------------------------------------------------------------------
About the same as the custom-built one, no segfaults or errors.
comment:21 Changed 10 years ago by Janik Zikovsky
Without TCmalloc:
--------------------------------------------------------------------------- Thu Jan 27 14:16:55 2011: Script execution started. --------------------------------------------------------------------------- vanadium loaded. Time = 202.329317808 vanadium binned and focussed. Time = 332.317056894 sample loaded. Time = 446.396880865 background loaded. Time = 825.023164988 subtraction. Time = 997.560561895 (gave up - physical memory is full, lots of swapping)
comment:22 Changed 10 years ago by Vickie Lynch
comment:23 Changed 10 years ago by Vickie Lynch
comment:24 Changed 10 years ago by Vickie Lynch
comment:25 Changed 10 years ago by Vickie Lynch
comment:26 Changed 10 years ago by Vickie Lynch
comment:27 Changed 10 years ago by Vickie Lynch
- Status changed from accepted to verify
- Resolution set to fixed
Results from ISAW look OK and it runs in 12 minutes for RunNumber=1817, BackgroundNumber=1748, VanadiumNumber=1789, and defaults for other input.
comment:28 Changed 10 years ago by Vickie Lynch
comment:29 Changed 10 years ago by Vickie Lynch
comment:30 Changed 10 years ago by Anders Markvardsen
- Status changed from verify to verifying
- Tester set to Anders Markvardsen
comment:31 Changed 10 years ago by Anders Markvardsen
- Status changed from verifying to verify
- Tester Anders Markvardsen deleted
Have somewhat blindfolded tried to test this tickets. Tried renaming the TOPAZ files TOPAZ_1249_neutron_event.dat to TOPAZ_1249_event.nxs from Test\Data\sns_event_prenexus, and then in GUI setting RunNumber=1249. However get the message:
Error in execution of algorithm LoadLogsFromSNSNexus: NXopen(C:\MantidInstall\bin\TOPAZ_1249_event.nxs, 1) failed Error while loading Logs from SNS Nexus. Some sample logs may be missing. Error in execution of algorithm LoadSNSEventNexus: NXopen(C:\MantidInstall\bin\TOPAZ_1249_event.nxs, 1) failed Error in execution of algorithm SNSSingleCrystalReduction: Python error: An error occurred while running LoadSNSEventNexus. See results log for details.
possibly because I am not suppose to do the above...?
Note could not find wiki doc for this algorithm, at least no wiki page at http://www.mantidproject.org/SNSSingleCrystalReduction.
Since no other TOPAZ files appears to be in SVN probably best if the ticket is passed on to someone on the other side of the Atlantic ocean
comment:32 Changed 10 years ago by Andrei Savici
- Status changed from verify to verifying
- Tester set to Andrei Savici
comment:33 Changed 10 years ago by Vickie Lynch
comment:34 Changed 10 years ago by Andrei Savici
- Status changed from verifying to closed
Seems to be working. The output workspace has 0 events, so it should be deleted. Wiki is till missing
comment:35 Changed 10 years ago by Vickie Lynch
- Status changed from closed to reopened
- Resolution fixed deleted
comment:36 Changed 10 years ago by Vickie Lynch
comment:38 Changed 10 years ago by Vickie Lynch
- Status changed from accepted to verify
- Resolution set to fixed
comment:41 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 2870
(In [7943]) Refs #2023 Saving corrected TOPAZ data for ISAW