Ticket #4274 (closed: fixed)
Save MDHistoWorkspace to Michal's HDF format
Reported by: | Janik Zikovsky | Owned by: | Janik Zikovsky |
---|---|---|---|
Priority: | critical | Milestone: | Release 2.1 |
Component: | Mantid | Keywords: | |
Cc: | vel@…, pf9@… | Blocked By: | |
Blocking: | Tester: | Vickie Lynch |
Description (last modified by Janik Zikovsky) (diff)
This file contains key portions of the email exchanges with Michal Chodkiewicz (michal.chodkiewicz@gmail.com) of the SINERGIA project, regarding the ZODS hdf5 file format. Michal put together a very simple example of the format, which is included. ZODS_hdf5_format.txt is an ASCII form of the example; ZODS_file_example.h5 is the actual HDF5 file. These files should be very helpful for figuring out the details of the format. I've also attached the relevant portion of IsawEV's java code for writing the format from IsawEV: WriteSlicesToHDF_5.java. Several more extensive example files, still using 1D arrays, are on the ISAW ftp site in: http://ftp.sns.gov/ISAW/SAMPLE_SLICE_DATA_HDF_5/. One change should be made: the examples have the arrays written as long 1D arrays, but Michal and I agreed to write individual slices as 2D arrays. That change allows hdfview to show the slices as images, though the hdfview color scales must be tweaked to get a useful display. The 2D version is included in the attached java code. If a "real" example file using 2D images would be helpful, let me know. It will just take a few minutes for me to make one from TOPAZ data. THE BASIC IDEA: Michal wrote: "In general it contains collection of grids with intensities and each grid is described by specifying origin, size of grid (in each direction) and grid base vectors (a1,a2,a3) so a point at node (i,j,k) of grid has coordinates r= origin+i*a1+j*a2+k*a3." SOME QUESTIONS I HAD AND Michal's ANSWERS (i.e. a FAQ list): Dennis: 1. Will this always be done in terms of h,k,l? That is, will a direction vector such as direction 1 vector (0.04,0,0) always represent a step of 0.04 in the (1,0,0) direction in reciprocal space. Will it ever be necessary to deal with slices just in terms of Q-vectors, not specified relative to an indexed lattice? Michal: There are two possible systems of coordinates which ZODS can use (so only one of them needs to be implemented for output): (1) direction 1 vector (0.04,0,0) represents a step of 0.04 in the (1,0,0) direction (a*) in reciprocal space (2) direction 1 vector (0.04,0,0) represents a step of 0.04 Angstrom^(-1) in X direction of Cartesian coordinate system (with X colinear with a and Z with c*) Dennis: 2. What is the role of the GROUP "Data" and GROUP "CoordinateSystem" at the start of the ASCII example? Michal: CoordinateSystem has attribute isLocal - if the attribute is true (=1) then the "origin" and "direction_i".. are defined using coordinate system (1) if isLocal equals 0 then the vectors defining grid are defined with coordinate system 2, in this case CoordinateSystem has additional attribute UnitCell which is a vector with 6 components (a,b,c,alpha,beta,gamma) a,b,c in angstroms and angles in degrees. It is intended to group all arrays with data, so there will be: GROUP "/" { GROUP "CoordinateSystem" {..} GROUP "Data" { GROUP "Data_0"{...} GROUP "Data_1"{...} } Dennis: 3. The grid of points you refer to specifies the centers of histogram bins in 3D, not the corners. Is that correct? Michal: yes Dennis: 4. I assume that the numbers in the line DATASPACE SIMPLE { (16)/(16) } are the current_dims and max_dims. (Apparently the rank is not explicitly stated in the ASCII format.) Do you intend to always store the actual slice data as a one-dimensional array? That is, if the "size" was 10,20,30, you would still expect to get one 1-dimensional array with 6000 entries? Michal: yes, always 1-dimensional // NOTE: This has been changed to allow 2D Dennis: 5. If the data array is 1-dimensional, we need to be very clear on the order in which it is stored. In the above 10x20x30 example, do the first 10 entries represent entries Data[0][0][0] through Data[9][0][0]; the next 10 entries Data[0][1][0] through Data[9][1][0], etc, until finally, the last 10 entries represent Data[0][19][29] through Data[9][19][29]. Is that the convention you want? NOTE: This is the reverse of the "C" convention (row-major order). Michal: It is the reverse i.e. the first component is slowest varying, so the first 10 elements are Data[0][0][0] through Data[0][0][9] Dennis: 6. Direction 1 gives the direction from the sample point for Data[0][0][0] to the sample point for Data[1][0][0]. Direction 2 gives the direction from the sample point for Data[0][0][0] to the sample point for Data[0][1][0]. Direction 3 gives the direction from the sample point for Data[0][0][0] to the sample point for Data[0][0][1]. Is that correct? Michael: yes
Attachments
Change History
Changed 9 years ago by Janik Zikovsky
- Attachment ZODS_file_example.h5 added
comment:3 Changed 9 years ago by Janik Zikovsky
ZODS HDF5 format.txt:
HDF5 "call_1_ind_0.h5" { GROUP "/" { GROUP "CoordinateSystem" { ATTRIBUTE "isLocal" { DATATYPE H5T_STD_U32LE DATASPACE SCALAR DATA { (0): 0 } } } GROUP "Data" { GROUP "Data_0" { ATTRIBUTE "direction_1" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 3 ) / ( 3 ) } DATA { (0): 0.04, -2.44929e-018, -2.44929e-018 } } ATTRIBUTE "direction_2" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 3 ) / ( 3 ) } DATA { (0): 0, 0.04, -2.44929e-018 } } ATTRIBUTE "direction_3" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 3 ) / ( 3 ) } DATA { (0): 0, 0, 0.004 } } ATTRIBUTE "origin" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 3 ) / ( 3 ) } DATA { (0): -1.6, -1.6, 1.95943e-016 } } ATTRIBUTE "size" { DATATYPE H5T_STD_U32LE DATASPACE SIMPLE { ( 3 ) / ( 3 ) } DATA { (0): 9, 9, 1 } } DATASET "Data" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 81 ) / ( 81 ) } DATA { (0): 2753.1, 2471.5, 1687.15, 795.321, 184.415, 3.70492e-007, (6): 100.856, 221.549, 197.934, 2471.5, 2109.9, 1364.16, 604.413, (13): 129.954, 2.61231e-007, 55.5666, 94.8741, 46.9227, 1687.19, (19): 1364.18, 828.99, 340.561, 66.2694, 4.16443e-007, 19.0313, (25): 17.9868, 2.67682e-028, 795.321, 604.413, 340.568, 126.628, (31): 21.2919, 7.52455e-008, 2.63038, 6.77865e-029, 18.7726, (36): 184.433, 129.966, 66.2743, 21.2931, 2.78238, 1.96945e-008, (42): 8.88689e-030, 2.74621, 20.7392, 3.70492e-007, 2.61231e-007, (47): 4.16443e-007, 7.52455e-008, 1.96945e-008, 7.98771e-086, (51): 2.05633e-008, 8.20307e-008, 1.81061e-007, 100.856, 55.5666, (56): 19.0313, 2.63038, 8.88689e-030, 2.05633e-008, 3.03419, (61): 24.2472, 78.8271, 221.549, 94.8741, 17.9868, 6.77865e-029, (67): 2.74621, 8.20307e-008, 24.2472, 150.653, 423.309, 197.934, (73): 46.9227, 2.67682e-028, 18.7726, 20.7392, 1.81061e-007, (78): 78.8271, 423.309, 1077.31 } } } GROUP "Data_1" { ATTRIBUTE "direction_1" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 3 ) / ( 3 ) } DATA { (0): 0.6, -3.67394e-017, -3.67394e-017 } } ATTRIBUTE "direction_2" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 3 ) / ( 3 ) } DATA { (0): 0, 0.6, -3.67394e-017 } } ATTRIBUTE "direction_3" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 3 ) / ( 3 ) } DATA { (0): 0, 0, 0.044 } } ATTRIBUTE "origin" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 3 ) / ( 3 ) } DATA { (0): -1.6, -1.6, 0.4 } } ATTRIBUTE "size" { DATATYPE H5T_STD_U32LE DATASPACE SIMPLE { ( 3 ) / ( 3 ) } DATA { (0): 6, 6, 1 } } DATASET "Data" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 36 ) / ( 36 ) } DATA { (0): 2695.36, 4.27936e-007, 4.02729e-028, 2.23398e-027, 3763.04, (5): 9.40132e-027, 3.5661e-028, 6.22302e-061, 9.92598e-028, (9): 2.62496e-028, 8.012e-028, 2.36776e-086, 4.02729e-028, (13): 9.92598e-028, 21316.6, 4.84147e-026, 3.67199e-026, (17): 9.82572e-026, 1.21473e-026, 2.62496e-028, 7.28752e-026, (21): 5.16881e-085, 3.95531e-026, 1.4602e-028, 3763.04, (25): 8.36716e-007, 4.56749e-027, 5.44158e-028, 8492.05, (29): 1.42407e-026, 3.61517e-007, 1.23663e-085, 6.60426e-007, (33): 1.4602e-028, 8.29925e-026, 2.23827e-084 } } } } } }
comment:4 Changed 9 years ago by Nick Draper
- Milestone changed from Iteration 32 to Iteration 33
Moved to iteration 33 at iteration 32 code freeze
comment:5 Changed 9 years ago by Janik Zikovsky
Note the loader is available on the topaz computer
/SNS/TOPAZ/shared/ZODS/bin/analyzer
and examples here:
/SNS/TOPAZ/shared/ZODS/zods_examples/CuAu
See Tara or Vickie for more help
comment:6 Changed 9 years ago by Janik Zikovsky
- Status changed from new to accepted
Will call algorithm SaveZODS :)
comment:7 Changed 9 years ago by Janik Zikovsky
Refs #4274: stub of a SaveZODS algorithm
Changeset: 8580f3a3a3424c390c299a6732f7b69ff11a00f0
comment:8 Changed 9 years ago by Janik Zikovsky
Refs #4274 putAttr() for vectors. Does not work
Changeset: ecc6027b64ec53d63da1cc1da16d5cbc6012f74b
comment:9 Changed 9 years ago by Janik Zikovsky
Revert "Refs #4274 putAttr() for vectors. Does not work"
This reverts commit ecc6027b64ec53d63da1cc1da16d5cbc6012f74b.
Changeset: 8259dadce142b680145a53fce2fcbe1f8bc4bf94
comment:10 Changed 9 years ago by Janik Zikovsky
Refs #4274 SaveZODS fleshed out
Changeset: a7f931ef92887a10a6d964ceaf40ab74cc1175d9
comment:11 Changed 9 years ago by Janik Zikovsky
Refs #4274 fix dimensions
Changeset: e16a48de68222c1de959ee41b18430b77bd29a4f
comment:12 Changed 9 years ago by Janik Zikovsky
Refs #4274: ZODS file format updatE
Changeset: 5059e5b1cfac7dcfd12151a659d9299c1488646d
comment:13 Changed 9 years ago by Janik Zikovsky
Refs #4274: SaveZODS wiki and fixes
Changeset: a40acf9cddefd04941b65eedf91416a9b7500164
comment:14 Changed 9 years ago by Janik Zikovsky
Refs #4274: ZODS file format updatE
Changeset: 5059e5b1cfac7dcfd12151a659d9299c1488646d
comment:15 Changed 9 years ago by Janik Zikovsky
Refs #4274: SaveZODS wiki and fixes
Changeset: a40acf9cddefd04941b65eedf91416a9b7500164
comment:16 Changed 9 years ago by Janik Zikovsky
Hi Janik,
It works.
There are two more fields possible not present in the example you have sent but they are optional (so it is ok). I have listed them below to make sure that you have that information:
(1) If information about standard deviation is available it should be added as an array called 'sigma' (e.g. located at /Data/Data_0/sigma) with sigmas = -1 for missing or invalid data points (in the same format as corresponding data).
(2) If information about unit cell is available it should be written to /CoordinateSystem/unit_cell as 6D vector (a,b,c,alpha,beta,gamma)
thank you,
Michal
comment:17 Changed 9 years ago by Janik Zikovsky
Refs #4274 final fixes to SaveZODS
Changeset: 40b4db44cd3523c9fa1df19aefca6ff9090c2bff
comment:18 Changed 9 years ago by Janik Zikovsky
- Status changed from accepted to verify
- Resolution set to fixed
comment:19 Changed 8 years ago by Vickie Lynch
Refs #4274 ZODS reads UnitCell not unit_cell
Changeset: fad0938988ef4c5df72f52f7c895e63df099cd8b
comment:20 Changed 8 years ago by Vickie Lynch
Refs #4274 ZODS reads UnitCell not unit_cell
Changeset: fad0938988ef4c5df72f52f7c895e63df099cd8b
comment:21 Changed 8 years ago by Vickie Lynch
- Status changed from verify to verifying
- Tester set to Vickie Lynch
comment:22 Changed 8 years ago by Vickie Lynch
- Status changed from verifying to closed
ZODS code is able to read this input
comment:23 Changed 5 years ago by Stuart Campbell
This ticket has been transferred to github issue 5121
ZODS hdf5 sample file