Save MDHistoWorkspace to Michal's HDF format

[Janik Zikovsky]

This file contains key portions of the email exchanges with Michal Chodkiewicz
(michal.chodkiewicz@gmail.com) of the SINERGIA project, regarding the ZODS hdf5 
file format. 

  Michal put together a very simple example of the format, which is
included.  ZODS_hdf5_format.txt is an ASCII form of the example; 
ZODS_file_example.h5 is the actual HDF5 file.  These files should be very 
helpful for figuring out the details of the format.  I've also attached the
relevant portion of IsawEV's java code for writing the format from IsawEV:
WriteSlicesToHDF_5.java.  Several more extensive example files, still using 
1D arrays, are on the ISAW ftp site in: 
http://ftp.sns.gov/ISAW/SAMPLE_SLICE_DATA_HDF_5/.

  One change should be made: the examples have the arrays written as long 
1D arrays, but Michal and I agreed to write individual slices as 2D arrays. 
That change allows hdfview to show the slices as images, though the hdfview 
color scales must be tweaked to get a useful display.  The 2D version is 
included in the attached java code.  If a "real" example file using 2D 
images would be helpful, let me know. It will just take a few minutes for 
me to make one from TOPAZ data.

THE BASIC IDEA:

  Michal wrote:
"In general it contains collection of grids with intensities and each grid 
is described by specifying origin, size of grid (in each direction) and 
grid base vectors (a1,a2,a3) so a point at node (i,j,k) of grid has 
coordinates r= origin+i*a1+j*a2+k*a3."

SOME QUESTIONS I HAD AND Michal's ANSWERS (i.e. a FAQ list):

Dennis: 1. Will this always be done in terms of h,k,l?  That is, will a 
direction vector such as direction 1 vector (0.04,0,0) always represent a 
step of 0.04 in the (1,0,0) direction in reciprocal space.  Will it ever be 
necessary to deal with slices just in terms of Q-vectors, not specified 
relative to an indexed lattice?

Michal: There are two possible systems of coordinates which ZODS can use 
(so only one of them needs to be implemented for output):

(1) direction 1 vector (0.04,0,0) represents a step of 0.04 in the (1,0,0)
direction (a*) in reciprocal space

(2) direction 1 vector (0.04,0,0) represents a step of 0.04 Angstrom^(-1) in X
direction of Cartesian coordinate system (with X colinear with a and Z with
c*)

Dennis: 2. What is the role of the GROUP "Data" and GROUP "CoordinateSystem" at
the start of the ASCII example?
 
Michal: CoordinateSystem has attribute isLocal - if the attribute is true (=1) 
then the "origin" and "direction_i".. are defined using coordinate system (1)
if isLocal equals 0 then the vectors defining grid are defined with coordinate
system 2, in this case CoordinateSystem has additional attribute UnitCell
which is a vector with 6 components (a,b,c,alpha,beta,gamma) a,b,c in
angstroms and angles in degrees.  It is intended to group all arrays
with data, so there will be:
GROUP "/" {
GROUP "CoordinateSystem" {..}
GROUP "Data"  {
   GROUP "Data_0"{...}
   GROUP "Data_1"{...}
}

Dennis: 3. The grid of points you refer to specifies the centers of histogram 
bins in 3D, not the corners.  Is that correct?

Michal: yes

Dennis: 4. I assume that the numbers in the line DATASPACE SIMPLE { (16)/(16) }
    are the current_dims and max_dims.  (Apparently the rank is not explicitly
stated in the ASCII format.)  Do you intend to always store the actual slice
data as a one-dimensional array?  That is, if the "size" was 10,20,30, you
would still expect to get one 1-dimensional array with 6000 entries?

Michal:  yes, always 1-dimensional  // NOTE: This has been changed to allow 2D

Dennis: 5. If the data array is 1-dimensional, we need to be very clear on the
order in which it is stored.  In the above 10x20x30 example, do the first 10
entries represent entries Data[0][0][0] through Data[9][0][0]; the next 10
entries Data[0][1][0] through Data[9][1][0], etc, until finally, the last 10
entries represent Data[0][19][29] through Data[9][19][29].  Is that the
convention you want?  NOTE: This is the reverse of the "C" convention
(row-major order).

Michal: It is the reverse i.e. the first component is slowest varying, so the 
first 10 elements are Data[0][0][0] through Data[0][0][9]

Dennis: 6. Direction 1 gives the direction from the sample point for 
           Data[0][0][0] to the sample point for Data[1][0][0].
           Direction 2 gives the direction from the sample point for 
           Data[0][0][0] to the sample point for Data[0][1][0].
           Direction 3 gives the direction from the sample point for 
           Data[0][0][0] to the sample point for Data[0][0][1].
     Is that correct?

Michael: yes

[Janik Zikovsky]

ZODS hdf5 sample file

[Janik Zikovsky]

ZODS HDF5 format.txt:

HDF5 "call_1_ind_0.h5" {
GROUP "/" {
   GROUP "CoordinateSystem" {
      ATTRIBUTE "isLocal" {
         DATATYPE  H5T_STD_U32LE
         DATASPACE  SCALAR
         DATA {
         (0): 0
         }
      }
   }
   GROUP "Data" {
      GROUP "Data_0" {
         ATTRIBUTE "direction_1" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 0.04, -2.44929e-018, -2.44929e-018
            }
         }
         ATTRIBUTE "direction_2" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 0, 0.04, -2.44929e-018
            }
         }
         ATTRIBUTE "direction_3" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 0, 0, 0.004
            }
         }
         ATTRIBUTE "origin" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): -1.6, -1.6, 1.95943e-016
            }
         }
         ATTRIBUTE "size" {
            DATATYPE  H5T_STD_U32LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 9, 9, 1
            }
         }
         DATASET "Data" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 81 ) / ( 81 ) }
            DATA {
            (0): 2753.1, 2471.5, 1687.15, 795.321, 184.415, 3.70492e-007,
            (6): 100.856, 221.549, 197.934, 2471.5, 2109.9, 1364.16, 604.413,
            (13): 129.954, 2.61231e-007, 55.5666, 94.8741, 46.9227, 1687.19,
            (19): 1364.18, 828.99, 340.561, 66.2694, 4.16443e-007, 19.0313,
            (25): 17.9868, 2.67682e-028, 795.321, 604.413, 340.568, 126.628,
            (31): 21.2919, 7.52455e-008, 2.63038, 6.77865e-029, 18.7726,
            (36): 184.433, 129.966, 66.2743, 21.2931, 2.78238, 1.96945e-008,
            (42): 8.88689e-030, 2.74621, 20.7392, 3.70492e-007, 2.61231e-007,
            (47): 4.16443e-007, 7.52455e-008, 1.96945e-008, 7.98771e-086,
            (51): 2.05633e-008, 8.20307e-008, 1.81061e-007, 100.856, 55.5666,
            (56): 19.0313, 2.63038, 8.88689e-030, 2.05633e-008, 3.03419,
            (61): 24.2472, 78.8271, 221.549, 94.8741, 17.9868, 6.77865e-029,
            (67): 2.74621, 8.20307e-008, 24.2472, 150.653, 423.309, 197.934,
            (73): 46.9227, 2.67682e-028, 18.7726, 20.7392, 1.81061e-007,
            (78): 78.8271, 423.309, 1077.31
            }
         }
      }
      GROUP "Data_1" {
         ATTRIBUTE "direction_1" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 0.6, -3.67394e-017, -3.67394e-017
            }
         }
         ATTRIBUTE "direction_2" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 0, 0.6, -3.67394e-017
            }
         }
         ATTRIBUTE "direction_3" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 0, 0, 0.044
            }
         }
         ATTRIBUTE "origin" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): -1.6, -1.6, 0.4
            }
         }
         ATTRIBUTE "size" {
            DATATYPE  H5T_STD_U32LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 6, 6, 1
            }
         }
         DATASET "Data" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 36 ) / ( 36 ) }
            DATA {
            (0): 2695.36, 4.27936e-007, 4.02729e-028, 2.23398e-027, 3763.04,
            (5): 9.40132e-027, 3.5661e-028, 6.22302e-061, 9.92598e-028,
            (9): 2.62496e-028, 8.012e-028, 2.36776e-086, 4.02729e-028,
            (13): 9.92598e-028, 21316.6, 4.84147e-026, 3.67199e-026,
            (17): 9.82572e-026, 1.21473e-026, 2.62496e-028, 7.28752e-026,
            (21): 5.16881e-085, 3.95531e-026, 1.4602e-028, 3763.04,
            (25): 8.36716e-007, 4.56749e-027, 5.44158e-028, 8492.05,
            (29): 1.42407e-026, 3.61517e-007, 1.23663e-085, 6.60426e-007,
            (33): 1.4602e-028, 8.29925e-026, 2.23827e-084
            }
         }
      }
   }
}
}

[Nick Draper]

Moved to iteration 33 at iteration 32 code freeze

[Janik Zikovsky]

Note the loader is available on the topaz computer

/SNS/TOPAZ/shared/ZODS/bin/analyzer

and examples here:

/SNS/TOPAZ/shared/ZODS/zods_examples/CuAu

See Tara or Vickie for more help

[Janik Zikovsky]

Will call algorithm SaveZODS :)

[Janik Zikovsky]

Refs #4274: stub of a SaveZODS algorithm

Changeset: 8580f3a3a3424c390c299a6732f7b69ff11a00f0

[Janik Zikovsky]

Refs #4274 putAttr() for vectors. Does not work

Changeset: ecc6027b64ec53d63da1cc1da16d5cbc6012f74b

[Janik Zikovsky]

Revert "Refs #4274 putAttr() for vectors. Does not work"

This reverts commit ecc6027b64ec53d63da1cc1da16d5cbc6012f74b.

Changeset: 8259dadce142b680145a53fce2fcbe1f8bc4bf94

[Janik Zikovsky]

Refs #4274 SaveZODS fleshed out

Changeset: a7f931ef92887a10a6d964ceaf40ab74cc1175d9

[Janik Zikovsky]

Refs #4274 fix dimensions

Changeset: e16a48de68222c1de959ee41b18430b77bd29a4f

[Janik Zikovsky]

Refs #4274: ZODS file format updatE

Changeset: 5059e5b1cfac7dcfd12151a659d9299c1488646d

[Janik Zikovsky]

Refs #4274: SaveZODS wiki and fixes

Changeset: a40acf9cddefd04941b65eedf91416a9b7500164

[Janik Zikovsky]

Refs #4274: ZODS file format updatE

Changeset: 5059e5b1cfac7dcfd12151a659d9299c1488646d

[Janik Zikovsky]

Refs #4274: SaveZODS wiki and fixes

Changeset: a40acf9cddefd04941b65eedf91416a9b7500164

[Janik Zikovsky]

Hi Janik,

It works.

There are two more fields possible not present in the example you have sent but they are optional (so it is ok). I have listed them below to make sure that you have that information:

(1) If information about standard deviation is available it should be added as an array called 'sigma' (e.g. located at /Data/Data_0/sigma) with sigmas = -1 for missing or invalid data points (in the same format as corresponding data).

(2) If information about unit cell is available it should be written to /CoordinateSystem/unit_cell as 6D vector (a,b,c,alpha,beta,gamma)

thank you,

Michal

[Janik Zikovsky]

Refs #4274 final fixes to SaveZODS

Changeset: 40b4db44cd3523c9fa1df19aefca6ff9090c2bff

[Vickie Lynch]

Refs #4274 ZODS reads UnitCell not unit_cell

Changeset: fad0938988ef4c5df72f52f7c895e63df099cd8b

[Vickie Lynch]

Refs #4274 ZODS reads UnitCell not unit_cell

Changeset: fad0938988ef4c5df72f52f7c895e63df099cd8b

[Vickie Lynch]

ZODS code is able to read this input

[Stuart Campbell]

This ticket has been transferred to github issue ​5121