Ticket #4274 (closed: fixed)

Opened 9 years ago

Last modified 5 years ago

Save MDHistoWorkspace to Michal's HDF format

Reported by: Janik Zikovsky Owned by: Janik Zikovsky
Priority: critical Milestone: Release 2.1
Component: Mantid Keywords:
Cc: vel@…, pf9@… Blocked By:
Blocking: Tester: Vickie Lynch

Description (last modified by Janik Zikovsky) (diff)

This file contains key portions of the email exchanges with Michal Chodkiewicz
(michal.chodkiewicz@gmail.com) of the SINERGIA project, regarding the ZODS hdf5 
file format. 

  Michal put together a very simple example of the format, which is
included.  ZODS_hdf5_format.txt is an ASCII form of the example; 
ZODS_file_example.h5 is the actual HDF5 file.  These files should be very 
helpful for figuring out the details of the format.  I've also attached the
relevant portion of IsawEV's java code for writing the format from IsawEV:
WriteSlicesToHDF_5.java.  Several more extensive example files, still using 
1D arrays, are on the ISAW ftp site in: 
http://ftp.sns.gov/ISAW/SAMPLE_SLICE_DATA_HDF_5/.

  One change should be made: the examples have the arrays written as long 
1D arrays, but Michal and I agreed to write individual slices as 2D arrays. 
That change allows hdfview to show the slices as images, though the hdfview 
color scales must be tweaked to get a useful display.  The 2D version is 
included in the attached java code.  If a "real" example file using 2D 
images would be helpful, let me know. It will just take a few minutes for 
me to make one from TOPAZ data.

THE BASIC IDEA:

  Michal wrote:
"In general it contains collection of grids with intensities and each grid 
is described by specifying origin, size of grid (in each direction) and 
grid base vectors (a1,a2,a3) so a point at node (i,j,k) of grid has 
coordinates r= origin+i*a1+j*a2+k*a3."

SOME QUESTIONS I HAD AND Michal's ANSWERS (i.e. a FAQ list):

Dennis: 1. Will this always be done in terms of h,k,l?  That is, will a 
direction vector such as direction 1 vector (0.04,0,0) always represent a 
step of 0.04 in the (1,0,0) direction in reciprocal space.  Will it ever be 
necessary to deal with slices just in terms of Q-vectors, not specified 
relative to an indexed lattice?

Michal: There are two possible systems of coordinates which ZODS can use 
(so only one of them needs to be implemented for output):

(1) direction 1 vector (0.04,0,0) represents a step of 0.04 in the (1,0,0)
direction (a*) in reciprocal space

(2) direction 1 vector (0.04,0,0) represents a step of 0.04 Angstrom^(-1) in X
direction of Cartesian coordinate system (with X colinear with a and Z with
c*)

Dennis: 2. What is the role of the GROUP "Data" and GROUP "CoordinateSystem" at
the start of the ASCII example?
 
Michal: CoordinateSystem has attribute isLocal - if the attribute is true (=1) 
then the "origin" and "direction_i".. are defined using coordinate system (1)
if isLocal equals 0 then the vectors defining grid are defined with coordinate
system 2, in this case CoordinateSystem has additional attribute UnitCell
which is a vector with 6 components (a,b,c,alpha,beta,gamma) a,b,c in
angstroms and angles in degrees.  It is intended to group all arrays
with data, so there will be:
GROUP "/" {
GROUP "CoordinateSystem" {..}
GROUP "Data"  {
   GROUP "Data_0"{...}
   GROUP "Data_1"{...}
}

Dennis: 3. The grid of points you refer to specifies the centers of histogram 
bins in 3D, not the corners.  Is that correct?

Michal: yes

Dennis: 4. I assume that the numbers in the line DATASPACE SIMPLE { (16)/(16) }
    are the current_dims and max_dims.  (Apparently the rank is not explicitly
stated in the ASCII format.)  Do you intend to always store the actual slice
data as a one-dimensional array?  That is, if the "size" was 10,20,30, you
would still expect to get one 1-dimensional array with 6000 entries?

Michal:  yes, always 1-dimensional  // NOTE: This has been changed to allow 2D

Dennis: 5. If the data array is 1-dimensional, we need to be very clear on the
order in which it is stored.  In the above 10x20x30 example, do the first 10
entries represent entries Data[0][0][0] through Data[9][0][0]; the next 10
entries Data[0][1][0] through Data[9][1][0], etc, until finally, the last 10
entries represent Data[0][19][29] through Data[9][19][29].  Is that the
convention you want?  NOTE: This is the reverse of the "C" convention
(row-major order).

Michal: It is the reverse i.e. the first component is slowest varying, so the 
first 10 elements are Data[0][0][0] through Data[0][0][9]

Dennis: 6. Direction 1 gives the direction from the sample point for 
           Data[0][0][0] to the sample point for Data[1][0][0].
           Direction 2 gives the direction from the sample point for 
           Data[0][0][0] to the sample point for Data[0][1][0].
           Direction 3 gives the direction from the sample point for 
           Data[0][0][0] to the sample point for Data[0][0][1].
     Is that correct?

Michael: yes

Attachments

ZODS_file_example.h5 (8.3 KB) - added by Janik Zikovsky 9 years ago.
ZODS hdf5 sample file
WriteSlicesToHDF_5.java (9.6 KB) - added by Janik Zikovsky 9 years ago.

Change History

Changed 9 years ago by Janik Zikovsky

ZODS hdf5 sample file

Changed 9 years ago by Janik Zikovsky

comment:1 Changed 9 years ago by Janik Zikovsky

  • Description modified (diff)

comment:2 Changed 9 years ago by Janik Zikovsky

  • Description modified (diff)

comment:3 Changed 9 years ago by Janik Zikovsky

ZODS HDF5 format.txt:

HDF5 "call_1_ind_0.h5" {
GROUP "/" {
   GROUP "CoordinateSystem" {
      ATTRIBUTE "isLocal" {
         DATATYPE  H5T_STD_U32LE
         DATASPACE  SCALAR
         DATA {
         (0): 0
         }
      }
   }
   GROUP "Data" {
      GROUP "Data_0" {
         ATTRIBUTE "direction_1" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 0.04, -2.44929e-018, -2.44929e-018
            }
         }
         ATTRIBUTE "direction_2" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 0, 0.04, -2.44929e-018
            }
         }
         ATTRIBUTE "direction_3" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 0, 0, 0.004
            }
         }
         ATTRIBUTE "origin" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): -1.6, -1.6, 1.95943e-016
            }
         }
         ATTRIBUTE "size" {
            DATATYPE  H5T_STD_U32LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 9, 9, 1
            }
         }
         DATASET "Data" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 81 ) / ( 81 ) }
            DATA {
            (0): 2753.1, 2471.5, 1687.15, 795.321, 184.415, 3.70492e-007,
            (6): 100.856, 221.549, 197.934, 2471.5, 2109.9, 1364.16, 604.413,
            (13): 129.954, 2.61231e-007, 55.5666, 94.8741, 46.9227, 1687.19,
            (19): 1364.18, 828.99, 340.561, 66.2694, 4.16443e-007, 19.0313,
            (25): 17.9868, 2.67682e-028, 795.321, 604.413, 340.568, 126.628,
            (31): 21.2919, 7.52455e-008, 2.63038, 6.77865e-029, 18.7726,
            (36): 184.433, 129.966, 66.2743, 21.2931, 2.78238, 1.96945e-008,
            (42): 8.88689e-030, 2.74621, 20.7392, 3.70492e-007, 2.61231e-007,
            (47): 4.16443e-007, 7.52455e-008, 1.96945e-008, 7.98771e-086,
            (51): 2.05633e-008, 8.20307e-008, 1.81061e-007, 100.856, 55.5666,
            (56): 19.0313, 2.63038, 8.88689e-030, 2.05633e-008, 3.03419,
            (61): 24.2472, 78.8271, 221.549, 94.8741, 17.9868, 6.77865e-029,
            (67): 2.74621, 8.20307e-008, 24.2472, 150.653, 423.309, 197.934,
            (73): 46.9227, 2.67682e-028, 18.7726, 20.7392, 1.81061e-007,
            (78): 78.8271, 423.309, 1077.31
            }
         }
      }
      GROUP "Data_1" {
         ATTRIBUTE "direction_1" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 0.6, -3.67394e-017, -3.67394e-017
            }
         }
         ATTRIBUTE "direction_2" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 0, 0.6, -3.67394e-017
            }
         }
         ATTRIBUTE "direction_3" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 0, 0, 0.044
            }
         }
         ATTRIBUTE "origin" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): -1.6, -1.6, 0.4
            }
         }
         ATTRIBUTE "size" {
            DATATYPE  H5T_STD_U32LE
            DATASPACE  SIMPLE { ( 3 ) / ( 3 ) }
            DATA {
            (0): 6, 6, 1
            }
         }
         DATASET "Data" {
            DATATYPE  H5T_IEEE_F64LE
            DATASPACE  SIMPLE { ( 36 ) / ( 36 ) }
            DATA {
            (0): 2695.36, 4.27936e-007, 4.02729e-028, 2.23398e-027, 3763.04,
            (5): 9.40132e-027, 3.5661e-028, 6.22302e-061, 9.92598e-028,
            (9): 2.62496e-028, 8.012e-028, 2.36776e-086, 4.02729e-028,
            (13): 9.92598e-028, 21316.6, 4.84147e-026, 3.67199e-026,
            (17): 9.82572e-026, 1.21473e-026, 2.62496e-028, 7.28752e-026,
            (21): 5.16881e-085, 3.95531e-026, 1.4602e-028, 3763.04,
            (25): 8.36716e-007, 4.56749e-027, 5.44158e-028, 8492.05,
            (29): 1.42407e-026, 3.61517e-007, 1.23663e-085, 6.60426e-007,
            (33): 1.4602e-028, 8.29925e-026, 2.23827e-084
            }
         }
      }
   }
}
}

comment:4 Changed 9 years ago by Nick Draper

  • Milestone changed from Iteration 32 to Iteration 33

Moved to iteration 33 at iteration 32 code freeze

comment:5 Changed 9 years ago by Janik Zikovsky

Note the loader is available on the topaz computer

/SNS/TOPAZ/shared/ZODS/bin/analyzer

and examples here:

/SNS/TOPAZ/shared/ZODS/zods_examples/CuAu

See Tara or Vickie for more help

comment:6 Changed 9 years ago by Janik Zikovsky

  • Status changed from new to accepted

Will call algorithm SaveZODS :)

comment:7 Changed 9 years ago by Janik Zikovsky

Refs #4274: stub of a SaveZODS algorithm

Changeset: 8580f3a3a3424c390c299a6732f7b69ff11a00f0

comment:8 Changed 9 years ago by Janik Zikovsky

Refs #4274 putAttr() for vectors. Does not work

Changeset: ecc6027b64ec53d63da1cc1da16d5cbc6012f74b

comment:9 Changed 9 years ago by Janik Zikovsky

Revert "Refs #4274 putAttr() for vectors. Does not work"

This reverts commit ecc6027b64ec53d63da1cc1da16d5cbc6012f74b.

Changeset: 8259dadce142b680145a53fce2fcbe1f8bc4bf94

comment:10 Changed 9 years ago by Janik Zikovsky

Refs #4274 SaveZODS fleshed out

Changeset: a7f931ef92887a10a6d964ceaf40ab74cc1175d9

comment:11 Changed 9 years ago by Janik Zikovsky

Refs #4274 fix dimensions

Changeset: e16a48de68222c1de959ee41b18430b77bd29a4f

comment:12 Changed 9 years ago by Janik Zikovsky

Refs #4274: ZODS file format updatE

Changeset: 5059e5b1cfac7dcfd12151a659d9299c1488646d

comment:13 Changed 9 years ago by Janik Zikovsky

Refs #4274: SaveZODS wiki and fixes

Changeset: a40acf9cddefd04941b65eedf91416a9b7500164

comment:14 Changed 9 years ago by Janik Zikovsky

Refs #4274: ZODS file format updatE

Changeset: 5059e5b1cfac7dcfd12151a659d9299c1488646d

comment:15 Changed 9 years ago by Janik Zikovsky

Refs #4274: SaveZODS wiki and fixes

Changeset: a40acf9cddefd04941b65eedf91416a9b7500164

comment:16 Changed 9 years ago by Janik Zikovsky

Hi Janik,

It works.

There are two more fields possible not present in the example you have sent but they are optional (so it is ok). I have listed them below to make sure that you have that information:

(1) If information about standard deviation is available it should be added as an array called 'sigma' (e.g. located at /Data/Data_0/sigma) with sigmas = -1 for missing or invalid data points (in the same format as corresponding data).

(2) If information about unit cell is available it should be written to /CoordinateSystem/unit_cell as 6D vector (a,b,c,alpha,beta,gamma)

thank you,

Michal

comment:17 Changed 9 years ago by Janik Zikovsky

Refs #4274 final fixes to SaveZODS

Changeset: 40b4db44cd3523c9fa1df19aefca6ff9090c2bff

comment:18 Changed 9 years ago by Janik Zikovsky

  • Status changed from accepted to verify
  • Resolution set to fixed

comment:19 Changed 8 years ago by Vickie Lynch

Refs #4274 ZODS reads UnitCell not unit_cell

Changeset: fad0938988ef4c5df72f52f7c895e63df099cd8b

comment:20 Changed 8 years ago by Vickie Lynch

Refs #4274 ZODS reads UnitCell not unit_cell

Changeset: fad0938988ef4c5df72f52f7c895e63df099cd8b

comment:21 Changed 8 years ago by Vickie Lynch

  • Status changed from verify to verifying
  • Tester set to Vickie Lynch

comment:22 Changed 8 years ago by Vickie Lynch

  • Status changed from verifying to closed

ZODS code is able to read this input

comment:23 Changed 5 years ago by Stuart Campbell

This ticket has been transferred to github issue 5121

Note: See TracTickets for help on using tickets.