Store data in HDF5 with H5Py

Question

How to store NumPy datetime objects in HDF5 using h5py ?

 In [1]: import h5py In [2]: import numpy as np In [3]: f = h5py.File('foo.hdfs', 'w') In [4]: d = f.create_dataset('data', shape=(2, 2), dtype=np.datetime64) TypeError: No conversion path for dtype: dtype('<M8') 

Answer 1

Currently, HDF5 does not provide a time type (H5T_TIME is no longer supported), so there is no obvious mapping for datetime64.

One of the design goals for h5py was to enter the HDF5 core set of features. This allows people to write data to their files and know that they will go both ways and can be restored by people using other applications that support HDF5, such as IDL and Matlab. Earlier we made some minor exceptions; for example, NumPy bools and complex numbers are mapped to HDF5 enumerations and composite types, respectively. But datetime64 seems to be a much more complex animal.

If there is no convincing suggestion that ensures that (1) crawl information and (2) other HDF5 clients can reasonably understand, I think we are not going to implement native datetime64 support.

In HDF5, people usually store their dates / times as string values ​​using some variation of the ISO date format. You can think of it as a workaround.

See also: https://github.com/h5py/h5py/issues/443

Answer 2

H5py does not currently support time type ( FAQ , Issue ).

NumPy datetime64s is 8 bytes long. Therefore, as a workaround, you can view data as '<i8' store int in hdf5 file and view it as np.datetime64 after extraction:

 import numpy as np import h5py arr = np.linspace(0, 10000, 4).astype('<i8').view('<M8[D]').reshape((2,2)) print(arr) # [['1970-01-01' '1979-02-16'] # ['1988-04-02' '1997-05-19']] with h5py.File('/tmp/out.h5', "w") as f: dset = f.create_dataset('data', (2, 2), '<i8') dset[:,:] = arr.view('<i8') with h5py.File('/tmp/out.h5', "r") as f: dset = f.get('data') print(dset.value.view('<M8[D]')) # [['1970-01-01' '1979-02-16'] # ['1988-04-02' '1997-05-19']]