Chunked HDF5 DataSet and slabsize

Question

We evaluate the performance of HDF5 in relation to distributed datasets. Especially we are trying to figure out whether it is possible to read through different adjacent pieces and how does this affect performance? For example. we have a data set with a block size of 10, a data set with 100 values ​​and want to read values ​​from 23 to 48. Will there be a big performance loss?

Many thanks!

Answer 1

I do not know how to specifically answer your question, but I suggest you use a block size of 1024 (or any higher power of two). I do not know the internal components of HDF5, but from my knowledge of file systems, and from the rough test that we did, 1024 was right.