Adding Structure to HDF5 Files - NetCDF Equivalent to "Agreement" for HDF5

Question

NetCDF4 has a Conventions convention for adding structure to NetCDF. I am looking for a similar thing, but specifically for HDF5.

My common goal is to add structure to my HDF5 files in a standard way. I want something like what HDF5 does with images to determine the type using attributes for groups and ~ like datasets:

CLASS: IMAGE IMAGE_VERSION: 1.2 IMAGE_SUBCLASS: IMAGE_TRUECOLOR ... 

But as far as I can tell, the image specification is autonomous. Maybe I should just reuse NetCDF "conventions"?

Update:

I know that NetCDF4 is implemented on top of HDF5. In this case, we have data from turbulence modeling and experiments, rather than geodata. This data is usually limited to <= 4D. We use HDF5 to store this data already, but we do not have developed standards. Pseudo-standard formats are simply sorted organically within the organization.

Answer 1

The best choice for the standard one really depends on what data you want to store. CF conventions are most useful for geo-referenced measurement data, such as satellite data. It would be helpful to know what your data is made of.

Assuming you have data with a binding, I think you have two options:

• Reuse HDF CFs as you expected. There are many people who look at this, a quick google search gave me this .
• HDF-EOS (disclaimer, I have never used it). It stores data in HDF files using a specific structure, but apparently requires the use of an extension library. I did not find the structure specification, only the API . Also, there seems to be no active community outside of NASA.

So I will probably go with option 1: use CF conventions in your HDF files and see if there is a third-party tool like Panoply that can use it.

Answer 2

NetCDF4 files are actually stored in HDF5 format ( http://www.unidata.ucar.edu/publications/factsheets/current/factsheet_netcdf.pdf ), however they use netCDF4 conventions for attributes, sizes, etc. Files are self-describing, which is a big plus. HDF5 without netCDF4 allows you much more freedom in defining your data. Is there a specific reason why you would like to use HDF5 instead of netCDF4?

I would say that if you do not have any specific restrictions (for example, models or visualization software that are found on netCDF4 files), you would be better off using netCDF. netCDF4 can be used by NCO / CDO, ncl (ncl also accepts HDF5), idl, python netCDF4 module, ferret, etc. Personally, I think netCDF4 is very convenient for storing climate or meteorological data. There are many operators for this, and you do not need to worry about developing a standard for your own data - this is already done for you. CMOR ( http://cmip-pcmdi.llnl.gov/cmip5/output_req.html ) can be used to write CF-compatible data. It was used for the most recent climate model assessment project.

HDF5, on the other hand, may be worth it if you have a different data type and you are looking for some very specific functions for which you need a more customizable file format. Could you clarify your needs for comments?

Update:

Unfortunately, the standards for variable and field names are a little less clear and well organized for HDF5 files than netCDF, as it was the format of choice for large climate modeling projects such as CMIP or CORDEX. The problem essentially melts in the use of the EOSDIS or CF conventions, but finding the currently supported librairies that implement these standards for HDF5 files and having clear documentation is not quite easy (if you probably wouldn’t ask a question).

If you really need a standard, NASA explains here all the possible metadata standards: http://gcmd.nasa.gov/add/standards/index.html .

For information, HDF-EOS and HDF5 do not have the exact same format (HDF-EOS already contains cartographic data and is standardized for earth science data), so I don’t know if this format will be too restrictive for you. Tools for working with this format are described here: http://hdfeos.net/software/tool.php and summarized here http://hdfeos.org/help/reference/HTIC_Brochure_Examples.pdf .

If you still prefer to use HDF5, the best option is probably to download an HDF5 file from NASA to get similar data and use it as the basis for creating your own tools in your language of choice. Here is a list of detailed examples using the HDF5, HDF4, and HDF-EOS formats with scripts for processing and visualizing data in Python, MATLAB, IDL, and NCL: http://hdfeos.net/zoo/index_openLAADS_Examples.php#MODIS

Essentially, the problem is that NASA provides affordable tools so that you can work with your data, but it is not necessary that you can re-create similar structures in your own laboratory.

Here are some more specifications / information about hdf5 for NASA Earth data: MERRA Product https://gmao.gsfc.nasa.gov/products/documents/MERRA_File_Specification.pdf GRADS HDF5 compatible information http: //disc.sci. gsfc.nasa.gov/recipes/?q=recipes/How-to-Read-Data-in-HDF-5-Format-with-GrADS HDF data processing tools in NASA Atmospheric Science Data Center: https: //eosweb.larc .nasa.gov / HBDOCS / hdf_data_manipulation.html

Hope this helps a bit.