Geek Answers Handbook

Python, NetCDF4, and HDF5

I do not know why these packages are always such a pain to install. I have been using NetCDF / HDF5 for a long time, and it has always been a pure horror ride, allowing them to install or run correctly, whether on Linux or OSX, whether C or Cython or Python. The simple relationship between netcdf4 and hdf5 is a source of great pain for many people, and I really want the developers of these packages to finally do something about it.

So, the last specific problem I encountered is this: I am trying to install netCDF4 for python. I get the following error:

Package hdf5 was not found in the pkg-config search path Perhaps you should add the directory containing `hdf5.pc'

I tried installing hdf5 packages using apt-get, including:

  • libhdf5-serial-dev
  • libhdf5-serial
  • libhdf5-7
  • python h5py
  • libhdf5-dev
  • hdf5 tools
  • hdf5 helpers
  • libhdf5-7-DBG

Using pip, I tried:

 pip install h5py

which failed to resolve a dependency with Cython, which I installed manually. After that, he installed (apparently), but I can not find the hdf5.pc file anywhere.

I pull my hair out. Does anyone know how to get around this problem?

+7
python pip hdf5 netcdf
source share
1 answer

When you mix distribution packages and self-tuning packages, you increase your chances of problems (as you will find out).

Also, do you want h5py or want netcdf-python? I don't think netcdf-python has a dependency on h5py. Rather, netcdf-python links to the netcdf library, which in turn depends on the HDF5 C library.

h5py also associates with HDF5 C

There is a lot of software, that's true. Work it out step by step and it will make more sense in the end (says the guy who has been doing this for 15 years ... it gets easier!)

  • If you intend to do parallel programming, you will need an MPI implementation.
  • HDF5 now provides the foundation for NetCDF4. If you need concurrent programming, create HDF5 versus MPI implementation.
  • Install NetCDF4 C Library
  • now python bindings can get what they need from NetCDF4, HDF5 and MPI

Yes, there is a lot of software to configure and build. pkg-config can help here! When you see Package hdf5 was not found in the pkg-config search path , this means that you must configure PKG_CONFIG_DIR to indicate the location of the package configuration files. Unfortunately, hdf5 does not provide a .pc file (package-config), so you just have to do this part manually. Oh, and netcdf does not provide pkg-config: it provides a nc-config script that will use netcdf-python.

Let me give you a concrete example:

  • MPICH master installed in /home/robl/soft/mpich-master
  • HDF5 installed in /home/robl/soft/hdf5-1.8.16
    • for example, configured as ../../hdf5-1.8.16/configure --prefix=/home/robl/work/soft/hdf5-1.8.16 CC=/home/robl/work/soft/mpich/bin/mpicc --enable-parallel
  • NetCDF4 installed in /home/robl/soft/netcdf-master
    • eg. configured as ./configure CC=${HOME}/work/soft/mpich/bin/mpicc --prefix=${HOME}/work/soft/netcdf-master CPPFLAGS=-I${HOME}/work/soft/hdf5-1.8.16/include LDFLAGS=-L${HOME}/work/soft/hdf5-1.8.16/lib

now you have all the prerequisites for netcdf-python

by the way, http://unidata.imtqy.com/netcdf4-python/ contains the prerequisites and necessary settings

Do not get hung up on the pickiness of hdf5.pc. If you have nc-config in your path, it will provide the necessary information.

If you are creating concurrent programming, install CC in your MPI compiler. If not, you can skip the step `export CC = ... '':

 cd netcdf-python export CC=${HOME}/work/soft/mpich/bin/mpicc export PATH=${HOME}/work/soft/netcdf-master/bin:${PATH} python setup.py build
+1
source share

More articles:


All Articles