NetCDF-4 developments

[m.hadfield]

As David Robertson explained at the ROMS/TOMS Asia-Pacific Workshop a couple of weeks ago, the latest netCDF release (netcdf-4.0.1, released on 30 March), includes prototype support for OpenDAP, though it's disabled by default. This support means, for example, that the netCDF ncdump utility can read OpenDAP URLs. The drawback is that the OpenDAP library in version 4.0.1 is written in C++, which makes it difficult to link Fortran programs against the netCDF library. So effectively you can have either OpenDAP or the Fortran interface, but not both.

The netCDF developers always said they were going to rewrite the OpenDAP library in C for version 4.1. The good news is that they seem to have already done this in the snapshots, which can be found here.

ftp://ftp.unidata.ucar.edu/pub/netcdf/s ... ily.tar.gz

I have built the latest one (netcdf-4.1-beta1-snapshot2009041409) on my Cygwin system with Gortran and G95 and confirmed that I can enable OpenDAP and also have a functional Fortran interface usable by ROMS. This should make it possible for ROMS to use OpenDAP URLs (but not ordinary URLs) as input files. More mundanely, it means you don't have to keep separate netCDF installations to support OpenDAP applications on the one hand and ROMS on the other.

There is a complication. If OpenDAP was enabled when netCDF was built, then some extra linker arguments (-lcurl -lrpc) are required when building ROMS; however if you haven't enabled OpenDAP you might not have these libraries, so adding these arguments will cause an error. Luckily the clever netCDF developers thought of a way to help the user cope with the extra complexity: they introduced the nc-config utility, which prints out information needed in building netCDF applications. I have modified CYGWIN-gfortran.mk and CYGWIN-g95.mk to use this utility. The relevant section from CYGWIN-gfortran.mk is

Code: Select all

ifdef USE_NETCDF4
        NC_CONFIG ?= nc-config
    NETCDF_INCDIR ?= $(shell $(NC_CONFIG) --fflags | grep -o "\-I.*" | cut -f 1 | cut -c "3-")
             LIBS := $(shell $(NC_CONFIG) --flibs)
else
    NETCDF_INCDIR ?= /usr/local/include
    NETCDF_LIBDIR ?= /usr/local/lib
             LIBS := -L$(NETCDF_LIBDIR) -lnetcdf
endif

If USE_NETCDF4 is not defined, it uses NETCDF_INCDIR and NETCDF_LIBDIR as usual. If USE_NETCDF4 is defined it creates a variable called NC_CONFIG equal to "nc-config", but set via the "?=" operator, so it can be overridden via an environment variable. The necessary values of NETCDF_INCDIR and LIBS are deduced from the output of the NC_CONFIG command. With most installations this will work with no modification, meaning that ROMS setup becomes a little easier than before. If the default nc-config command doesn't point to the right place, you can either modify your PATH or create an NC_CONFIG environment variable to specify it explicitly.

A couple of notes:

• The nc-config command is present in netCDF 4.0.1 but not in 4.0. However it's just a shell script that can be retrofitted if necessary.
• NETCDF_INCDIR is needed by the ROMS make file to point to the location of NETCDF.mod and TYPESIZES.mod. We get these from the output of "nc-config --fflags", but only after a bit mucking about with grep and cut. Perhaps the netCDF developers could be persuaded to add an option to provide this information directly.

[m.hadfield]

From the netCDF developers:

• The 4.0.1 release does not contain the C++ based opendap release, it contains the new, C-based one. So you can have OpenDAP and the Fortran interface with 4.0.1.
• Remember that the opendap features are still being tested. What is available in 4.0.1 and the daily snapshot represents a work in progress, which is scheduled for completion with the 4.1 release. It should not be used operationally before that release, because testing is still in progress.
• An --includedir option will be added to nc-config. This will direct it to print the path to the include directory (the one containing NETCDF.mod and TYPESIZES.mod). This is not there in the latest snapshot, netcdf-4.1-beta1-snapshot2009041500, but should make it into the next one.

And one more point: for versions of nc-config that lack the --include-dir option, the following is the simplest, and probably the best, way of determining the include directory:

Code: Select all

NETCDF_INCDIR ?= $(shell $(NC_CONFIG) --prefix)/include

It will fail if the include directory name was overridden when netCDF was configured, but why would anyone do that?

[m.hadfield]

I want to make a passing comment on a change documented in a recent Trac ticket and this thread seems like the best place to do it.

The ticket is 418 and it adds the following to the make file

Code: Select all

ifdef USE_DAP
             LIBS += $(shell curl-config --libs)
endif

This is unnecessarily elaborate IMHO, as the netCDF developers have added a command similar to (and inspired by) curl-config, namely nc-config. With this command, as I've described earlier in the thread, the ROMS make process can cope with whatever netCDF facilities are and aren't enabled without the user needing to define environment variables like USE_DAP.

The only fly in the ointment is that nc-config was introduced only in 4.0.1. However it's pretty trivial to retro-fit it to 4.0.

[arango]

Good point Mark. My problem with nc-config is that it is only available for version 4.0.1 or higher. We will need to have additional code in the make configuration files to test if this nc-config script exists. I have to admit that it is elegant an cleaner to have:

Code: Select all

ifdef USE_NETCDF4
        NC_CONFIG ?= nc-config
    NETCDF_INCDIR ?= $(shell $(NC_CONFIG) --prefix)/include
             LIBS := $(shell $(NC_CONFIG) --flibs)
else
    NETCDF_INCDIR ?= /usr/local/include
    NETCDF_LIBDIR ?= /usr/local/lib
endif

instead of

Code: Select all

ifdef USE_NETCDF4
    NETCDF_INCDIR ?= /usr/local/netcdf4/include
    NETCDF_LIBDIR ?= /usr/local/netcdf4/lib
      HDF5_LIBDIR ?= /usr/local/hdf5/lib
else
    NETCDF_INCDIR ?= /usr/local/include
    NETCDF_LIBDIR ?= /usr/local/lib
endif
             LIBS := -L$(NETCDF_LIBDIR) -lnetcdf
ifdef USE_NETCDF4
             LIBS += -L$(HDF5_LIBDIR) -lhdf5_hl -lhdf5 -lz
  ifdef USE_DAP
             LIBS += $(shell curl-config --libs)
  endif
endif

However, I like to manipulate all these libraries with the build script to use various combinations of the different compiled libraries versions (optimzed or debug) and various compilers and their versions. Perhaps, I need to play with the PATH for the various nc-config. I probably will consider that in the future. It is a good sugestion.

Anyway, you still need to have the cURL libraries for OPeNDAP. The same goes for HDF5 (serial or parallel) and the pnetcdf libraries. This NetCDF library business is becoming complicated nowadays. Even for people with a lot of experience compiling this in the past.

[m.hadfield]

It's not too hard to come up with an nc-config script for a netCDF version lacking one. Here's a not-quite minimal example

Code: Select all

#! /bin/sh
#
# A minimal netCDF configuration script nc-config

prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

libs="-L${libdir} -lnetcdf"
flibs="-L${libdir} -lnetcdf -lnetcdff"

usage()
{
    cat <<EOF
Usage: nc-config [OPTION]

Available values for OPTION include:

  --help        display this help message and exit
  --libs        library linking information for netcdf
  --flibs       libraries needed to link a Fortran program
  --prefix      Install prefix

EOF

    exit $1
}

if test $# -eq 0; then
    usage 1
fi

while test $# -gt 0; do
    case "$1" in
    --help)
	usage 0
	;;
     --libs)
       	echo $libs
       	;;
    --flibs)
       	echo $flibs
       	;;
    --prefix)
       	echo "${prefix}"
       	;;
    *)
        echo "unknown option: $1"
	usage
	exit 1
	;;
    esac
    shift
done

exit 0

Modify the prefix, libs and flibs variables near the top to suit your needs and you're set. (You don't need libs for ROMS, but I left it in for possible linking with C programs.)

The script above is for a netCDF installation that doesn't support the HDF5-based netCDF4 format, so doesn't need the HDF5 libraries, but it does need an extra library switch (-lnetcdff) for Fortran.

You can either put it on the PATH with the name nc-config, or you can get the ROMS make system to invoke it by passing an appropriate value of NC_CONFIG to the makefile. The latter approach might be helpful if you're switching between multiple netCDF installations.

You're right that netCDF is getting more complicated. The configuration script helps make it simpler again.