NetCDF4 compression for ROMS output files

[rsignell]

Mathieu wrote:

The real problem of the mask is the excessive use of disk space. The READ_WATER, WRITE_WATER options of ROMS allow to write only the sea points and are certainly nice. However, Hernan indicated me that the resulting files are not CF-compliant. What would be a better solution?

Another way to save disk space that WOULD be CF-compliant would be to use the internal compression feature in NetCDF, available in NetCDF4. Many groups are starting to use just the parallel I/O and compression features of NetCDF4, using the same data model as NetCDF3.

I have seen the NetCDF4 option in the ROMS makefile, but haven't tried it out yet. To enable compression, you should be able to just add a line specifying "deflation" in def_var.F (found in ROMS/Utility).

Existing code (without deflation):

Code: Select all

          IF ((nVdim.eq.1).and.(Vdim(1).eq.0)) THEN
            status=nf90_def_var(ncid, TRIM(Vinfo(1)), Vtype,          &
   &                            varid = Vid)

New code (with deflation):

Code: Select all

            status=nf90_def_var(ncid, TRIM(Vinfo(1)), Vtype,          &
   &                            varid = Vid)
            status=nf90_def_var_deflate(ncid, TRIM(Vinfo(1)),         &
   &                            shuffle, deflate,deflate_level)

where shuffle, deflate and deflate_level are parameters described in the NetCDF-4 Fortran 90 Interface Guide.

To access these NetCDF4 files, tools like NcVIEW (or any other tool that wanted to access the files) would just need to be relinked with the NetCDF4 libraries instead of NetCDF3 libraries. (These tools linked with the NetCDF4 libraries would still work with "classic" NetCDF3 files).

Please report back if you try this out!

-Rich

[m.hadfield]

Thanks for that, Rich.

Don't you also have to ensure the file is created in NetCDF-4 format? By default, NetCDF version 4.x creates files in the NetCDF Classic format. See

http://www.unidata.ucar.edu/software/ne ... ormat.html

I have been linking ROMS against NetCDF 4.0 and 4.1 for some time, but haven't yet tried telling it to create NetCDF-4 format files--mostly because the analysis system I use, IDL, can't read them yet.

[rsignell]

Folks,

I went ahead and tested my own suggestion about testing NetCDF4 deflation and I'm happy to report that it worked great, reducing the ROMS NetCDF output file size significantly. On the test case I was using, which is 40% masked, the history file ended up 8 times smaller!.

Here's what I did (but Hernan, please don't beat me up -- I just did a quick hack to test the concept).

In all 15 calls to "nf90_create" (def_his, def_rst, ...) I changed "nf90_clobber" to "NF90_NETCDF4" to create NetCDF4 files instead of the "classic" NetCDF3 files.

In the Makefile (or actually, the build.bash file) I set

Code: Select all

 export     USE_NETCDF4=on.

to link with the NetCDF4 libraries.

Then in "def_var.F", I changed:

Code: Select all

     integer :: i, j, latt, status

to

Code: Select all

     integer :: i, j, latt, status, shuffle, deflate, deflate_level

and changed

Code: Select all

            IF ((nVdim.eq.1).and.(Vdim(1).eq.0)) THEN
              status=nf90_def_var(ncid, TRIM(Vinfo(1)), Vtype,          &
     &                            varid = Vid)
            ELSE
              status=nf90_def_var(ncid, TRIM(Vinfo(1)), Vtype,          &
     &                            Vdim(1:nVdim), Vid)
            END IF

to

Code: Select all

            IF ((nVdim.eq.1).and.(Vdim(1).eq.0)) THEN
              status=nf90_def_var(ncid, TRIM(Vinfo(1)), Vtype,          &
     &                            varid = Vid)
            ELSE
              status=nf90_def_var(ncid, TRIM(Vinfo(1)), Vtype,          &
     &                            Vdim(1:nVdim), Vid)
              shuffle = 1
              deflate = 1
              deflate_level = 1
              status=nf90_def_var_deflate(ncid, Vid,                    &
     &                            shuffle, deflate, deflate_level)
            END IF

to turn on deflation.

That's it! The resulting netcdf history file was 8 times smaller, and worked fine both with ncdump (the one that comes in the NetCDF4 distribution, of course) and with "ncview", which I relinked with NetCDF4. It also worked fine with NetCDF-Java4.

-Rich

[arango]

Thanks Rich for testing this. I will change it a little by using an internal CPP option for this so we can still support both NetCDF3 and NetCDF4 libraries. I need to read about the deflation levels. Perhaps, we need an input parameter to play with such values.

[m.hadfield]

...I need to read about the deflation levels. Perhaps, we need an input parameter to play with such values.

Yes. It would be wrong to assume that, because you're linking with the netCDF4 libraries, you want to write files in the NETCDF4 format or use deflation.

[rsignell]

Yes, I agree also that supplying the compression parameters should be controllable by the user. The folks testing NETCDF4 so far have found that while there is a huge savings going from deflate=0 to deflate=1, there is not much savings going to deflate=2 or higher. For more information on deflation and other NETCDF4 features such as chunking, see the NetCDF4 Performance Report (June, 2008):
http://www.hdfgroup.org/pubs/papers/200 ... report.pdf
-Rich

[arango]

I updated the svn repository to add this capability. For more information check the following track ticket. Notice that there are several possibilites for the NetCDF library needed to compile. It depends if you are running serial, shared-memory, and distributed-memory applications. The HDF5 library is also needed in NetCDF-4. We have to compile both the NetCDF-4 and HDF5 libraries in serial and parallel (MPI). In parallel we have additional libraries for MPICH, MPICH2, and OpenMPI. This even becomes more complicated for us because we use ifort, pgi, and gfortran. Luckly, we only build all these libraries once.

Because all the possible combinations, I highly recommnend to use either the new version of build.sh or build.bash to compile.

To produce NetCDF-4/HDF5 file format use new cpp option NETCDF4. If in addition file compression is desired, use new cpp option DEFLATE. Notice that all the ROMS standard input (ocean*.in) files were modified to include the file compression parameters.

[m.hadfield]

It works for me!