NETCDF4

[ocecept]

Hi guys;

I'm trying to have the NETCFD4 library working in serial and I get some kind of problem. That's what I did:

• I installed the NETCFD4/HDF5 libraries for the ifort compiler with no problems, all tests worked and I used h5dump to double check if I have a netcdf4 file.

• I added netcdf4 and hdf5 paths in the my compiler file (Linux-ifort.mk).

• Added #define NETCDF4 and #define DEFLATE in my .h file

• Added NetCDF-4/HDF5 compression parameters in my .in file.

• Added export USE_NETCDF4=on in my makefile and build files.

Then, I tried the upwelling test and I got the following error :


/opt/intel/fce/10.1.015/bin/ifort -c -g -check bounds -traceback -free mod_netcdf.f90
fortcom: Error: mod_netcdf.f90, line 106: This symbol must be a defined parameter or an argument of an inquiry function that evaluates to a compile-time constant. [NF90_NETCDF4]
integer :: CMODE = nf90_netcdf4 ! NetCDF-4/HDF5 format file
-------------------------^
fortcom: Error: mod_netcdf.f90, line 106: This is not a valid initialization expression. [NF90_NETCDF4]
integer :: CMODE = nf90_netcdf4 ! NetCDF-4/HDF5 format file
-------------------------^
fortcom: Error: mod_netcdf.f90, line 106: This name does not have a type, and must have an explicit type. [NF90_NETCDF4]
integer :: CMODE = nf90_netcdf4 ! NetCDF-4/HDF5 format file
-------------------------^
compilation aborted for mod_netcdf.f90 (code 1)
make: *** [Build/mod_netcdf.o] Error 1

I'm using ifort version 10.1 with the ROMS version 319.
Is there anything wrong with my recipe above? Any idea how to solve that?

Now, if I want to run the model in serial using MPICH2 do I need to have a second version of the NETCDF4/HDF5 library?

[robertson]

This indicates that you are still compiling against NetCDF-3. Remember that any NETCDF/HDF5 paths you set in you makefile will override the ones in the Linux-ifort.mk file. Likewise, NetCDF/HDF5 paths set in your build script will override your makefile AND Linux-ifort.mk.

If you are using the build script, make sure that you are following the logic carefully. Also, the build script has undergone many changes so make sure you are using an updated version.

Now, if I want to run the model in serial using MPICH2 do I need to have a second version of the NETCDF4/HDF5 library?

Fist of all I'm assuming you mean running in parallel with MPICH2. Yes, you will need to recompile HDF5 and NetCDF4 using the mpi compilers (mpif90,mpicc, etc.) Both the HDF5 and NetCDF4 packages have info on how to do this. Also check this post for more compiling hints.

[ocecept]

Robertson, thanks for your replay.

The problem was in my Linux-ifort.mk, it's not a ROMS default file and it was not updated. I had added in the netcdf4 and hdf5 paths , but I missed something else. I started it again following Darwin-ifort.mk and now everything works fine.

Sorry I'm always a little bit confused about shared-memory, distributed memory, serial and parallel.
The idea is to use MPICH2 to run the model using shared memory, but not PARALELL I/O. After I fix the problem above, I can to run the model using MPICH2 (8 cores - 2 x 4 tiling) and the same nectdf4/hdf5 library build using ifort.

Many thanks for your help!!

[robertson]

MPICH2 is an MPI implementation. MPI is only used for distributed memory. For shared memory you will need to set USE_OpenMP in your build script. This will set the OpenMP flag for your compiler.

I realize you don't wish to use NetCDF's parallel I/O but keep in mind that NetCDF Parallel I/O is only possible with MPI (distribute memory). Since you don't wish to use parallel I/O, I'm wondering why you want to use NetCDF-4 instead of NetCDF-3? NetCDF-3 is still faster than NetCDF-4 because it doesn't have have the overhead of using the HDF5 libraries to read and write files. If you don't have the need for NetCDF-4/HDF5 files I would suggest sticking with the NetCDF-3 libraries at this point.

[kate]

Actually, my serial timing was slightly faster creating HDF files than Netcdf files even without the DEFLATE option. Using compression could result in smaller files, though I don't know if that would be faster or not.

[ocecept]

Hi Robertson/ Kate; thanks for your comments.

Sorry for the late replay I was out for some days.

Yes. The NECTCDF4 does take longer to run, but it reduces the size of my output files a lot.

I did some tests...

For the upwelling test case using netcdf3 the ocean_his file is 29 MB, if I use netcdf4 it become 1 MB. In my tests the faster result is when I don't use the heap-arrays fortran Flag. Without this option using NETCDF4 the test takes 20% longer.

In my application (a 2 x 4 tiling grid, not a lot of written, analytical forcing), the output change from 15 GB to 5.8 GB using the NETCDF4 and it takes only 8% longer.

I'm using the deflate level 1 and the suffle is in both cases.

In my case 8% more time to run the model is fine since I will take a lot less to process and plot my results, but I guess if your application has a lot of written/reading the Netcdf-4 is not a good idea.