Unable to write large restart file with NetCDF 3.6.3

[lanerolle]

I have a large application (1044 x 724 grid points with 30 vertical levels) and I find that upon writing out the perfect restart file, ROMS crashes saying:

NETCDF_ENDDEF - unable to end definition mode for file:
cook_inlet_rst1.nc
call from: def_rst.F

If run the same application with the RST_SINGLE CPP option turned on, it runs successfully and writes out a restart NetCDF file which is ~2.2Gb. I however would prefer to have a double precision restart file so that I can continue running the simulation in an exact fashion. I am using the NetCDF library version 3.6.3.

Has anyone seen this error before and if so, how did you remedy it? Does NetCDF 3.6.3 have a file size limit associated with it?

Thank you.

[kate]

I believe you have found your answer to be "yes, there's a limit". Newer versions of NetCDF allow you to open files in a 64-bit mode, allowing large files. Searching in the netcdf.inc file shows:

Code: Select all

      parameter (nf_format_classic = 1)
      parameter (nf_format_64bit = 2)

Do you have the 64bit option available in yours? The solution might be as simple as changing the mode parameter when creating new files. Alternately, you may need a newer version of NetCDF.

[lanerolle]

Yes, I have those two lines in their exact form in my netcdf.inc. What should I do? Could I switch them around so that

parameter (nf_format_classic = 1)
parameter (nf_format_64bit = 2)

becomes

parameter (nf_format_classic = 2)
parameter (nf_format_64bit = 1)

or are there more things I need to do? In my makefile, I have: USE_LARGE ?= on

Thanks.

[kate]

Don't touch the include file.

I don't see anything about 64-bit mode in the nf90_open options. It might be in how NetCDF was compiled or it might be in the version. Sometime late in NetCDF 3 they created the 64-bit option. I suggest you download recent versions of NetCDF 4 and the NetCDF Fortran packages. 64-bit might be the new default - I don't remember having this problem in recent years. Anyway, look through the configure options before doing the build.

As for USE_LARGE, that may or may not change the ROMS compile in the system-compiler.mk file in the Compilers directory. Check there for what it does.

[lanerolle]

I am still attempting to generate a ROMS code executable capable of writing files > 2.18 Gb. I am using ROMS svn version 562.

When I compile the code, it says the CPU configuration is x86_64 meaning that the compilation is being carried out with 64-bit processing. The NetCDF libraries I use too seem to have been compiled with 64-bit processing. In short, the entire environment on the supercomputer I use seems to use 64-bit computing.

When I contacted the sys admin for help on the supercomputer, this is something they said:

".... for files > 2GB, NetCDF3 does not do that by default, you have to specify large file support in your NetCDF API calls in your code."

When I compile the code, in my ROMS makefile, I have:

# If applicable, activate 64-bit compilation:

USE_LARGE ?= on

So I should be able to activate the 64-bit file processing in the NetCDF libraries (I use) and thereby write-out files > 2.18 Gb!

Is there anything else I need to do with the ROMS compilation/linking/execution to tell it to do 64-bit file processing?

[lanerolle]

You are correct Kate, the only time USE_LARGE is used is in the *.mk files in the /Compilers directory and for the following architectures only:

Compilers/AIX-xlf.mk
Compilers/IRIX64-f90.mk
Compilers/Linux-path.mk
Compilers/SunOS-f95.mk

I use Linux-ifort.mk and it does not use USE_LARGE and this switch is not used anywhere else in ROMS.

As I mentioned in my earlier posting, from the CPU type seen during the compilation (x86_64), it appears that ROMS is being compiled in 64-bit.