Perfect restart problem for large grid

[peisach]

We've encountered a problem when using PERFECT_RESTART. The model complains upon start-up for a setup where the grid is rather large (1022x0578x032). The console output ends with:

Code: Select all

   STEP  time[DAYS]  KINETIC_ENRG    POTEN_ENRG    TOTAL_ENRG   NET_VOLUME  trd

      0     0.00000  1.580116E-03  1.251719E+04  1.251719E+04  9.014432E+15   0
      DEF_HIS   - creating history file: /work/paalei/runROMS_Nordic4km/ocean_his.nc
      WRT_HIS   - wrote history  fields (Index=1,1) into time record = 0000001
      DEF_AVG   - creating average file: /work/paalei/runROMS_Nordic4km/ocean_avg.nc

 WRT_INFO - error while writing variable: ntimes
            into NetCDF file: /work/paalei/runROMS_Nordic4km/ocean_rst.nc

 ROMS/TOMS - Output error ............ exit_flag:   3

The model begins to create the restart file but fails.

We get this behavior on two different architectures when running under MPI. The model runs fine under otherwise similar conditions when a) we use a smaller grid and b) when compiling without PERFECT_RESTART.

If any of the developers would like to look into this but need more specific info, please let me know.

Pål Erik

[kate]

The first variable that gets written will be making the file large enough for 1 time record rather than zero time records. Do you expect this to be greater than 2 GB? Do you have an old netcdf library with a 2 GB limit? PERFECT_RESTART is going to increase the size of your restart file substantially. Hernan - is it a factor of about two?

[peisach]

Thanks for the reply Kate,

We're using netcdf 3.6.2, so I don't think this should be a problem. When running without PERFECT_RESTART other .nc files (_his, _avg) become much larger than 2GB (_rst stays below 2GB though).

Pål E

[kate]

Oh, I see it's failing on ntimes, not the time variable. Never mind...
Is def_info doing something different in the case of PERFECT_RESTART?