Grid size issue?

[jcharris]

I've encountered two errors that I can't seem to explain, both of which appear when I move from a small grid (240x160x10) to something larger for a real application.

Gridding issue:
---
With a 240x160 grid and a purely tidally forced domain (i.e., no input files which depend on the number of levels, or even for that matter when using purely analytical forcings and initial conditions), when I increase the number of levels beyond 10, ROMS crashes at the first timestep (e.g., the line after the first DEF_HIS). I assume this is because an invalid grid is formed, but I found no mention in the documentation why a grid would work with 5 or 10 levels without a problem, but then not work with 15 vertical levels.

Tidal forcing issue:
---
With a 240x160 grid and a purely tidally forced simulation, I am able to produce qualitatively good results, but on a relatively coarse grid. When I increase the resolution (e.g., 320x240, 480x320) there is an error in reading the forcing file (e.g., error message after printing "GET_NGFLD - tidal period"), before the beginning of the simulation. No other change was made other than the number of points in the grid, and the exact same Matlab scripts were used to create the grid and forcing files. It is almost as if there is a limit to the size of the input file that is allowed, but I do not see why.

Any suggestions as to what is causing either problem would be most welcome.

[kate]

There could be a memory limitation in your computer setup. Can you run the BENCHMARK application that comes with ROMS? It's got three versions, small, medium and large. I can run the largest, which takes over 2 GB (a common memory barrier). Can you run UPWELLING which has N=16, but a smaller horizontal grid?

Some limits can be seen with either "ulimit -a" or "limit", depending on your unix shell. What do they show you? I get:

ognipa 1013% limit
cputime 2:00:00
filesize unlimited
datasize unlimited
stacksize 8192 kbytes
coredumpsize 0 kbytes
memoryuse unlimited
vmemoryuse 4194304 kbytes
descriptors 1023
memorylocked 32 kbytes
maxproc 64

Look at the memoryuse and the datasize, also stacksize. Here I'm on the front end of a cluster, hence the limit on cputime.

[jcharris]

I don't believe it is a (system) memory limitation, but I will check on this, and if I can't solve it soon may try a different compiler.

I am able to run the BENCHMARK test with the small grid, but it does not work with the medium or large grids (failing at successively earlier points).

If I run 'ulimit -a', I get:

core file size (blocks, -c) 0
data seg size (kbytes, -d) 6144
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1000
virtual memory (kbytes, -v) unlimited

but of these, only the data seg size seems to be a potential issue. If I set this to unlimited and restart, it fails in the same way, and likewise with the stack size (e.g., increasing it to 12000).

The UPWELLING test case I have run many times before. I can in fact get it to run with 160 vertical levels without complaint (at least for the first few timesteps). Although (out of my curiosity at extremes), if I set N=1600, I get the error:

.
.
.
UV_C4VADVECTION Fourth-order centered vertical advection of momentum.
UV_LDRAG Linear bottom stress.
UV_VIS2 Harmonic mixing of momentum.
VAR_RHO_2D Variable density barotropic mode.
oceanS(29856) malloc: *** mmap(size=390967296) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
oceanS(29856) malloc: *** mmap(size=390967296) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
? FORTRAN Runtime Error:
? Runtime memory allocation fails

[kate]

Is that how the others failed? Right after the call to checkdefs (which prints out the cpp options chosen) is a call to the routine which allocates all the 3-D arrays from the data part of memory. A 6 MB limit on it is tiny (these days).

[jcharris]

It is not how the others failed. The others failed with e.g., a segmentation fault. I have tried recompiling with gfortran, and that fixes my original problems. I may return to try using Absoft in the future and figuring out what the problem is, and so I would be curious if there were standard things to check when moving ROMS to new platforms, but for now it seems using gfortran is working solution.

The error of allocating memory for the upwelling case with a large number of vertical levels appears even when using gfortran -- although >1000 levels is far beyond anything that I imagine needing anyway.

[kate]

Does gfortran allow you to build a bigger grid than the other? One way to get a seg fault is to try to allocate too much memory - it just might not be as graceful about telling you why you have a problem than gfortran.