Blow up in hydrodinamic flow model

[nils.lindeen]

Hi:

I was running a hydrodinamic flow model of a Chilean Southern Channel and got a blow up. At the moment I am experimenting with the sensitivity of the grid size, in order to determine the appropriate grid for my model. In one of my models I got the following error after around 19.000 repetitions:

18936 2 04:36:00 2.814362E-03 5.035338E-03 7.849700E-03 3.231526E+13
WRT_HIS - wrote history fields (Index=1) into time record = 0002368
WRT_HIS - wrote history fields (Index=1) into time record = 0002369
WRT_HIS - wrote history fields (Index=1) into time record = 0002370
WRT_HIS - wrote history fields (Index=1) into time record = 0002371
WRT_HIS - wrote history fields (Index=1) into time record = 0002372
WRT_HIS - wrote history fields (Index=1) into time record = 0002373
WRT_HIS - wrote history fields (Index=1) into time record = 0002374
WRT_HIS - wrote history fields (Index=1) into time record = 0002375
WRT_HIS - wrote history fields (Index=1) into time record = 0002376
19008 2 04:48:00 2.309543E-03 5.540860E-03 7.850403E-03 3.231473E+13
WRT_HIS - wrote history fields (Index=1) into time record = 0002377

WRT_HIS - error while writing variable: zeta
into history NetCDF file for time record: 2378

Elapsed CPU time (seconds):

Thread # 0 CPU: 181.719
Total: 181.719

Nonlinear model elapsed time profile:

Initialization ................................... 0.012 ( 0.0066 %)
Reading of input data ............................ 0.020 ( 0.0110 %)
Processing of input data ......................... 0.624 ( 0.3434 %)
Computation of vertical boundary conditions ...... 3.544 ( 1.9504 %)
Computation of global information integrals ...... 0.132 ( 0.0726 %)
Writing of output data ........................... 2.808 ( 1.5453 %)
Model 2D kernel .................................. 139.901 (76.9873 %)
Tidal forcing .................................... 33.826 (18.6144 %)
Total: 180.867 99.5311

All percentages are with respect to total time = 181.719

ROMS/TOMS - Output NetCDF summary for Grid 01:
number of time records written in HISTORY file = 00002378

Analytical header files used:

ROMS/Functionals/ana_fsobc.h
ROMS/Functionals/ana_initial.h
ROMS/Functionals/ana_m2obc.h
ROMS/Functionals/ana_smflux.h

ROMS/TOMS - Output error ............ exit_flag: 3


ERROR: Abnormal termination: NetCDF OUTPUT.
REASON: NetCDF: Numeric conversion not representable

What could have happened with the zeta variable? Does anyone know how I could solve this error? Thanks alot.

Cheers,

Nils

[kate]

It's likely that ROMS blew up, giving you NaN or something similar. Many of us ask for "NINFO == 1" in the ocean.in file, which asks ROMS to check for velocities and densities getting "out of bounds" after each timestep. ROMS will then dump its state into the restart file so you can see what is getting "out of bounds" and where this is happening. In my experience, checking every 70-80 timesteps isn't enough because it can go from "good" to "bad" in a very short time (1-2 steps).

[nils.lindeen]

Thank you very much kate. This is very useful. I followed your recommendation, and started receiving info on every iteration. I keep running the model and getting a blow up, now I get the following:

49902 1 03:43:24 1.808319E-03 2.144465E-03 3.952785E-03 3.574247E+13
49903 1 03:43:26 1.807611E-03 2.145216E-03 3.952828E-03 3.574247E+13
49904 1 03:43:28 1.806903E-03 2.145968E-03 3.952871E-03 3.574247E+13

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=1) into time record = 0000003

Elapsed CPU time (seconds):

Thread # 0 CPU: 9398.475
Total: 9398.475

Nonlinear model elapsed time profile:

Initialization ................................... 0.256 ( 0.0027 %)
Reading of input data ............................ 0.128 ( 0.0014 %)
Processing of input data ......................... 28.566 ( 0.3039 %)
Computation of vertical boundary conditions ...... 163.938 ( 1.7443 %)
Computation of global information integrals ...... 273.865 ( 2.9139 %)
Writing of output data ........................... 3.176 ( 0.0338 %)
Model 2D kernel .................................. 7305.660 (77.7324 %)
Tidal forcing .................................... 1620.173 (17.2387 %)
Total: 9395.763 99.9711

All percentages are with respect to total time = 9398.475

ROMS/TOMS - Output NetCDF summary for Grid 01:
number of time records written in HISTORY file = 00000084
number of time records written in RESTART file = 00000084

Analytical header files used:

ROMS/Functionals/ana_fsobc.h
ROMS/Functionals/ana_initial.h
ROMS/Functionals/ana_m2obc.h
ROMS/Functionals/ana_smflux.h

ROMS/TOMS: DONE... Tuesday - March 22, 2011 - 9:22:35 PM

I can't manage to track where my error is occurring, because my RST file only registers the last three iterations. Would you happen to have and idea of where my error can be? What is the best way to track errors? What should I be looking for?

I checked the output file and noticed there are several NaN's, yet I thought this was OK because of the presence of land mask. Or should I have no NaN's in my output file?

Thank you very much.

Cheers,

Nils

[kate]

What I usually do is look at the rst file in ncview, especially the third time record in there. This will have either a large velocity somewhere or a large density (T/S) somewhere. I've added print statements to my diag.F to tell me which to look for. I guess this could get unwieldy for large grids. Anyway, your trouble could be mid-depth in the open ocean, at the bottom somewhere shallow, or perhaps at the surface. The location will tell you something about which processes are suspect. Well, honestly, I've had runs that just randomly blow up and I get past them by temporarily using a smaller timestep. Use your oldest record from the restart file (not the one that's already bad) and see if you can get past the trouble spot.

Some of mine:
* Mid-depth in the middle of nowhere - got past with shorter timestep.
* Surface above the Arctic Circle using DIURNAL_SRFLUX - ugly kludge to keep srflux finite.
* Shallow bottom where the tides are large (larger than in nature) - added more bottom drag there, then had to keep that finite to avoid a timestepping instability.