blowing up after run 4285 time-steps

[fannyzhang]

Hi everyone!

I run a Gulf of Mexico case. It is blowing-up at the 4285th time-step. And I have no idea why it blew up.

Here is the running log file:

Resolution, Grid 01: 0179x0134x015, Parallel Nodes: 2, Tiling: 002x001


Physical Parameters, Grid: 01
=============================

8640 ntimes Number of timesteps for 3-D equations.
600.000 dt Timestep size (s) for 3-D equations.
30 ndtfast Number of timesteps for 2-D equations between
each 3D timestep.
1 ERstr Starting ensemble/perturbation run number.
1 ERend Ending ensemble/perturbation run number.
0 nrrec Number of restart records to read from disk.
F LcycleRST Switch to recycle time-records in restart file.
144 nRST Number of timesteps between the writing of data
into restart fields.
1 ninfo Number of timesteps between print of information
to standard output.
T ldefout Switch to create a new output NetCDF file(s).
144 nHIS Number of timesteps between the writing fields
into history file.
1 ntsAVG Starting timestep for the accumulation of output
time-averaged data.
1440 nAVG Number of timesteps between the writing of
time-averaged data into averages file.
5.0000E+01 nl_tnu2(01) NLM Horizontal, harmonic mixing coefficient
(m2/s) for tracer 01: temp
5.0000E+01 nl_tnu2(02) NLM Horizontal, harmonic mixing coefficient
(m2/s) for tracer 02: salt
1.0000E+02 nl_visc2 NLM Horizontal, harmonic mixing coefficient
(m2/s) for momentum.
F LuvSponge Turning OFF sponge on horizontal momentum.
F LtracerSponge(01) Turning OFF sponge on tracer 01: temp
F LtracerSponge(02) Turning OFF sponge on tracer 02: salt
1.0000E-06 Akt_bak(01) Background vertical mixing coefficient (m2/s)
for tracer 01: temp
1.0000E-06 Akt_bak(02) Background vertical mixing coefficient (m2/s)
for tracer 02: salt
1.0000E-05 Akv_bak Background vertical mixing coefficient (m2/s)
for momentum.
3.0000E-04 rdrg Linear bottom drag coefficient (m/s).
3.0000E-03 rdrg2 Quadratic bottom drag coefficient.
2.0000E-02 Zob Bottom roughness (m).
1 lmd_Jwt Jerlov water type.
2 Vtransform S-coordinate transformation equation.
2 Vstretching S-coordinate stretching function.
7.0000E+00 theta_s S-coordinate surface control parameter.
1.0000E-01 theta_b S-coordinate bottom control parameter.
100.000 Tcline S-coordinate surface/bottom layer width (m) used
in vertical coordinate stretching.
1025.000 rho0 Mean density (kg/m3) for Boussinesq approximation.
0.000 dstart Time-stamp assigned to model initialization (days).
20130801.00 time_ref Reference time for units attribute (yyyymmdd.dd)
0.0000E+00 Tnudg(01) Nudging/relaxation time scale (days)
for tracer 01: temp
1.2000E+02 Tnudg(02) Nudging/relaxation time scale (days)
for tracer 02: salt
0.0000E+00 Znudg Nudging/relaxation time scale (days)
for free-surface.
0.0000E+00 M2nudg Nudging/relaxation time scale (days)
for 2D momentum.
0.0000E+00 M3nudg Nudging/relaxation time scale (days)
for 3D momentum.
0.0000E+00 obcfac Factor between passive and active
open boundary conditions.
T VolCons(1) NLM western edge boundary volume conservation.
T VolCons(2) NLM southern edge boundary volume conservation.
F VolCons(3) NLM eastern edge boundary volume conservation.
F VolCons(4) NLM northern edge boundary volume conservation.
10.000 T0 Background potential temperature (C) constant.
35.000 S0 Background salinity (PSU) constant.
1.000 gamma2 Slipperiness variable: free-slip (1.0) or
no-slip (-1.0).
F LuvSrc Turning OFF momentum point Sources/Sinks.
F LwSrc Turning OFF volume influx point Sources/Sinks.
F LtracerSrc(01) Turning OFF point Sources/Sinks on tracer 01: temp
F LtracerSrc(02) Turning OFF point Sources/Sinks on tracer 02: salt
F LsshCLM Turning OFF processing of SSH climatology.
F Lm2CLM Turning OFF processing of 2D momentum climatology.
F Lm3CLM Turning OFF processing of 3D momentum climatology.
T LtracerCLM(01) Turning ON processing of climatology tracer 01: temp
T LtracerCLM(02) Turning ON processing of climatology tracer 02: salt
F LnudgeM2CLM Turning OFF nudging of 2D momentum climatology.
F LnudgeM3CLM Turning OFF nudging of 3D momentum climatology.
F LnudgeTCLM(01) Turning OFF nudging of climatology tracer 01: temp
F LnudgeTCLM(02) Turning OFF nudging of climatology tracer 02: salt
T Hout(idFsur) Write out free-surface.
T Hout(idUbar) Write out 2D U-momentum component.
T Hout(idVbar) Write out 2D V-momentum component.
T Hout(idUvel) Write out 3D U-momentum component.
T Hout(idVvel) Write out 3D V-momentum component.
T Hout(idWvel) Write out W-momentum component.
T Hout(idOvel) Write out omega vertical velocity.
T Hout(idTvar) Write out tracer 01: temp
T Hout(idTvar) Write out tracer 02: salt
T Hout(idUsms) Write out surface U-momentum stress.
T Hout(idVsms) Write out surface V-momentum stress.
T Hout(idTsur) Write out surface net heat flux.
T Hout(idTsur) Write out surface net salt flux.
T Hout(idSrad) Write out shortwave radiation flux.
T Hout(idDano) Write out density anomaly.
T Hout(idHsbl) Write out depth of surface boundary layer.

T Aout(idFsur) Write out averaged free-surface.
T Aout(idUbar) Write out averaged 2D U-momentum component.
T Aout(idVbar) Write out averaged 2D V-momentum component.
T Aout(idUvel) Write out averaged 3D U-momentum component.
T Aout(idVvel) Write out averaged 3D V-momentum component.
T Aout(idWvel) Write out averaged W-momentum component.
T Aout(idOvel) Write out averaged omega vertical velocity.
T Aout(idTvar) Write out averaged tracer 01: temp
T Aout(idTvar) Write out averaged tracer 02: salt

Output/Input Files:

Output Restart File: output_data/mexico_rst_03.nc
Output History File: output_data/mexico_his_03.nc
Output Averages File: output_data/mexico_avg_03.nc
Input Grid File: input_data/t0303_smth9_grid.nc
Input Nonlinear Initial File: input_data/t0303_ini.nc
Input Forcing File 01: input_data/gom_shflux_era_12.nc
Input Forcing File 02: input_data/gom_sms_era_12.nc
Input Forcing File 03: input_data/gom_swflux_era_12.nc
Input Forcing File 04: input_data/gom_swrad_era_12.nc
Input Forcing File 05: input_data/gom_SST_era.nc
Input Climatology File: input_data/t0303_clm.nc
Input Boundary File: input_data/t0303_bry.nc

Tile partition information for Grid 01: 0179x0134x0015 tiling: 002x001

tile Istr Iend Jstr Jend Npts

0 1 90 1 134 180900
1 91 179 1 134 178890

Tile minimum and maximum fractional coordinates for Grid 01:
(interior points only)

tile Xmin Xmax Ymin Ymax grid

0 0.50 90.50 0.50 134.50 RHO-points
1 90.50 179.50 0.50 134.50 RHO-points

0 1.00 90.50 0.50 134.50 U-points
1 90.50 179.00 0.50 134.50 U-points

0 0.50 90.50 1.00 134.00 V-points
1 90.50 179.50 1.00 134.00 V-points

Maximum halo size in XI and ETA directions:

HaloSizeI(1) = 306
HaloSizeJ(1) = 435
TileSide(1) = 139
TileSize(1) = 13344


Lateral Boundary Conditions: NLM
============================

Variable Grid West Edge South Edge East Edge North Edge
--------- ---- ---------- ---------- ---------- ----------

zeta 1 Closed Gradient Gradient Closed

ubar 1 Closed Clamped Clamped Closed

vbar 1 Closed Clamped Clamped Closed

u 1 Closed Clamped Clamped Closed

v 1 Closed Clamped Clamped Closed

temp 1 Closed Clamped Clamped Closed

salt 1 Closed Clamped Clamped Closed

Activated C-preprocessing Options:

MEXICO Mexico
ANA_BSFLUX Analytical kinematic bottom salinity flux.
ANA_BTFLUX Analytical kinematic bottom temperature flux.
ANA_DQDSST Analytical surface heat flux sensitivity to SST.
ANA_SSFLUX Analytical kinematic surface salinity flux.
ASSUMED_SHAPE Using assumed-shape arrays.
AVERAGES Writing out time-averaged nonlinear model fields.
CURVGRID Orthogonal curvilinear grid.
DIFF_GRID Horizontal diffusion coefficient scaled by grid size.
DIURNAL_SRFLUX Modulate shortwave radiation by the local diurnal cycle.
DJ_GRADPS Parabolic Splines density Jacobian (Shchepetkin, 2002).
DOUBLE_PRECISION Double precision arithmetic.
LMD_CONVEC LMD convective mixing due to shear instability.
LMD_MIXING Large/McWilliams/Doney interior mixing.
LMD_NONLOCAL LMD convective nonlocal transport.
LMD_RIMIX LMD diffusivity due to shear instability.
LMD_SKPP KPP surface boundary layer mixing.
MASKING Land/Sea masking.
MIX_GEO_TS Mixing of tracers along geopotential surfaces.
MIX_S_UV Mixing of momentum along constant S-surfaces.
MPI MPI distributed-memory configuration.
NONLINEAR Nonlinear Model.
NONLIN_EOS Nonlinear Equation of State for seawater.
PERFECT_RESTART Processing perfect restart variables.
POWER_LAW Power-law shape time-averaging barotropic filter.
PROFILE Time profiling activated .
QCORRECTION Surface net heat flux correction.
!RST_SINGLE Double precision fields in restart NetCDF file.
SALINITY Using salinity.
SOLAR_SOURCE Solar Radiation Source Term.
SOLVE3D Solving 3D Primitive Equations.
SPLINES Conservative parabolic spline reconstruction.
TS_MPDATA Recursive flux corrected MPDATA 3D advection of tracers.
TS_DIF2 Harmonic mixing of tracers.
UV_ADV Advection of momentum.
UV_COR Coriolis term.
UV_U3HADVECTION Third-order upstream horizontal advection of 3D momentum.
UV_C4VADVECTION Fourth-order centered vertical advection of momentum.
UV_QDRAG Quadratic bottom stress.
UV_VIS2 Harmonic mixing of momentum.
VAR_RHO_2D Variable density barotropic mode.
VISC_GRID Horizontal viscosity coefficient scaled by grid size.

Process Information:

Node # 0 (pid= 19663) is active.
Node # 1 (pid= 19664) is active.

INITIAL: Configuring and initializing forward nonlinear model ...
*******

Vertical S-coordinate System, Grid 01:

level S-coord Cs-curve Z at hmin at hc half way at hmax

15 0.0000000 0.0000000 0.000 0.000 0.000 0.000
14 -0.0666667 -0.0004989 2.218 -3.358 -7.756 -9.254
13 -0.1333333 -0.0032168 4.418 -6.828 -21.397 -30.545
12 -0.2000000 -0.0100379 6.583 -10.502 -45.920 -74.091
11 -0.2666667 -0.0229247 8.698 -14.480 -86.531 -150.539
10 -0.3333333 -0.0439342 10.745 -18.863 -148.685 -271.046
9 -0.4000000 -0.0752335 12.706 -23.762 -238.129 -447.365
8 -0.4666667 -0.1191055 14.563 -29.289 -360.919 -691.881
7 -0.5333333 -0.1779212 16.295 -35.563 -523.342 -1017.453
6 -0.6000000 -0.2540303 17.883 -42.702 -731.630 -1436.828
5 -0.6666667 -0.3494770 19.310 -50.807 -991.205 -1961.091
4 -0.7333333 -0.4653579 20.566 -59.935 -1304.976 -2596.193
3 -0.8000000 -0.6004851 21.663 -70.024 -1669.791 -3335.689
2 -0.8666667 -0.7487245 22.650 -80.770 -2069.383 -4146.308
1 -0.9333333 -0.8938704 23.662 -91.360 -2460.770 -4940.147
0 -1.0000000 -1.0000000 25.000 -100.000 -2748.678 -5522.356

Time Splitting Weights for Grid 01: ndtfast = 30 nfast = 42
==================================

Primary Secondary Accumulated to Current Step

1-0.0008094437383769 0.0333333333333333-0.0008094437383769 0.0333333333333333
2-0.0014053566728197 0.0333603147912792-0.0022148004111966 0.0666936481246126
3-0.0017877524645903 0.0334071600137066-0.0040025528757869 0.1001008081383191
4-0.0019566842408176 0.0334667517625262-0.0059592371166046 0.1335675599008453
5-0.0019122901320372 0.0335319745705535-0.0078715272486418 0.1670995344713988
6-0.0016548570247459 0.0335957175749547-0.0095263842733877 0.2006952520463536
7-0.0011849025289723 0.0336508794757796-0.0107112868023600 0.2343461315221331
8-0.0005032751608631 0.0336903762267453-0.0112145619632232 0.2680365077488784
9 0.0003887272597151 0.0337071520654408-0.0108258347035081 0.3017436598143192
10 0.0014892209965583 0.0336941944901169-0.0093366137069498 0.3354378543044362
11 0.0027955815694920 0.0336445537902317-0.0065410321374578 0.3690824080946679
12 0.0043042707117221 0.0335513677379153-0.0022367614257356 0.4026337758325831
13 0.0060106451121704 0.0334078920475245 0.0037738836864348 0.4360416678801076
14 0.0079087469427945 0.0332075372104522 0.0116826306292293 0.4692492050905598
15 0.0099910761708920 0.0329439123123590 0.0216737068001213 0.5021931174029188
16 0.0122483446563884 0.0326108764399960 0.0339220514565097 0.5348039938429148
17 0.0146692120341107 0.0322025982847830 0.0485912634906204 0.5670065921276978
18 0.0172400033810439 0.0317136245503127 0.0658312668716643 0.5987202166780105
19 0.0199444086685725 0.0311389577709445 0.0857756755402368 0.6298591744489550
20 0.0227631639997064 0.0304741441486588 0.1085388395399432 0.6603333185976138
21 0.0256737146312911 0.0297153720153352 0.1342125541712342 0.6900486906129490
22 0.0286498597812016 0.0288595815276255 0.1628624139524359 0.7189082721405746
23 0.0316613792205220 0.0279045862015855 0.1945237931729578 0.7468128583421600
24 0.0346736416507075 0.0268492068942347 0.2291974348236653 0.7736620652363948
25 0.0376471948657328 0.0256934188392112 0.2668446296893981 0.7993554840756060
26 0.0405373376992232 0.0244385123436867 0.3073819673886213 0.8237939964192927
27 0.0432936737565710 0.0230872677537126 0.3506756411451924 0.8468812641730054
28 0.0458596469320356 0.0216441452951603 0.3965352880772280 0.8685254094681656
29 0.0481720587108284 0.0201154903974257 0.4447073467880565 0.8886408998655914
30 0.0501605672561820 0.0185097551070648 0.4948679140442384 0.9071506549726561
31 0.0517471682814030 0.0168377361985254 0.5466150823256415 0.9239883911711815
32 0.0528456577069106 0.0151128305891453 0.5994607400325521 0.9391012217603267
33 0.0533610761022577 0.0133513086655816 0.6528218161348098 0.9524525304259084
34 0.0531891349131379 0.0115726061288397 0.7060109510479478 0.9640251365547481
35 0.0522156244733761 0.0097996349650684 0.7582265755213239 0.9738247715198165
36 0.0503158038019030 0.0080591141492892 0.8085423793232269 0.9818838856691057
37 0.0473537721847154 0.0063819206892258 0.8558961515079423 0.9882658063583315
38 0.0431818225418188 0.0048034616164019 0.8990779740497611 0.9930692679747334
39 0.0376397765791564 0.0033640675316746 0.9367177506289175 0.9964333355064080
40 0.0305543017255206 0.0021094083123694 0.9672720523544381 0.9985427438187774
41 0.0217382098544504 0.0010909315881854 0.9890102622088885 0.9996336754069628
42 0.0109897377911118 0.0003663245930371 1.0000000000000000 0.9999999999999999

ndtfast, nfast = 30 42 nfast/ndtfast = 1.40000

Centers of gravity and integrals (values must be 1, 1, approx 1/2, 1, 1):

1.000000000000 1.047601458608 0.523800729304 1.000000000000 1.000000000000

Power filter parameters, Fgamma, gamma = 0.28400 0.18933

Metrics information for Grid 01:
===============================

Minimum X-grid spacing, DXmin = 9.53173644E+00 km
Maximum X-grid spacing, DXmax = 1.06053729E+01 km
Minimum Y-grid spacing, DYmin = 1.11200424E+01 km
Maximum Y-grid spacing, DYmax = 1.11200424E+01 km
Minimum Z-grid spacing, DZmin = -2.21806494E+00 m
Maximum Z-grid spacing, DZmax = 8.10618752E+02 m

Minimum barotropic Courant Number = 3.66226057E-02
Maximum barotropic Courant Number = 6.06729077E-01
Maximum Coriolis Courant Number = 4.45309430E-02

Horizontal mixing scaled by grid size, GRDMAX = 1.08596592E+01 km

NLM: GET_STATE - Read state initial conditions, t = 0 00:00:00
(Grid 01, File: t0303_ini.nc, Rec=0001, Index=1)
- free-surface
(Min = -9.41191056E-02 Max = 5.66728856E-01)
- vertically integrated u-momentum component
(Min = -1.53128034E-01 Max = 3.58059365E-01)
- vertically integrated v-momentum component
(Min = -4.51806903E-02 Max = 5.25055491E-01)
- u-momentum component
(Min = -3.83800638E-01 Max = 3.66731936E-01)
- v-momentum component
(Min = -1.24178428E-01 Max = 5.96819757E-01)
- potential temperature
(Min = 0.00000000E+00 Max = 3.00954853E+01)
- salinity
(Min = 0.00000000E+00 Max = 3.68432644E+01)
GET_2DFLD - surface u-momentum stress, t = 0 **:**:00
(Rec=0000248, Index=1, File: gom_sms_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -6.28735473E-04 Max = 1.13510756E-04)
GET_2DFLD - surface v-momentum stress, t = 0 **:**:00
(Rec=0000248, Index=1, File: gom_sms_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -3.79535163E-04 Max = 3.32130543E-04)
GET_2DFLD - solar shortwave radiation flux, t = 0 **:**:00
(Rec=0000248, Index=1, File: gom_swrad_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = 1.74109873E-05 Max = 1.17213358E-04)
GET_2DFLD - surface net heat flux, t = 0 **:**:00
(Rec=0000248, Index=1, File: gom_shflux_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -1.49345902E-05 Max = 8.47850475E-05)
GET_2DFLD - sea surface temperature climatology, t = 0 00:00:00
(Rec=0000063, Index=1, File: gom_SST_era.nc)
(Tmin= -31.0000 Tmax= 92.0000)
(Min = 5.01014471E+00 Max = 3.08119125E+01)
GET_NGFLD - 2D u-momentum eastern boundary condition, t = 0 00:00:00
(Rec=0000002, Index=2, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -8.05667788E-03 Max = 2.80851156E-01)
GET_NGFLD - 2D v-momentum eastern boundary condition, t = 0 00:00:00
(Rec=0000002, Index=2, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -2.07083095E-02 Max = 5.28398715E-02)
GET_NGFLD - 2D u-momentum southern boundary condition, t = 0 00:00:00
(Rec=0000002, Index=2, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -1.15728527E-01 Max = 1.95017345E-02)
GET_NGFLD - 2D v-momentum southern boundary condition, t = 0 00:00:00
(Rec=0000002, Index=2, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -2.07083095E-02 Max = 7.89264739E-02)
GET_NGFLD - 3D u-momentum eastern boundary condition, t = 0 00:00:00
(Rec=0000002, Index=2, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -1.46516804E-02 Max = 3.48577231E-01)
GET_NGFLD - 3D v-momentum eastern boundary condition, t = 0 00:00:00
(Rec=0000002, Index=2, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -1.23747356E-01 Max = 8.88440236E-02)
GET_NGFLD - 3D u-momentum southern boundary condition, t = 0 00:00:00
(Rec=0000002, Index=2, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -3.88944149E-01 Max = 5.51934391E-02)
GET_NGFLD - 3D v-momentum southern boundary condition, t = 0 00:00:00
(Rec=0000002, Index=2, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -1.17037132E-01 Max = 2.46601552E-01)
GET_NGFLD - temperature eastern boundary condition, t = 0 00:00:00
(Rec=0000002, Index=2, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = 0.00000000E+00 Max = 3.00768490E+01)
GET_NGFLD - salinity eastern boundary condition, t = 0 00:00:00
(Rec=0000002, Index=2, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = 0.00000000E+00 Max = 3.66071701E+01)
GET_NGFLD - temperature southern boundary condition, t = 0 00:00:00
(Rec=0000002, Index=2, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = 0.00000000E+00 Max = 2.94918575E+01)
GET_NGFLD - salinity southern boundary condition, t = 0 00:00:00
(Rec=0000002, Index=2, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = 0.00000000E+00 Max = 3.66492538E+01)
GET_3DFLD - potential temperature climatology, t = 0 00:00:00
(Rec=0000002, Index=1, File: t0303_clm.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = 0.00000000E+00 Max = 3.00954853E+01)
GET_3DFLD - salinity climatology, t = 0 00:00:00
(Rec=0000002, Index=1, File: t0303_clm.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = 0.00000000E+00 Max = 3.68432644E+01)

Basin information for Grid 01:

Maximum grid stiffness ratios: rx0 = 5.842748E-01 (Beckmann and Haidvogel)
rx1 = 1.225403E+01 (Haney)

Initial basin volumes: TotVolume = 3.4707308962E+15 m3
MinVolume = 1.1843849025E+08 m3
MaxVolume = 9.5545296460E+10 m3
Max/Min = 8.0670815931E+02

NL ROMS/TOMS: started time-stepping: (Grid: 01 TimeSteps: 00000001 - 00008640)

GET_2DFLD - surface u-momentum stress, t = 0 01:30:00
(Rec=0000249, Index=2, File: gom_sms_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -3.73395788E-04 Max = 4.19573827E-05)
GET_2DFLD - surface v-momentum stress, t = 0 01:30:00
(Rec=0000249, Index=2, File: gom_sms_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -8.77013259E-05 Max = 2.52759311E-04)
GET_2DFLD - solar shortwave radiation flux, t = 0 01:30:00
(Rec=0000249, Index=2, File: gom_swrad_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = 0.00000000E+00 Max = 6.80798259E-06)
GET_2DFLD - surface net heat flux, t = 0 01:30:00
(Rec=0000249, Index=2, File: gom_shflux_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -6.63252940E-05 Max = -5.63216467E-06)
GET_2DFLD - sea surface temperature climatology, t = 0 12:00:00
(Rec=0000064, Index=2, File: gom_SST_era.nc)
(Tmin= -31.0000 Tmax= 92.0000)
(Min = 5.01014471E+00 Max = 3.08119125E+01)
GET_NGFLD - 2D u-momentum eastern boundary condition, t = 31 00:00:00
(Rec=0000003, Index=1, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -4.73714434E-03 Max = 2.51743019E-01)
GET_NGFLD - 2D v-momentum eastern boundary condition, t = 31 00:00:00
(Rec=0000003, Index=1, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -2.11681854E-02 Max = 5.89200519E-02)
GET_NGFLD - 2D u-momentum southern boundary condition, t = 31 00:00:00
(Rec=0000003, Index=1, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -1.27171963E-01 Max = 1.79775469E-02)
GET_NGFLD - 2D v-momentum southern boundary condition, t = 31 00:00:00
(Rec=0000003, Index=1, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -2.11681854E-02 Max = 8.94291550E-02)
GET_NGFLD - 3D u-momentum eastern boundary condition, t = 31 00:00:00
(Rec=0000003, Index=1, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -1.62272770E-02 Max = 3.11885953E-01)
GET_NGFLD - 3D v-momentum eastern boundary condition, t = 31 00:00:00
(Rec=0000003, Index=1, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -1.14472292E-01 Max = 7.79567361E-02)
GET_NGFLD - 3D u-momentum southern boundary condition, t = 31 00:00:00
(Rec=0000003, Index=1, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -3.80327731E-01 Max = 3.59288864E-02)
GET_NGFLD - 3D v-momentum southern boundary condition, t = 31 00:00:00
(Rec=0000003, Index=1, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = -1.13985889E-01 Max = 2.92329460E-01)
GET_NGFLD - temperature eastern boundary condition, t = 31 00:00:00
(Rec=0000003, Index=1, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = 0.00000000E+00 Max = 2.99025517E+01)
GET_NGFLD - salinity eastern boundary condition, t = 31 00:00:00
(Rec=0000003, Index=1, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = 0.00000000E+00 Max = 3.65748367E+01)
GET_NGFLD - temperature southern boundary condition, t = 31 00:00:00
(Rec=0000003, Index=1, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = 0.00000000E+00 Max = 2.96639423E+01)
GET_NGFLD - salinity southern boundary condition, t = 31 00:00:00
(Rec=0000003, Index=1, File: t0303_bry.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = 0.00000000E+00 Max = 3.66007767E+01)
GET_3DFLD - potential temperature climatology, t = 31 00:00:00
(Rec=0000003, Index=2, File: t0303_clm.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = 0.00000000E+00 Max = 2.99627298E+01)
GET_3DFLD - salinity climatology, t = 31 00:00:00
(Rec=0000003, Index=2, File: t0303_clm.nc)
(Tmin= -31.0000 Tmax= 61.0000)
(Min = 0.00000000E+00 Max = 3.66464908E+01)

STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
C => (i,j,k) Cu Cv Cw Max Speed

0 0 00:00:00 8.373425E-04 1.555623E+04 1.555623E+04 3.450651E+15
(130,029,14) 1.440493E-02 2.934347E-02 0.000000E+00 6.239570E-01
DEF_HIS - creating history file, Grid 01: output_data/mexico_his_03.nc
WRT_HIS - wrote history fields (Index=1,1) into time record = 0000001
DEF_AVG - creating average file, Grid 01: output_data/mexico_avg_03.nc
1 0 00:10:00 8.201037E-04 1.555612E+04 1.555612E+04 3.450642E+15
(123,038,01) 1.015407E-02 2.403806E-02 1.050709E-01 6.799810E-01
2 0 00:20:00 8.348257E-04 1.555605E+04 1.555605E+04 3.450632E+15
(109,009,15) 3.478563E-02 1.430757E-02 4.279355E-01 7.247020E-01
3 0 00:30:00 8.549379E-04 1.555599E+04 1.555599E+04 3.450623E+15
(108,009,15) 1.662098E-02 7.813799E-03 6.510960E-01 6.572079E-01
4 0 00:40:00 8.785594E-04 1.555595E+04 1.555595E+04 3.450613E+15
(109,009,15) 9.408465E-03 2.974762E-04 8.512362E-01 6.707246E-01
5 0 00:50:00 9.006629E-04 1.555594E+04 1.555595E+04 3.450604E+15
(108,009,15) 1.312489E-02 2.784656E-03 8.825336E-01 6.965441E-01
6 0 01:00:00 9.237621E-04 1.555593E+04 1.555594E+04 3.450595E+15
(109,009,15) 3.017856E-03 2.323845E-03 7.446584E-01 7.146092E-01
7 0 01:10:00 9.415762E-04 1.555588E+04 1.555588E+04 3.450585E+15
(108,010,15) 9.267883E-03 2.859104E-03 1.083121E+00 7.439257E-01
8 0 01:20:00 9.525200E-04 1.555583E+04 1.555583E+04 3.450576E+15
(109,010,15) 8.705311E-03 2.679885E-02 1.039416E+00 7.574351E-01
9 0 01:30:00 9.649829E-04 1.555580E+04 1.555580E+04 3.450566E+15
(108,009,15) 8.934994E-03 3.190377E-03 7.714740E-01 8.540037E-01
GET_2DFLD - surface u-momentum stress, t = 0 04:30:00
(Rec=0000250, Index=1, File: gom_sms_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -1.73625312E-04 Max = 7.04354789E-05)
GET_2DFLD - surface v-momentum stress, t = 0 04:30:00
(Rec=0000250, Index=1, File: gom_sms_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -6.42496638E-05 Max = 2.49396039E-04)
GET_2DFLD - solar shortwave radiation flux, t = 0 04:30:00
(Rec=0000250, Index=1, File: gom_swrad_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -9.60765239E-09 Max = 9.60765239E-09)
GET_2DFLD - surface net heat flux, t = 0 04:30:00
(Rec=0000250, Index=1, File: gom_shflux_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -6.87354444E-05 Max = -4.80841693E-06)
10 0 01:40:00 9.780003E-04 1.555577E+04 1.555578E+04 3.450557E+15
(109,011,15) 0.000000E+00 3.552180E-02 9.788387E-01 7.747634E-01
11 0 01:50:00 9.900710E-04 1.555575E+04 1.555575E+04 3.450547E+15
(109,012,15) 0.000000E+00 1.793685E-02 5.304011E-01 7.786198E-01
12 0 02:00:00 1.005203E-03 1.555573E+04 1.555573E+04 3.450538E+15
(109,010,15) 4.731156E-03 2.215677E-02 8.174260E-01 7.803961E-01
13 0 02:10:00 1.023734E-03 1.555570E+04 1.555571E+04 3.450528E+15
(109,009,15) 2.256130E-02 7.324817E-03 5.892840E-01 7.804574E-01
14 0 02:20:00 1.043117E-03 1.555569E+04 1.555569E+04 3.450519E+15
(108,009,15) 1.160801E-02 2.009693E-03 9.553634E-01 7.780441E-01
15 0 02:30:00 1.063154E-03 1.555569E+04 1.555570E+04 3.450509E+15
(108,009,15) 1.509686E-03 1.154925E-02 4.800190E-01 7.810754E-01
16 0 02:40:00 1.085991E-03 1.555571E+04 1.555571E+04 3.450500E+15
(108,010,15) 2.713017E-03 4.185126E-03 6.845616E-01 7.843168E-01
17 0 02:50:00 1.110530E-03 1.555572E+04 1.555572E+04 3.450490E+15
(108,009,15) 2.063445E-04 1.150062E-02 4.048985E-01 7.844606E-01
18 0 03:00:00 1.136734E-03 1.555572E+04 1.555572E+04 3.450481E+15
(112,041,15) 3.718473E-03 9.559786E-03 5.115687E-01 7.928745E-01
19 0 03:10:00 1.163579E-03 1.555572E+04 1.555573E+04 3.450471E+15
(108,010,15) 1.479228E-03 1.621168E-03 5.293445E-01 7.972300E-01
20 0 03:20:00 1.190078E-03 1.555573E+04 1.555573E+04 3.450462E+15
(108,009,15) 1.727800E-02 2.088589E-03 4.869233E-01 8.068215E-01
21 0 03:30:00 1.215185E-03 1.555574E+04 1.555574E+04 3.450452E+15
(108,009,15) 8.085104E-03 8.416320E-03 8.837520E-01 8.152721E-01
22 0 03:40:00 1.239452E-03 1.555577E+04 1.555577E+04 3.450443E+15
(112,041,15) 7.628623E-03 2.022114E-03 4.900574E-01 8.178964E-01
23 0 03:50:00 1.262438E-03 1.555581E+04 1.555581E+04 3.450433E+15
(108,010,15) 3.770898E-03 7.659892E-03 4.374833E-01 8.187403E-01
24 0 04:00:00 1.284244E-03 1.555583E+04 1.555583E+04 3.450424E+15
(109,010,15) 2.293186E-03 2.664022E-02 3.373776E-01 8.203904E-01
25 0 04:10:00 1.305861E-03 1.555585E+04 1.555585E+04 3.450414E+15
(109,011,15) 0.000000E+00 2.003550E-02 5.976586E-01 8.206311E-01
26 0 04:20:00 1.327647E-03 1.555587E+04 1.555587E+04 3.450405E+15
(109,011,15) 0.000000E+00 8.889696E-03 2.976321E-01 8.193659E-01
27 0 04:30:00 1.348737E-03 1.555589E+04 1.555589E+04 3.450396E+15
(108,009,15) 3.058606E-04 3.479753E-03 3.581366E-01 8.190006E-01
GET_2DFLD - surface u-momentum stress, t = 0 07:30:00
(Rec=0000251, Index=2, File: gom_sms_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -9.72893724E-05 Max = 1.01963000E-04)
GET_2DFLD - surface v-momentum stress, t = 0 07:30:00
(Rec=0000251, Index=2, File: gom_sms_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -5.81437140E-05 Max = 2.07357974E-04)
GET_2DFLD - solar shortwave radiation flux, t = 0 07:30:00
(Rec=0000251, Index=2, File: gom_swrad_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_2DFLD - surface net heat flux, t = 0 07:30:00
(Rec=0000251, Index=2, File: gom_shflux_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -7.06396773E-05 Max = -4.15281741E-06)
28 0 04:40:00 1.368938E-03 1.555592E+04 1.555592E+04 3.450386E+15
(165,043,01) 2.518856E-03 1.492532E-02 3.080648E-01 8.203019E-01
29 0 04:50:00 1.388507E-03 1.555594E+04 1.555595E+04 3.450377E+15
(109,010,15) 2.165899E-03 1.716650E-02 3.928882E-01 8.287412E-01
30 0 05:00:00 1.407890E-03 1.555597E+04 1.555597E+04 3.450367E+15
(169,093,15) 8.336025E-03 0.000000E+00 3.667059E-01 8.379802E-01
31 0 05:10:00 1.426798E-03 1.555598E+04 1.555598E+04 3.450358E+15
(109,011,15) 0.000000E+00 2.375041E-02 4.790941E-01 8.448549E-01
32 0 05:20:00 1.445477E-03 1.555599E+04 1.555599E+04 3.450348E+15
... ...
GET_2DFLD - surface u-momentum stress, t = 29 19:30:00
(Rec=0000487, Index=2, File: gom_sms_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -3.60897617E-04 Max = 2.34752478E-04)
GET_2DFLD - surface v-momentum stress, t = 29 19:30:00
(Rec=0000487, Index=2, File: gom_sms_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = -3.65916023E-04 Max = 1.55999981E-04)
GET_2DFLD - solar shortwave radiation flux, t = 29 19:30:00
(Rec=0000487, Index=2, File: gom_swrad_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = 9.19471533E-05 Max = 2.16037668E-04)
GET_2DFLD - surface net heat flux, t = 29 19:30:00
(Rec=0000487, Index=2, File: gom_shflux_era_12.nc)
(Tmin= -30.9375 Tmax= 91.9375)
(Min = 1.73041981E-06 Max = 1.72706021E-04)
4276 29 16:40:00 4.002859E-03 1.553041E+04 1.553041E+04 3.414685E+15
(165,044,02) 1.140422E-02 3.158231E-02 1.546675E+00 1.438609E+00
4277 29 16:50:00 4.002749E-03 1.553040E+04 1.553040E+04 3.414677E+15
(165,044,02) 1.142009E-02 3.155711E-02 1.547802E+00 1.443782E+00
4278 29 17:00:00 4.002648E-03 1.553039E+04 1.553040E+04 3.414670E+15
(165,044,02) 1.145702E-02 3.149749E-02 1.550760E+00 1.448547E+00
4279 29 17:10:00 4.002553E-03 1.553039E+04 1.553039E+04 3.414663E+15
(165,044,02) 1.150741E-02 3.142819E-02 1.550801E+00 1.453797E+00
4280 29 17:20:00 4.002445E-03 1.553038E+04 1.553039E+04 3.414655E+15
(165,044,02) 1.151194E-02 3.140673E-02 1.549692E+00 1.459217E+00
4281 29 17:30:00 4.002351E-03 1.553038E+04 1.553038E+04 3.414648E+15
(165,044,02) 1.151461E-02 3.129309E-02 1.548540E+00 1.464030E+00
4282 29 17:40:00 4.002256E-03 1.553037E+04 1.553038E+04 3.414640E+15
(165,044,02) 1.149941E-02 3.123923E-02 1.550050E+00 1.468305E+00
4283 29 17:50:00 4.002154E-03 1.553037E+04 1.553037E+04 3.414633E+15
(165,044,02) 1.149261E-02 3.130889E-02 1.549774E+00 1.473229E+00
4284 29 18:00:00 4.002066E-03 1.553036E+04 1.553036E+04 3.414626E+15
(165,044,02) 1.141509E-02 3.147300E-02 1.553077E+00 1.477975E+00
4285 29 18:10:00 4.001982E-03 1.553035E+04 1.553036E+04 3.414618E+15
(165,044,02) 1.139055E-02 3.141720E-02 1.555586E+00 1.481969E+00

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=1,2) into time record = 0000030

Elapsed CPU time (seconds):

Node # 0 CPU: 3746.068
Node # 1 CPU: 3746.068
Total: 7492.137

Nonlinear model elapsed time profile:

Allocation and array initialization .............. 0.186 ( 0.0025 %)
Ocean state initialization ....................... 0.036 ( 0.0005 %)
Reading of input data ............................ 82.362 ( 1.0993 %)
Processing of input data ......................... 48.700 ( 0.6500 %)
Processing of output time averaged data .......... 165.235 ( 2.2054 %)
Computation of vertical boundary conditions ...... 5.068 ( 0.0676 %)
Computation of global information integrals ...... 46.541 ( 0.6212 %)
Writing of output data ........................... 53.405 ( 0.7128 %)
Model 2D kernel .................................. 2339.978 (31.2324 %)
2D/3D coupling, vertical metrics ................. 107.025 ( 1.4285 %)
Omega vertical velocity .......................... 60.200 ( 0.8035 %)
Equation of state for seawater ................... 232.204 ( 3.0993 %)
KPP vertical mixing parameterization ............. 707.888 ( 9.4484 %)
3D equations right-side terms .................... 165.593 ( 2.2102 %)
3D equations predictor step ...................... 422.083 ( 5.6337 %)
Pressure gradient ................................ 156.037 ( 2.0827 %)
Harmonic mixing of tracers, geopotentials ........ 198.626 ( 2.6511 %)
Harmonic stress tensor, S-surfaces ............... 76.380 ( 1.0195 %)
Corrector time-step for 3D momentum .............. 236.618 ( 3.1582 %)
Corrector time-step for tracers .................. 2330.753 (31.1093 %)
Total: 7434.917 99.2363

Nonlinear model message Passage profile:

Message Passage: 2D halo exchanges ............... 204.467 ( 2.7291 %)
Message Passage: 3D halo exchanges ............... 39.737 ( 0.5304 %)
Message Passage: 4D halo exchanges ............... 105.957 ( 1.4142 %)
Message Passage: data broadcast .................. 58.253 ( 0.7775 %)
Message Passage: data reduction .................. 47.260 ( 0.6308 %)
Message Passage: data gathering .................. 7.963 ( 0.1063 %)
Message Passage: data scattering.................. 0.099 ( 0.0013 %)
Total: 463.737 6.1897

All percentages are with respect to total time = 7492.137

ROMS/TOMS - Output NetCDF summary for Grid 01:
number of time records written in HISTORY file = 00000030
number of time records written in RESTART file = 00000030
number of time records written in AVERAGE file = 00000002

Analytical header files used:

ROMS/Functionals/ana_btflux.h
ROMS/Functionals/ana_srflux.h
ROMS/Functionals/ana_stflux.h

==========================================================================
I really appreciate your help on this.

Thanks in advance.

[kate]

Can you look at record 30 of your restart file and see where things are bad? That should give you a better clue than we can.

When the model blows up in diag like that, the default is to not give you very much information. I changed my code to tell me whether u/v or rho went bad first and at which location. Your grid is fairly small so you can probably just find the bad spot using ncview. For larger grids, I've got a python tool to tell me where on the grid the min/max of some field is.

[rtopper]

Minimum Z-grid spacing, DZmin = -2.21806494E+00 m
Maximum Z-grid spacing, DZmax = 8.10618752E+02 m

Your bathymetry seems to contain some negative depth values. Unless you specify the WET_DRY CPP option, the bathymetry cannot have values of 0 or negative values.

rx0 = 5.842748E-01 (Beckmann and Haidvogel)
rx1 = 1.225403E+01 (Haney)

Furthermore, you may want to consider smoothing your bathymetry; the rx0 and rx1 values are too high.