Blow up NaN at the second time step

[papaya]

Hi,

I am running one configuration with all boundaries closed.

But it blows up at the second time-step, gives NaN.

I checked the pm and pn are all positive. Masks are OK. The input initial values are OK.

Could anyone help guess what may be the potential causes?

Thanks in advance for your help.

I am using 6-hour wind forcing and monthly heat fluxes from input files.

below is cppdefs
/*
** Options for North Pacific grid (coarse)
*/
/* Model Physics */
#define SOLVE3D
#define UV_COR
#define UV_VIS2
#define TS_DIF2
#define UV_ADV
#define SALINITY
#define NONLIN_EOS
#define LMD_MIXING

/* Numerical schemes options */
#define TS_U3HADVECTION
#define TS_SVADVECTION
#define DJ_GRADPS
#define MIX_S_UV
#define MIX_S_TS
#define CURVGRID
#define MASKING
#define SPLINES
#ifdef LMD_MIXING
# define DIURNAL_SRFLUX
# define SOLAR_SOURCE
# define LMD_RIMIX
# define LMD_CONVEC
# define LMD_SKPP
#endif
/* Computational grid and initial conditions */
/* Surface Boundary Conditions */
#define QCORRECTION
#undef SCORRECTION

/* Bottom Boundary Conditions */
#define UV_QDRAG
#define ANA_BSFLUX
#define ANA_BTFLUX
/* Other options */
# define AVERAGES
# define AVERAGES_FLUXES



here is part of the log file:
Vertical S-coordinate System:

level S-coord Cs-curve at_hmin over_slope at_hmax

24 0.0000000 0.0000000 0.000 0.000 0.000
23 -0.0416667 -0.0416667 -0.208 -96.201 -192.193
22 -0.0833333 -0.0833333 -0.417 -192.401 -384.386
21 -0.1250000 -0.1250000 -0.625 -288.602 -576.579
20 -0.1666667 -0.1666667 -0.833 -384.803 -768.772
19 -0.2083333 -0.2083333 -1.042 -481.004 -960.965
18 -0.2500000 -0.2500000 -1.250 -577.204 -1153.158
17 -0.2916667 -0.2916667 -1.458 -673.405 -1345.352
16 -0.3333333 -0.3333333 -1.667 -769.606 -1537.545
15 -0.3750000 -0.3750000 -1.875 -865.806 -1729.738
14 -0.4166667 -0.4166667 -2.083 -962.007 -1921.931
13 -0.4583333 -0.4583333 -2.292 -1058.208 -2114.124
12 -0.5000000 -0.5000000 -2.500 -1154.408 -2306.317
11 -0.5416667 -0.5416667 -2.708 -1250.609 -2498.510
10 -0.5833333 -0.5833333 -2.917 -1346.810 -2690.703
9 -0.6250000 -0.6250000 -3.125 -1443.011 -2882.896
8 -0.6666667 -0.6666667 -3.333 -1539.211 -3075.089
7 -0.7083333 -0.7083333 -3.542 -1635.412 -3267.282
6 -0.7500000 -0.7500000 -3.750 -1731.613 -3459.475
5 -0.7916667 -0.7916667 -3.958 -1827.813 -3651.668
4 -0.8333333 -0.8333333 -4.167 -1924.014 -3843.861
3 -0.8750000 -0.8750000 -4.375 -2020.215 -4036.055
2 -0.9166667 -0.9166667 -4.583 -2116.415 -4228.248
1 -0.9583333 -0.9583333 -4.792 -2212.616 -4420.441
0 -1.0000000 -1.0000000 -5.000 -2308.817 -4612.634


Time Splitting Weights: ndtfast = 20 nfast = 29

Primary Secondary Accumulated to Current Step

1-0.0009651193358779 0.0500000000000000-0.0009651193358779 0.0500000000000000
2-0.0013488780126037 0.0500482559667939-0.0023139973484816 0.1000482559667939
3-0.0011514592651645 0.0501156998674241-0.0034654566136461 0.1501639558342179
4-0.0003735756740661 0.0501732728306823-0.0038390322877122 0.2003372286649002
5 0.0009829200513762 0.0501919516143856-0.0028561122363360 0.2505291802792859
6 0.0029141799764308 0.0501428056118168 0.0000580677400948 0.3006719858911026
7 0.0054132615310267 0.0499970966129953 0.0054713292711215 0.3506690825040979
8 0.0084687837865132 0.0497264335364439 0.0139401130576347 0.4003955160405418
9 0.0120633394191050 0.0493029943471183 0.0260034524767397 0.4496985103876600
10 0.0161716623600090 0.0486998273761630 0.0421751148367487 0.4983983377638231
11 0.0207585511322367 0.0478912442581626 0.0629336659689854 0.5462895820219856
12 0.0257765478740990 0.0468533167015507 0.0887102138430845 0.5931428987235363
13 0.0311633730493854 0.0455644893078458 0.1198735868924698 0.6387073880313821
14 0.0368391158442262 0.0440063206553765 0.1567127027366960 0.6827137086867586
15 0.0427031802506397 0.0421643648631652 0.1994158829873357 0.7248780735499238
16 0.0486309868367616 0.0400292058506332 0.2480468698240974 0.7649072794005569
17 0.0544704302037592 0.0375976565087951 0.3025173000278565 0.8025049359093520
18 0.0600380921294286 0.0348741349986072 0.3625553921572852 0.8373790709079591
19 0.0651152103984763 0.0318722303921357 0.4276706025557615 0.8692513013000948
20 0.0694434033194840 0.0286164698722119 0.4971140058752455 0.8978677711723067
21 0.0727201499285570 0.0251442997062377 0.5698341558038025 0.9230120708785446
22 0.0745940258796570 0.0215082922098099 0.6444281816834595 0.9445203630883544
23 0.0746596950216180 0.0177785909158270 0.7190878767050775 0.9622989540041814
24 0.0724526566618460 0.0140456061647461 0.7915405333669235 0.9763445601689275
25 0.0674437485167026 0.0104229733316538 0.8589842818836260 0.9867675335005814
26 0.0590334053485720 0.0070507859058187 0.9180176872321981 0.9938183194064001
27 0.0465456732896125 0.0040991156383901 0.9645633605218105 0.9979174350447902
28 0.0292219798521905 0.0017718319739095 0.9937853403740011 0.9996892670186996
29 0.0062146596259991 0.0003107329813000 1.0000000000000000 0.9999999999999996

ndtfast, nfast = 20 29 nfast/ndtfast = 1.45000

Centers of gravity and integrals (values must be 1, 1, approx 1/2, 1, 1):

1.000000000000 1.060707743385 0.530353871693 1.000000000000 1.000000000000

Power filter parameters, Fgamma, gamma = 0.28400 0.14200

inimum X-grid spacing, DXmin = 3.56516012E+00 km
Maximum X-grid spacing, DXmax = 6.89992369E+00 km
Minimum Y-grid spacing, DYmin = 3.57250659E+00 km
Maximum Y-grid spacing, DYmax = 6.91414262E+00 km
Minimum Z-grid spacing, DZmin = 2.08333333E-01 m
Maximum Z-grid spacing, DZmax = 1.92193074E+02 m

Minimum barotropic Courant Number = 1.65488657E-02
Maximum barotropic Courant Number = 6.00894638E-01
Maximum Coriolis Courant Number = 2.85801427E-02


NLM: GET_STATE - Read state initial conditions, t = 15.0000
(File: ross-init.nc, Rec=0001, Index=1)
- free-surface
(Min = -6.54475465E-02 Max = 1.45651758E-01)
- vertically integrated u-momentum component
(Min = -1.51900845E-02 Max = 2.84496117E-02)
- vertically integrated v-momentum component
(Min = -2.38564610E-02 Max = 2.63735764E-02)
- u-momentum component
(Min = -6.55869395E-02 Max = 7.05071539E-02)
- v-momentum component
(Min = -6.27952293E-02 Max = 8.01485404E-02)
- potential temperature
(Min = -2.11928606E+00 Max = 1.90658104E+00)
- salinity
(Min = 3.37327423E+01 Max = 3.47482071E+01)
GET_2DFLD - surface u-momentum stress, t = 15.0000
(File: ross-forc-winds-2008-2009.nc, Rec=0057, Index=1)
(Tmin= 1.0000 Tmax= 732.0000)
(Min = -2.87512303E-04 Max = 2.39183044E-04)
GET_2DFLD - surface v-momentum stress, t = 15.0000
(File: ross-forc-winds-2008-2009.nc, Rec=0057, Index=1)
(Tmin= 1.0000 Tmax= 732.0000)
(Min = -8.58551236E-05 Max = 4.35613448E-04)
GET_2DFLD - solar shortwave radiation flux, t = 15.0000
(File: ross-forc-others-2008-2009.nc, Rec=0001, Index=1)
(Tmin= 15.0000 Tmax= 715.0000)
(Min = 4.17997764E-05 Max = 8.87597491E-05)

GET_2DFLD - surface net heat flux, t = 15.0000
(File: ross-forc-others-2008-2009.nc, Rec=0001, Index=1)
(Tmin= 15.0000 Tmax= 715.0000)
(Min = -2.58137770E-06 Max = 3.63685964E-05)
GET_2DFLD - sea surface temperature climatology, t = 15.0000
(File: ross-forc-others-2008-2009.nc, Rec=0001, Index=1)
(Tmin= 15.0000 Tmax= 715.0000)
(Min = -2.10824072E+00 Max = 2.95414394E-01)
GET_2DFLD - surface net heat flux sensitivity to SST, t = 15.0000
(File: ross-forc-others-2008-2009.nc, Rec=0001, Index=1)
(Tmin= 15.0000 Tmax= 715.0000)
(Min = -8.83265974E-06 Max = -5.76379994E-06)
GET_2DFLD - surface net freswater flux, (E-P), t = 15.0000
(File: ross-forc-others-2008-2009.nc, Rec=0001, Index=1)
(Tmin= 15.0000 Tmax= 715.0000)
(Min = -3.40211011E-08 Max = 1.11327358E-09)

Maximum grid stiffness ratios: rx0 = 3.205928E-01 (Beckmann and Haidvogel)
rx1 = 1.506786E+01 (Haney)


Initial basin volumes: TotVolume = 4.55713112298191E+15 m3
MinVolume = 2.71359085971592E+06 m3
MaxVolume = 9.12060424140386E+09 m3
Max/Min = 3.36108304932847E+03

NL ROMS/TOMS: started time-stepping:( TimeSteps: 00006481 - 00026325)

GET_2DFLD - surface u-momentum stress, t = 15.2500
(File: ross-forc-winds-2008-2009.nc, Rec=0058, Index=2)
(Tmin= 1.0000 Tmax= 732.0000)
(Min = -1.87966836E-04 Max = 2.06705132E-04)
GET_2DFLD - surface v-momentum stress, t = 15.2500
(File: ross-forc-winds-2008-2009.nc, Rec=0058, Index=2)
(Tmin= 1.0000 Tmax= 732.0000)
(Min = -6.85601343E-05 Max = 5.07467804E-04)
GET_2DFLD - solar shortwave radiation flux, t = 46.0000
(File: ross-forc-others-2008-2009.nc, Rec=0002, Index=2)
(Tmin= 15.0000 Tmax= 715.0000)
(Min = 2.91693736E-05 Max = 5.75547898E-05)
GET_2DFLD - surface net heat flux, t = 46.0000
(File: ross-forc-others-2008-2009.nc, Rec=0002, Index=2)
(Tmin= 15.0000 Tmax= 715.0000)
(Min = -2.36543777E-05 Max = 2.08928959E-05)
GET_2DFLD - sea surface temperature climatology, t = 46.0000
(File: ross-forc-others-2008-2009.nc, Rec=0002, Index=2)
(Tmin= 15.0000 Tmax= 715.0000)
(Min = -2.23526547E+00 Max = 2.84039967E-01)
GET_2DFLD - surface net heat flux sensitivity to SST, t = 46.0000
(File: ross-forc-others-2008-2009.nc, Rec=0002, Index=2)
(Tmin= 15.0000 Tmax= 715.0000)
(Min = -9.38512202E-06 Max = -6.00503005E-06)
GET_2DFLD - surface net freswater flux, (E-P), t = 46.0000
(File: ross-forc-others-2008-2009.nc, Rec=0002, Index=2)
(Tmin= 15.0000 Tmax= 715.0000)
(Min = -2.53971858E-08 Max = 8.78294886E-09)

STEP time[DAYS] KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME trd

6480 15.000000 5.120143E-05 1.748784E+04 1.748784E+04 4.558697E+15 0
DEF_HIS - creating history file: /home/fan/ROMS/roms-examples/ross/out/his.nc
WRT_HIS - wrote history fields (Index=1,1) into time record = 0000001
DEF_AVG - creating average file: /home/fan/ROMS/roms-examples/ross/out/avg.nc
6481 15.002315 NaN NaN NaN 4.558697E+15 0

ROMS/TOMS - Blows up ................ exit_flag: 1

Blowing-up: Saving latest model state into RESTART file

Node # 2 CPU: 28.317
Node # 3 CPU: 28.317
WRT_RST - wrote re-start fields (Index=2,2) into time record = 0000001

Elapsed CPU time (seconds):

Node # 0 CPU: 28.318
Total: 113.270

Nonlinear model elapsed time profile:

Initialization ................................... 3.112 ( 2.7471 %)
Reading of input data ............................ 0.245 ( 0.2161 %)
Processing of input data ......................... 0.038 ( 0.0334 %)
Processing of output time averaged data .......... 0.121 ( 0.1071 %)
Computation of vertical boundary conditions ...... 0.005 ( 0.0043 %)
Computation of global information integrals ...... 0.084 ( 0.0744 %)
Writing of output data ........................... 20.646 (18.2273 %)
Model 2D kernel .................................. 3.420 ( 3.0198 %)
2D/3D coupling, vertical metrics ................. 0.183 ( 0.1613 %)
Omega vertical velocity .......................... 0.119 ( 0.1047 %)
Equation of state for seawater ................... 0.626 ( 0.5531 %)
KPP vertical mixing parameterization ............. 0.786 ( 0.6942 %)
3D equations right-side terms .................... 0.380 ( 0.3357 %)
3D equations predictor step ...................... 0.582 ( 0.5139 %)
Pressure gradient ................................ 0.312 ( 0.2759 %)
Harmonic mixing of tracers, S-surfaces ........... 0.197 ( 0.1740 %)
Harmonic stress tensor, S-surfaces ............... 0.321 ( 0.2834 %)
Corrector time-step for 3D momentum .............. 0.305 ( 0.2692 %)
Corrector time-step for tracers .................. 0.471 ( 0.4156 %)
Total: 31.954 28.2103

Nonlinear model message Passage profile:

Message Passage: 2D halo exchanges ............... 0.443 ( 0.3908 %)
Message Passage: 3D halo exchanges ............... 0.193 ( 0.1704 %)
Message Passage: 4D halo exchanges ............... 0.083 ( 0.0735 %)
Message Passage: data broadcast .................. 74.533 (65.8015 %)
Message Passage: data reduction .................. 0.019 ( 0.0169 %)
Message Passage: data gathering .................. 1.378 ( 1.2168 %)
Message Passage: data scattering.................. 2.273 ( 2.0071 %)
Total: 78.923 69.6770

All pecentages are with respect to total time = 113.270
Node # 1 CPU: 28.317

ROMS/TOMS - Output NetCDF summary for Grid 01:
number of time records written in HISTORY file = 00000001
number of time records written in RESTART file = 00000001

MAIN: Abnormal termination: BLOWUP.

[papaya]

We found out that it is due to the QCORRECTION in cppdefs file

Why is that? What factors may conflict with QCORRECTION?