blowing up: Diag Speed Trouble

[yxli]

Dear All,

I'm trying to run ROMs by forcing with wrf output
It was blowing up after running for four hours.

I've already posted this question in the COAWST forum. Because it's related to ROMS error, I'm reposting it here again.

The error:

Diag Speed Trouble

(001,068,01) 3.272224E-03 1.006189E-01 1.642780E-01 2.868435E+01
DIAG speed trouble 28.6843469722120

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=1,1) in record = 0000003 01

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=1,1) in record = 0000003 02

Elapsed CPU time (seconds):

Node # 0 CPU: 11081.486
Total: 11081.486

Nonlinear ocean model elapsed time profile, Grid: 01

Allocation and array initialization .............. 0.153 ( 0.0014 %)
Ocean state initialization ....................... 0.104 ( 0.0009 %)
Reading of input data ............................ 0.185 ( 0.0017 %)
Processing of input data ......................... 0.057 ( 0.0005 %)
Processing of output time averaged data .......... 0.282 ( 0.0025 %)
Computation of vertical boundary conditions ...... 0.229 ( 0.0021 %)
Computation of global information integrals ...... 1.752 ( 0.0158 %)
Writing of output data ........................... 1.597 ( 0.0144 %)
Model 2D kernel .................................. 46.138 ( 0.4164 %)
2D/3D coupling, vertical metrics ................. 1.167 ( 0.0105 %)
Omega vertical velocity .......................... 1.041 ( 0.0094 %)
Equation of state for seawater ................... 2.417 ( 0.0218 %)
Atmosphere-Ocean bulk flux parameterization ...... 0.129 ( 0.0012 %)
GLS vertical mixing parameterization ............. 47.199 ( 0.4259 %)
3D equations right-side terms .................... 4.797 ( 0.0433 %)
3D equations predictor step ...................... 7.606 ( 0.0686 %)
Pressure gradient ................................ 3.755 ( 0.0339 %)
Harmonic mixing of tracers, geopotentials ........ 4.109 ( 0.0371 %)
Harmonic stress tensor, S-surfaces ............... 2.608 ( 0.0235 %)
Corrector time-step for 3D momentum .............. 4.280 ( 0.0386 %)
Corrector time-step for tracers .................. 4.472 ( 0.0404 %)
Total: 134.078 1.2099

Nonlinear model message Passage profile, Grid: 01

Message Passage: 2D halo exchanges ............... 0.384 ( 0.0035 %)
Message Passage: 3D halo exchanges ............... 0.036 ( 0.0003 %)
Message Passage: 4D halo exchanges ............... 0.014 ( 0.0001 %)
Message Passage: data broadcast .................. 0.013 ( 0.0001 %)
Message Passage: data reduction .................. 0.018 ( 0.0002 %)
Message Passage: data gathering .................. 0.191 ( 0.0017 %)
Message Passage: data scattering.................. 0.006 ( 0.0001 %)
Message Passage: boundary data gathering ......... 10276.047 (92.7317 %)
Message Passage: point data gathering ............ 6.168 ( 0.0557 %)
Total: 10282.878 92.7933

All percentages are with respect to total time = 11081.486


Nonlinear ocean model elapsed time profile, Grid: 02

Allocation and array initialization .............. 0.153 ( 0.0014 %)
Ocean state initialization ....................... 0.100 ( 0.0009 %)
Reading of input data ............................ 0.004 ( 0.0000 %)
Processing of input data ......................... 0.086 ( 0.0008 %)
Processing of output time averaged data .......... 0.926 ( 0.0084 %)
Computation of vertical boundary conditions ...... 0.772 ( 0.0070 %)
Computation of global information integrals ...... 6.748 ( 0.0609 %)
Writing of output data ........................... 3.185 ( 0.0287 %)
Model 2D kernel .................................. 203.097 ( 1.8328 %)
2D/3D coupling, vertical metrics ................. 3.295 ( 0.0297 %)
Omega vertical velocity .......................... 3.152 ( 0.0284 %)
Equation of state for seawater ................... 9.332 ( 0.0842 %)
Atmosphere-Ocean bulk flux parameterization ...... 0.359 ( 0.0032 %)
GLS vertical mixing parameterization ............. 188.940 ( 1.7050 %)
3D equations right-side terms .................... 19.003 ( 0.1715 %)
3D equations predictor step ...................... 29.087 ( 0.2625 %)
Pressure gradient ................................ 14.677 ( 0.1324 %)
Harmonic mixing of tracers, geopotentials ........ 18.980 ( 0.1713 %)
Harmonic stress tensor, S-surfaces ............... 10.593 ( 0.0956 %)
Corrector time-step for 3D momentum .............. 16.305 ( 0.1471 %)
Corrector time-step for tracers .................. 17.423 ( 0.1572 %)
Total: 546.216 4.9291

Nonlinear model message Passage profile, Grid: 02

Message Passage: 2D halo exchanges ............... 1.041 ( 0.0094 %)
Message Passage: 3D halo exchanges ............... 0.068 ( 0.0006 %)
Message Passage: 4D halo exchanges ............... 0.025 ( 0.0002 %)
Message Passage: data broadcast .................. 0.008 ( 0.0001 %)
Message Passage: data reduction .................. 0.031 ( 0.0003 %)
Message Passage: data gathering .................. 9.487 ( 0.0856 %)
Message Passage: data scattering.................. 0.010 ( 0.0001 %)
Message Passage: boundary data gathering ......... 0.353 ( 0.0032 %)
Message Passage: point data gathering ............ 3.295 ( 0.0297 %)
Total: 14.319 0.1292

All percentages are with respect to total time = 11081.488


ROMS/TOMS - Output NetCDF summary for Grid 01:
number of time records written in HISTORY file = 00000010
number of time records written in RESTART file = 00000010
number of time records written in AVERAGE file = 00000001

ROMS/TOMS - Output NetCDF summary for Grid 02:
number of time records written in HISTORY file = 00000010
number of time records written in RESTART file = 00000010
number of time records written in AVERAGE file = 00000003

Analytical header files used:

ROMS/Functionals/ana_btflux.h
ROMS/Functionals/ana_stflux.h