Blowing up problem

Message

nmichelet · #1 Unread post by **nmichelet** » Tue Nov 03, 2015 4:27 pm

Hello,
My model is blowing and I don't know why. I tried to change the time step but the only thing it does is to delay the blowing up.
I need help on this one
Thanks

517 0 08:37:00 7.672425E-03 5.298350E+02 5.298427E+02 1.803663E+12
(071,101,01) 2.688424E-02 1.100356E-02 3.424691E+00 1.221882E+00

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=1,2) into time record = 0000001

Elapsed CPU time (seconds):

Thread # 0 CPU: 307.121
Thread # 2 CPU: 307.120
Thread # 1 CPU: 307.121
Total: 921.362

Nonlinear model elapsed time profile, Grid: 01

Allocation and array initialization .............. 1.369 ( 0.1486 %)
Ocean state initialization ....................... 0.098 ( 0.0106 %)
Reading of input data ............................ 0.131 ( 0.0142 %)
Processing of input data ......................... 7.868 ( 0.8540 %)
Computation of vertical boundary conditions ...... 0.437 ( 0.0474 %)
Computation of global information integrals ...... 11.426 ( 1.2401 %)
Writing of output data ........................... 8.178 ( 0.8876 %)
Model 2D kernel .................................. 292.704 (31.7686 %)
Tidal forcing .................................... 58.418 ( 6.3404 %)
2D/3D coupling, vertical metrics ................. 10.977 ( 1.1914 %)
Omega vertical velocity .......................... 10.351 ( 1.1235 %)
Equation of state for seawater ................... 14.234 ( 1.5449 %)
GLS vertical mixing parameterization ............. 259.034 (28.1143 %)
3D equations right-side terms .................... 1.966 ( 0.2134 %)
3D equations predictor step ...................... 54.138 ( 5.8759 %)
Pressure gradient ................................ 24.947 ( 2.7076 %)
Corrector time-step for 3D momentum .............. 32.397 ( 3.5162 %)
Corrector time-step for tracers .................. 35.871 ( 3.8933 %)
Total: 824.544 89.4919

All percentages are with respect to total time = 921.362

ROMS/TOMS - Output NetCDF summary for Grid 01:
number of time records written in HISTORY file = 00000035
number of time records written in RESTART file = 00000001

Analytical header files used:

ROMS/Functionals/ana_btflux.h
ROMS/Functionals/ana_initial.h
ROMS/Functionals/ana_m2obc.h
ROMS/Functionals/ana_pair.h

ROMS/TOMS: DONE... Tuesday - November 3, 2015 - 4:42:37 PM

kate · #2 Unread post by **kate** » Tue Nov 03, 2015 5:42 pm

This made it into the FAQ, where I wrote:

ROMS will check for extreme values of velocity and density at run time, including looking for NaN and Inf. If found, it will save a record to the restart file and die. If this happens early in the run, you might just try a shorter timestep. If it happens after many days/years of model time, you need to dig more deeply. Find out which field went bad and where in the domain it happened. The restart record can be viewed to look for this trouble. I've had runs go bad at the bottom with a bottom stress instability, at the top with a surface forcing problem, at the boundary with boundary condition issues. I've also had it go bad in the middle of the water column in no place special due to some other instability - it goes from fine to terrible in just a few timesteps. This last can often be run past with a shorter timestep, while the other problems call for more thoughtful responses.

lalvarez · #3 Unread post by **lalvarez** » Wed Nov 04, 2015 7:58 pm

Hello

I'm having the same problem the model blows up at start, i'm using this timing

NTIMES == 349020
DT == 120
NDTFAST == 20
NRST == 3600
NSTA == 60
NFLT == 30
NINFO == 60
NHIS == 720
NAVG == 3600
NDIA == 720

and get the same as mmichelet

STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
C => (i,j,k) Cu Cv Cw Max Speed

720 1 00:00:00 2.197459E-08 1.847180E+04 1.847180E+04 3.649350E+15
(135,720,24) 0.000000E+00 1.688193E-03 0.000000E+00 4.603431E-02
DEF_HIS - creating history file, Grid 01: out/Leo_his.nc
WRT_HIS - wrote history fields (Index=1,1) into time record = 0000001
DEF_AVG - creating average file, Grid 01: out/Leo_avg.nc
DEF_DIAGS - creating diagnostics file, Grid 01: out/Leo_dia.nc
DEF_STATION - creating stations file, Grid 01: out/Leo_sta.nc
DEF_FLOATS - creating floats file, Grid 01: out/Leo_flt.nc
780 1 02:00:00 NaN NaN NaN NaN
(140,180,01) 0.000000E+00 0.000000E+00 0.000000E+00 0.000000E+00

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=1,1) into time record = 0000001

Shall i try a shorter DT & NDTFAST?

Thanks in advance

kate · #4 Unread post by **kate** » Wed Nov 04, 2015 8:30 pm

For you I would change NINFO to 1 and try again. Perhaps you'll see what's going bad before it gets all the way to NaN. Still, what I said above goes for you too.

nmichelet · Thu Nov 05, 2015 10:59 am

Hi,
I still have some trouble with that, the thing is the running is fine, the results are good, and the velocity suddenly blow up
Any idea why it does that?

nmichelet · #6 Unread post by **nmichelet** » Thu Nov 05, 2015 4:46 pm

I attached my configuration, if someone can help me with my blowing up problem.
Thanks

j.throatlocust · Fri Nov 06, 2015 4:50 am

*not an expert response*
have you tried compiling with lower optimization? i've experienced seemingly random blowups with some combinations of fortran&mpi using -O3 while -O2 ran just fine.

lalvarez · #8 Unread post by **lalvarez** » Fri Nov 06, 2015 8:00 pm

Hello j.throutlocust

Could you please remind me, where in the compilation procces the optimization -O3, -O2 is handle?

I'll try that

Any way I attach here my *.txt run with the NINFO == 1

Any help will be welcome

kate · #9 Unread post by **kate** » Fri Nov 06, 2015 8:20 pm

I didn't realize it was blowing up on timestep 4. That's a candidate for using a shorter timestep, especially considering this:

Code: Select all

 Maximum barotropic Courant Number =  7.51904786E-01

The compilation flags are set in one of the files in the Compilers directory. For me, it's one of Linux-gfortran.mk, Linux-pgi.mk or Linux-ifort.mk, depending on which compiler I use. Do you know which one you are using?

lalvarez · #10 Unread post by **lalvarez** » Sat Nov 07, 2015 5:24 pm

Hello Kate

I´m using linux-gfortran

Thanks for your replay, how small or big the Courant number shall be?

j.throatlocust · Sun Nov 08, 2015 4:58 am

Thanks for your replay, how small or big the Courant number shall be?

try checking out discussion archives, eg. viewtopic.php?f=14&t=3203

also

Minimum X-grid spacing, DXmin = 0.00000000E+00 km
Maximum X-grid spacing, DXmax = 2.50567216E+00 km
Minimum Y-grid spacing, DYmin = 0.00000000E+00 km
Maximum Y-grid spacing, DYmax = 2.64336073E+00 km
Minimum Z-grid spacing, DZmin = 0.00000000E+00 m
Maximum Z-grid spacing, DZmax = 5.83204037E+02 m

i've never used wet/dry or tides. i'm failure sure that depths and lateral spacing equal to zero are problematic

lalvarez · #12 Unread post by **lalvarez** » Tue May 24, 2016 4:26 pm

Hello to All

I'm having problems running the model, it blows up at the very start, as says inhere before I have change the DT=120 & NDTFAST=20 to DT=90 & NDTFAST=15 and get the same.

For DT=120 & NDTFAST=20, Cw seems to be the problem on the top (surface)

STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
C => (i,j,k) Cu Cv Cw Max Speed

0 0 00:00:00 7.629983E-04 1.709507E+04 1.709507E+04 1.829035E+15
(024,496,24) 1.495936E-02 1.956801E-02 0.000000E+00 5.645270E-01
DEF_HIS - creating history file, Grid 01: out/Leo_his.nc
WRT_HIS - wrote history fields (Index=1,1) into time record = 0000001
DEF_AVG - creating average file, Grid 01: out/Leo_avg.nc
DEF_DIAGS - creating diagnostics file, Grid 01: out/Leo_dia.nc
DEF_STATION - creating stations file, Grid 01: out/Leo_sta.nc
DEF_FLOATS - creating floats file, Grid 01: out/Leo_flt.nc
1 0 00:02:00 7.660043E-04 1.709522E+04 1.709522E+04 1.829017E+15
(190,147,02) 1.004694E-02 4.771787E-03 8.541715E-02 5.603352E-01
2 0 00:04:00 8.229286E-04 1.709520E+04 1.709520E+04 1.829014E+15
(249,001,24) 8.591022E-04 2.043021E-02 5.998961E-01 7.098817E-01
3 0 00:06:00 1.128838E-03 1.709518E+04 1.709519E+04 1.829010E+15
(249,002,24) 4.047536E-03 1.839679E-02 6.097147E-01 1.268815E+00
4 0 00:08:00 1.430059E-03 1.709515E+04 1.709515E+04 1.829006E+15
(249,003,24) 8.117144E-03 1.679275E-02 8.162755E-01 1.704026E+00
5 0 00:10:00 2.073490E-03 1.709511E+04 1.709511E+04 1.829002E+15
(249,004,24) 8.083570E-03 1.482645E-02 1.148147E+00 2.648130E+00
6 0 00:12:00 3.054326E-03 1.709507E+04 1.709507E+04 1.828998E+15
(145,487,24) 2.971580E-02 8.389082E-03 1.578579E+00 3.648552E+00
7 0 00:14:00 4.518798E-03 1.709502E+04 1.709503E+04 1.828993E+15
(202,282,24) 1.536393E-02 4.071418E-03 1.639278E+00 4.859915E+00
8 0 00:16:00 6.838625E-03 1.709498E+04 1.709499E+04 1.828989E+15
(144,488,24) 4.343268E-02 1.757173E-02 1.848257E+00 6.189621E+00
9 0 00:18:00 1.071863E-02 1.709494E+04 1.709495E+04 1.828985E+15
(145,487,24) 3.864271E-02 1.181540E-02 2.988795E+00 7.086973E+00
10 0 00:20:00 1.765047E-02 1.709490E+04 1.709492E+04 1.828981E+15
(036,495,05) 2.379341E-01 5.427742E-02 4.292391E+00 1.496650E+01
11 0 00:22:00 7.740497E-02 1.713411E+04 1.713419E+04 1.828976E+15
(036,495,06) 8.083927E-01 5.499493E-01 7.429847E+00 8.646381E+01

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=2,2) into time record = 0000001

As well as the DT=90 & NDTFAST=15

STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
C => (i,j,k) Cu Cv Cw Max Speed

0 0 00:00:00 7.629983E-04 1.709507E+04 1.709507E+04 1.829035E+15
(024,496,24) 1.121952E-02 1.467601E-02 0.000000E+00 5.645270E-01
DEF_HIS - creating history file, Grid 01: out/Leo_his.nc
WRT_HIS - wrote history fields (Index=1,1) into time record = 0000001
DEF_AVG - creating average file, Grid 01: out/Leo_avg.nc
DEF_DIAGS - creating diagnostics file, Grid 01: out/Leo_dia.nc
DEF_STATION - creating stations file, Grid 01: out/Leo_sta.nc
DEF_FLOATS - creating floats file, Grid 01: out/Leo_flt.nc
1 0 00:01:30 7.609989E-04 1.709522E+04 1.709522E+04 1.829018E+15
(190,147,02) 7.165161E-03 3.594490E-03 6.407607E-02 5.616428E-01
2 0 00:03:00 7.868065E-04 1.709520E+04 1.709520E+04 1.829015E+15
(249,001,24) 2.299166E-04 1.442612E-02 5.490264E-01 5.591896E-01
3 0 00:04:30 9.532306E-04 1.709520E+04 1.709520E+04 1.829013E+15
(249,002,24) 1.940113E-03 1.084835E-02 2.870952E-01 9.368728E-01
4 0 00:06:00 1.114391E-03 1.709518E+04 1.709519E+04 1.829010E+15
(249,002,24) 3.150762E-03 1.387032E-02 5.173575E-01 1.214382E+00
5 0 00:07:30 1.449336E-03 1.709516E+04 1.709516E+04 1.829007E+15
(249,003,24) 6.693739E-03 1.174276E-02 6.067252E-01 1.874076E+00
6 0 00:09:00 1.931803E-03 1.709513E+04 1.709513E+04 1.829004E+15
(249,003,24) 3.595443E-03 1.324369E-02 8.304611E-01 2.528241E+00
7 0 00:10:30 2.567905E-03 1.709510E+04 1.709510E+04 1.829001E+15
(145,487,24) 1.758485E-02 6.285132E-03 9.800466E-01 3.206420E+00
8 0 00:12:00 3.397174E-03 1.709506E+04 1.709507E+04 1.828998E+15
(145,487,24) 2.181912E-02 7.839011E-03 1.229455E+00 3.949442E+00
9 0 00:13:30 4.486488E-03 1.709503E+04 1.709503E+04 1.828994E+15
(202,282,24) 1.407477E-02 3.106545E-03 1.230775E+00 4.792613E+00
10 0 00:15:00 5.928210E-03 1.709500E+04 1.709500E+04 1.828991E+15
(144,488,24) 2.725591E-02 1.147215E-02 1.401741E+00 5.661016E+00
11 0 00:16:30 7.888896E-03 1.709497E+04 1.709497E+04 1.828988E+15
(145,487,24) 1.046769E-02 2.907613E-03 1.650414E+00 6.454720E+00
12 0 00:18:00 1.052264E-02 1.709494E+04 1.709495E+04 1.828985E+15
(145,487,24) 2.815872E-02 4.701853E-03 2.600716E+00 7.127135E+00
13 0 00:19:30 1.390535E-02 1.709490E+04 1.709492E+04 1.828982E+15
(036,495,07) 2.175534E-01 2.647289E-02 2.881235E+00 7.665643E+00
14 0 00:21:00 1.933889E-02 1.709487E+04 1.709489E+04 1.828979E+15
(033,492,08) 2.404581E-01 3.163855E-02 4.130301E+00 2.457876E+01

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=1,1) into time record = 0000001

I'm using now the daily HYCOM data for the boundaries and CCMP (6 hr) for wind forcing, my boundary structure is like this:

! 1 2 3 4

LBC(isFsur) == Che Che Clo Che ! free-surface
LBC(isUbar) == Shc Shc Clo Shc ! 2D U-momentum
LBC(isVbar) == Shc Shc Clo Shc ! 2D V-momentum
LBC(isUvel) == RadNud RadNud Clo RadNud ! 3D U-momentum
LBC(isVvel) == RadNud RadNud Clo RadNud ! 3D V-momentum
LBC(isMtke) == Gra Gra Clo Gra ! mixing TKE

LBC(isTvar) == RadNud RadNud Clo RadNud \ ! temperature
RadNud RadNud Clo RadNud ! salinity

and also change

! Nudging/relaxation time scales, inverse scales will be computed
! internally, [1:Ngrids].

TNUDG == 2*360.0d0 ! days
ZNUDG == 360.0d0 ! days
M2NUDG == 360.0d0 ! days
M3NUDG == 360.0d0 ! days

! Factor between passive (outflow) and active (inflow) open boundary
! conditions, [1:Ngrids]. If OBCFAC > 1, nudging on inflow is stronger
! than on outflow (recommended).

OBCFAC == 120.0d0 ! nondimensional

As suggested by Kate in

viewtopic.php?f=23&t=3975&p=16072&hilit ... yer#p16072

Thanks in advance to All
Have a nice day

kate · #13 Unread post by **kate** » Tue May 24, 2016 5:44 pm

You need to look at the fields in that restart file that it so nicely saved for you. We don't have enough information from the text output to know what's really going on. The model doesn't report whether it's a density problem or a velocity problem - you need to figure that out. Is it trouble at the surface or a lateral boundary or something else?

Ocean Modeling Discussion

Blowing up problem

Blowing up problem

Re: Blowing up problem

Re: Blowing up problem

Re: Blowing up problem

Re: Blowing up problem

Re: Blowing up problem

Re: Blowing up problem

Re: Blowing up problem

Re: Blowing up problem

Re: Blowing up problem

Re: Blowing up problem

Re: Blowing up problem

Re: Blowing up problem