Blowing-up: Saving latest model state into RESTART file

Discussion on computers, ROMS installation and compiling

Moderators: arango, robertson

Post Reply
Message
Author
vivek
Posts: 27
Joined: Tue Oct 20, 2015 1:40 pm
Location: IISc (Indian Institute of Science)

Blowing-up: Saving latest model state into RESTART file

#1 Unread post by vivek »

Dear All.,

I am using ROMS/TOMS version 3.7

My model is blowing and I don't know why. I tried to change the time step but the only thing it does is to delay the blowing up.

if i use daily forcing files its work fine. But if i use 3 hourly data, i am getting error..
I need help on this one


(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.43742007E+01 Max = 3.60705191E+01)

STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
C => (i,j,k) Cu Cv Cw Max Speed

0 0 00:00:00 3.813944E-14 2.032113E+04 2.032113E+04 1.374941E+17
(359,085,40) 2.942222E-06 0.000000E+00 0.000000E+00 1.264248E-04
DEF_HIS - creating history file: output/ocean_his_s_000001.nc
WRT_HIS - wrote history fields (Index=1,1) into time record = 0000001
1 0 00:15:00 NaN NaN NaN NaN
(000,000,00) 0.000000E+00 0.000000E+00 0.000000E+00 NaN

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=2,2) into time record = 0000001


model details are:

Physical Parameters, Grid: 01
=============================

1086984 ntimes Number of timesteps for 3-D equations.
900.000 dt Timestep size (s) for 3-D equations.
54 ndtfast Number of timesteps for 2-D equations between
each 3D timestep.
1 ERstr Starting ensemble/perturbation run number.
1 ERend Ending ensemble/perturbation run number.
0 nrrec Number of restart records to read from disk.
T LcycleRST Switch to recycle time-records in restart file.
10519200 nRST Number of timesteps between the writing of data
into restart fields.
1 ninfo Number of timesteps between print of information
to standard output.
T ldefout Switch to create a new output NetCDF file(s).
2922 nHIS Number of timesteps between the writing fields
into history file.
2922 ndefHIS Number of timesteps between creation of new
history files.
1 ntsAVG Starting timestep for the accumulation of output
time-averaged data.
2922 nAVG Number of timesteps between the writing of
time-averaged data into averages file.
2922 ndefAVG Number of timesteps between creation of new
time-averaged file.

User avatar
kate
Posts: 4091
Joined: Wed Jul 02, 2003 5:29 pm
Location: CFOS/UAF, USA

Re: Blowing-up: Saving latest model state into RESTART file

#2 Unread post by kate »

This is what I wrote in the FAQ:
ROMS will check for extreme values of velocity and density at run time, including looking for NaN and Inf. If found, it will save a record to the restart file and die. If this happens early in the run, you might just try a shorter timestep. If it happens after many days/years of model time, you need to dig more deeply. Find out which field went bad and where in the domain it happened. The restart record can be viewed to look for this trouble. I've had runs go bad at the bottom with a bottom stress instability, at the top with a surface forcing problem, at the boundary with boundary condition issues. I've also had it go bad in the middle of the water column in no place special due to some other instability - it goes from fine to terrible in just a few timesteps. This last can often be run past with a shorter timestep, while the other problems call for more thoughtful responses.

vivek
Posts: 27
Joined: Tue Oct 20, 2015 1:40 pm
Location: IISc (Indian Institute of Science)

Re: Blowing-up: Saving latest model state into RESTART file

#3 Unread post by vivek »

Dear Kate sir,

i am resolve the Nan values, but till now i am getting error, i am using ERA 3 hourly data for spinup. here i am attaching std.out file.. please help
Last edited by vivek on Mon Jun 13, 2016 6:05 am, edited 1 time in total.

rtopper
Posts: 24
Joined: Fri Jul 04, 2014 2:32 pm
Location: MARUM, University of Bremen

Re: Blowing-up: Saving latest model state into RESTART file

#4 Unread post by rtopper »

Code: Select all

 7.0000E+00  Tnudg(01)       Nudging/relaxation time scale (days)
                               for tracer 01: temp
 7.0000E+00  Tnudg(02)       Nudging/relaxation time scale (days)
                               for tracer 02: salt
 7.0000E+00  Znudg           Nudging/relaxation time scale (days)
                               for free-surface.
 7.0000E+00  M2nudg          Nudging/relaxation time scale (days)
                               for 2D momentum.
 7.0000E+00  M3nudg          Nudging/relaxation time scale (days)
                               for 3D momentum.
 1.2000E+00  obcfac          Factor between passive and active
                               open boundary conditions.
This is a fairly strong nudging on the open boundary. If the solution in your domain doesn't match the open boundary forcing values, this may cause problems. Try a weaker nudging; if you search the forum you'll find plenty of suggestions for commonly used values.
Besides that I don't see any obvious issues in your output file (although I'm a bit surprised by the surface air temperature range)

ymamoutos
Posts: 71
Joined: Fri Nov 19, 2010 2:33 pm
Location: University of Aegean

Re: Blowing-up: Saving latest model state into RESTART file

#5 Unread post by ymamoutos »

Greetings,

from your logfile, it seems that you have a fairly large vertical courant
number (Cw) and maximum speed. From step 86 and after at point 078,153,32/31
the maximum speed is almost 8.0+(!!!) m/s. Is this one of your point sources (river)?
Is river runoff mass transport vertical profile (river_Vshape variable) linear?
Try to run with a smaller dt (600.0 sec) and see if your model will blowup again.
One suggestion is to use MIX_ISO_TS or MIX_GEO_TS and not MIX_S_TS for your tracers.
Finally your code is a bit out-of-date and (as many of ROMS developers have suggest numerous times...)
it's always good to start new projects with an up-to-date version from the repository.

Giannis

User avatar
wilkin
Posts: 922
Joined: Mon Apr 28, 2003 5:44 pm
Location: Rutgers University
Contact:

Re: Blowing-up: Saving latest model state into RESTART file

#6 Unread post by wilkin »

According to your stdout file std_bio_tsmpdata_kpp.txt you have Chapman open boundary conditions for the biological tracers. There is no such option for tracers, so I don't know what ROMS will have actually done in this situation. Viewing the logic in t3dbc_im.F I suspect it applies no open boundary condition at all to those tracers (so they persist at their initial value?)

That said, I think the suggestion from other users that you use MIX_GEO_TS and not MIX_S_TS is a sound one. With a harmonic mixing coefficient of tnu2 = 30 m2/s mixing on s-surfaces will produce a lot of diapycnal mixing.
John Wilkin: DMCS Rutgers University
71 Dudley Rd, New Brunswick, NJ 08901-8521, USA. ph: 609-630-0559 jwilkin@rutgers.edu

vivek
Posts: 27
Joined: Tue Oct 20, 2015 1:40 pm
Location: IISc (Indian Institute of Science)

Re: Blowing-up: Saving latest model state into RESTART file

#7 Unread post by vivek »

if i use MIX_GEO_TS, now also i am getting same error.. i am also use Updated version only.. please anybody help me

vivek
Posts: 27
Joined: Tue Oct 20, 2015 1:40 pm
Location: IISc (Indian Institute of Science)

Re: Blowing-up: Saving latest model state into RESTART file

#8 Unread post by vivek »

Please anybody help me..

as mentioned above suggestions, if i use MIX_GEO_TS, now also i am getting blow up.. i am also use Updated version only.. please anybody help me



(279,124,29) 4.726798E-02 2.895333E-03 1.003029E+00 9.149879E+00
3579 24 20:30:00 1.860639E-02 2.031322E+04 2.031323E+04 1.374847E+17
(276,128,14) 2.385851E-03 2.385922E-02 1.044685E+00 9.151166E+00
3580 24 20:40:00 1.862703E-02 2.031321E+04 2.031323E+04 1.374847E+17
(276,128,13) 1.337072E-03 2.747369E-02 1.065380E+00 9.150607E+00
3581 24 20:50:00 1.864813E-02 2.031321E+04 2.031323E+04 1.374847E+17
(276,128,16) 5.488207E-03 7.530018E-02 1.053068E+00 9.307917E+00
3582 24 21:00:00 1.866973E-02 2.031321E+04 2.031323E+04 1.374847E+17
(276,128,15) 4.358536E-03 1.115185E-01 1.177086E+00 9.841154E+00
3583 24 21:10:00 1.869183E-02 2.031321E+04 2.031323E+04 1.374847E+17
(276,128,14) 3.349585E-03 1.362503E-01 1.237248E+00 1.018747E+01
3584 24 21:20:00 1.871379E-02 2.031321E+04 2.031323E+04 1.374847E+17
(276,128,17) 7.552868E-03 1.692605E-01 1.162672E+00 1.353261E+01
3585 24 21:30:00 1.873556E-02 2.031321E+04 2.031323E+04 1.374847E+17
(276,128,16) 6.592472E-03 2.430906E-01 1.336575E+00 1.643204E+01
3586 24 21:40:00 1.875772E-02 2.031321E+04 2.031323E+04 1.374848E+17
(276,128,15) 6.163458E-03 2.491682E-01 1.273919E+00 1.898475E+01
3587 24 21:50:00 1.878134E-02 2.031321E+04 2.031322E+04 1.374848E+17
(276,128,18) 9.807678E-03 3.021962E-01 1.211662E+00 2.035079E+01

Elapsed CPU time (seconds):

Node # 0 CPU: 3348.065
Node # 2 CPU: 3353.116
Node # 3 CPU: 3348.248
Node # 4 CPU: 3353.549
Node # 5 CPU: 3352.831
Node #252 CPU: 3272.201
Node #181 CPU: 3353.365
Node # 97 CPU: 3353.273
Node # 72 CPU: 3353.343


Nonlinear model message Passage profile, Grid: 01

Message Passage: 2D halo exchanges ............... 363617.817 (39.7374 %)
Message Passage: 3D halo exchanges ............... 37806.719 ( 4.1316 %)
Message Passage: 4D halo exchanges ............... 49633.166 ( 5.4241 %)
Message Passage: data broadcast .................. 6810.268 ( 0.7442 %)
Message Passage: data reduction .................. 320774.063 (35.0553 %)
Message Passage: data gathering .................. 164.115 ( 0.0179 %)
Message Passage: data scattering.................. 17171.095 ( 1.8765 %)
Total: 795977.244 86.9871

All percentages are with respect to total time = 915051.832


ROMS/TOMS - Output NetCDF summary for Grid 01:
number of time records written in HISTORY file = 00000001

Analytical header files used:

ROMS/Functionals/ana_btflux.h
ROMS/Functionals/ana_stflux.h

Biology model header files used:

ROMS/Nonlinear/Biology/fennel.h

ROMS/TOMS - Blows up ................ exit_flag: 1


MAIN: Abnormal termination: BLOWUP.
16247,2 Bot

User avatar
kate
Posts: 4091
Joined: Wed Jul 02, 2003 5:29 pm
Location: CFOS/UAF, USA

Re: Blowing-up: Saving latest model state into RESTART file

#9 Unread post by kate »

Visualize your restart file. In this business, you need to build some debugging skills of your own.

lindandan
Posts: 17
Joined: Tue Nov 03, 2015 12:52 am
Location: School of Space Science of USTC

Re: Blowing-up: Saving latest model state into RESTART file

#10 Unread post by lindandan »

kate wrote:Visualize your restart file. In this business, you need to build some debugging skills of your own.
Hello! I have a problem when run a spinup with climatology data.
I have run 855 days without error but blowup suddenly. I see the restart file,it has 11 variables 'zeta' 'u' 'v' 'ubar' 'vbar' 'temp' 'salt' 'rho' 'akv' 'akt' 'aks',and I found that there is a abnormal value at right boundary with coordinate(245,276). so I correct the value with its near value. But I just have corrected 9 variables without 'Akt''Akv''Aks',because I don't know how to correct them. And after I correct 9 variable of the restart file. I try to run with restart mode, but failed. What should I do ?
-----------------------
Next is the output
Model Input Parameters: ROMS/TOMS version 3.7
Sunday - June 12, 2016 - 12:33:04 PM
-----------------------------------------------------------------------------
Region Ocena System, 1/12 degree resolution

Operating system : Linux
CPU/hardware : x86_64
Compiler system : ifort
Compiler command : /usr/local/mpi/bin/mpif90
Compiler flags : -heap-arrays -fp-model precise -I/usr/local/mpi/include -ip -O3 -free -free -free

Input Script :
SVN Root URL : https://www.myroms.org/svn/src/trunk
SVN Revision : 783M

Local Root : /vol/home/student/lshw/roms/trunk
Header Dir : /vol/home/student/lshw/roms/MYPROJECT/SSC12
Header file : ssc12.h
Analytical Dir: /vol/home/student/lshw/roms/MYPROJECT/Functionals

Resolution, Grid 01: 0275x0275x032, Parallel Nodes: 16, Tiling: 004x004


Physical Parameters, Grid: 01
=============================

424800 ntimes Number of timesteps for 3-D equations.
360.000 dt Timestep size (s) for 3-D equations.
72 ndtfast Number of timesteps for 2-D equations between
each 3D timestep.
1 ERstr Starting ensemble/perturbation run number.
1 ERend Ending ensemble/perturbation run number.
1 nrrec Number of restart records to read from disk.
T LcycleRST Switch to recycle time-records in restart file.
720 nRST Number of timesteps between the writing of data
into restart fields.
1 ninfo Number of timesteps between print of information
to standard output.
F ldefout Switch to create a new output NetCDF file(s).
1200 nHIS Number of timesteps between the writing fields
into history file.
14400 ndefHIS Number of timesteps between creation of new
history files.
8640000 ntsAVG Starting timestep for the accumulation of output
time-averaged data.
864000 nAVG Number of timesteps between the writing of
time-averaged data into averages file.
864000 ndefAVG Number of timesteps between creation of new
time-averaged file.
2.5000E+01 nl_tnu2(01) NLM Horizontal, harmonic mixing coefficient
(m2/s) for tracer 01: temp
2.5000E+01 nl_tnu2(02) NLM Horizontal, harmonic mixing coefficient
(m2/s) for tracer 02: salt
5.0000E+01 nl_visc2 NLM Horizontal, harmonic mixing coefficient
(m2/s) for momentum.
T LuvSponge Turning ON sponge on horizontal momentum.
T LtracerSponge(01) Turning ON sponge on tracer 01: temp
T LtracerSponge(02) Turning ON sponge on tracer 02: salt
1.0000E-05 Akt_bak(01) Background vertical mixing coefficient (m2/s)
for tracer 01: temp
1.0000E-05 Akt_bak(02) Background vertical mixing coefficient (m2/s)
for tracer 02: salt
1.0000E-05 Akv_bak Background vertical mixing coefficient (m2/s)
for momentum.
5.0000E-06 Akk_bak Background vertical mixing coefficient (m2/s)
for turbulent energy.
5.0000E-06 Akp_bak Background vertical mixing coefficient (m2/s)
for turbulent generic statistical field.
3.0000E-04 rdrg Linear bottom drag coefficient (m/s).
2.5000E-03 rdrg2 Quadratic bottom drag coefficient.
2.0000E-02 Zob Bottom roughness (m).
2 Vtransform S-coordinate transformation equation.
4 Vstretching S-coordinate stretching function.
6.0000E+00 theta_s S-coordinate surface control parameter.
5.0000E-01 theta_b S-coordinate bottom control parameter.
15.000 Tcline S-coordinate surface/bottom layer width (m) used
in vertical coordinate stretching.
1025.000 rho0 Mean density (kg/m3) for Boussinesq approximation.
0.000 dstart Time-stamp assigned to model initialization (days).
-1.00 time_ref Reference time for units attribute (yyyymmdd.dd)
3.6000E+02 Tnudg(01) Nudging/relaxation time scale (days)
for tracer 01: temp
3.6000E+02 Tnudg(02) Nudging/relaxation time scale (days)
for tracer 02: salt
3.6000E+02 Znudg Nudging/relaxation time scale (days)
for free-surface.
3.6000E+02 M2nudg Nudging/relaxation time scale (days)
for 2D momentum.
3.6000E+02 M3nudg Nudging/relaxation time scale (days)
for 3D momentum.
1.2000E+02 obcfac Factor between passive and active
open boundary conditions.
F VolCons(1) NLM western edge boundary volume conservation.
F VolCons(2) NLM southern edge boundary volume conservation.
F VolCons(3) NLM eastern edge boundary volume conservation.
F VolCons(4) NLM northern edge boundary volume conservation.
5.000 T0 Background potential temperature (C) constant.
35.000 S0 Background salinity (PSU) constant.
-1.000 gamma2 Slipperiness variable: free-slip (1.0) or
no-slip (-1.0).
F LuvSrc Turning OFF momentum point Sources/Sinks.
F LwSrc Turning OFF volume influx point Sources/Sinks.
F LtracerSrc(01) Turning OFF point Sources/Sinks on tracer 01: temp
F LtracerSrc(02) Turning OFF point Sources/Sinks on tracer 02: salt
F LsshCLM Turning OFF processing of SSH climatology.
F Lm2CLM Turning OFF processing of 2D momentum climatology.
F Lm3CLM Turning OFF processing of 3D momentum climatology.
T LtracerCLM(01) Turning ON processing of climatology tracer 01: temp
T LtracerCLM(02) Turning ON processing of climatology tracer 02: salt
F LnudgeM2CLM Turning OFF nudging of 2D momentum climatology.
F LnudgeM3CLM Turning OFF nudging of 3D momentum climatology.
T LnudgeTCLM(01) Turning ON nudging of climatology tracer 01: temp
T LnudgeTCLM(02) Turning ON nudging of climatology tracer 02: salt
T Hout(idFsur) Write out free-surface.
T Hout(idUbar) Write out 2D U-momentum component.
T Hout(idVbar) Write out 2D V-momentum component.
T Hout(idUvel) Write out 3D U-momentum component.
T Hout(idVvel) Write out 3D V-momentum component.
T Hout(idWvel) Write out W-momentum component.
T Hout(idOvel) Write out omega vertical velocity.
T Hout(idTvar) Write out tracer 01: temp
T Hout(idTvar) Write out tracer 02: salt
T Hout(idUsms) Write out surface U-momentum stress.
T Hout(idVsms) Write out surface V-momentum stress.
T Hout(idTsur) Write out surface net heat flux.
T Hout(idTsur) Write out surface net salt flux.
T Hout(idSrad) Write out shortwave radiation flux.
T Hout(idDano) Write out density anomaly.
T Hout(idVvis) Write out vertical viscosity: AKv.
T Hout(idTdif) Write out vertical diffusion: AKt(itemp).
T Hout(idSdif) Write out vertical diffusion: AKt(isalt).
T Hout(idMtke) Write out turbulent kinetic energy.
T Hout(idMtls) Write out turbulent generic length-scale.


Output/Input Files:

Output Restart File: ssc12_rst.nc
Prefix for History Files: ssc12_his
Node # 9 (pid= 10133) is active.
Prefix for Averages Files: ssc12_avg
Node # 12 (pid= 10136) is active.
Input Grid File: ../Data/ssc12_grd.nc
Node # 13 (pid= 10137) is active.
Input Nonlinear Initial File: ssc12_rst.nc
Node # 8 (pid= 10132) is active.
Input Forcing File 01: ../Data/ssc12_frc.nc
Node # 10 (pid= 10134) is active.
Input Climatology File: ../Data/ssc12_clm.nc
Node # 11 (pid= 10135) is active.
Input Boundary File: ../Data/ssc12_bry.nc
Node # 14 (pid= 10138) is active.
Input/Output USER File: MyFile.dat
Node # 15 (pid= 10139) is active.

Generic User Parameters:

2.0000E+01 user(01) User parameter 01.

Tile partition information for Grid 01: 0275x0275x0032 tiling: 004x004

tile Istr Iend Jstr Jend Npts

0 1 69 1 69 152352
1 70 138 1 69 152352
2 139 207 1 69 152352
3 208 275 1 69 150144
4 1 69 70 138 152352
5 70 138 70 138 152352
6 139 207 70 138 152352
7 208 275 70 138 150144
8 1 69 139 207 152352
9 70 138 139 207 152352
10 139 207 139 207 152352
11 208 275 139 207 150144
12 1 69 208 275 150144
13 70 138 208 275 150144
14 139 207 208 275 150144
15 208 275 208 275 147968

Tile minimum and maximum fractional coordinates for Grid 01:
(interior points only)

tile Xmin Xmax Ymin Ymax grid

0 0.50 69.50 0.50 69.50 RHO-points
1 69.50 138.50 0.50 69.50 RHO-points
2 138.50 207.50 0.50 69.50 RHO-points
3 207.50 275.50 0.50 69.50 RHO-points
4 0.50 69.50 69.50 138.50 RHO-points
5 69.50 138.50 69.50 138.50 RHO-points
6 138.50 207.50 69.50 138.50 RHO-points
7 207.50 275.50 69.50 138.50 RHO-points
8 0.50 69.50 138.50 207.50 RHO-points
9 69.50 138.50 138.50 207.50 RHO-points
10 138.50 207.50 138.50 207.50 RHO-points
11 207.50 275.50 138.50 207.50 RHO-points
12 0.50 69.50 207.50 275.50 RHO-points
13 69.50 138.50 207.50 275.50 RHO-points
14 138.50 207.50 207.50 275.50 RHO-points
15 207.50 275.50 207.50 275.50 RHO-points

0 1.00 69.50 0.50 69.50 U-points
1 69.50 138.50 0.50 69.50 U-points
2 138.50 207.50 0.50 69.50 U-points
3 207.50 275.00 0.50 69.50 U-points
4 1.00 69.50 69.50 138.50 U-points
5 69.50 138.50 69.50 138.50 U-points
6 138.50 207.50 69.50 138.50 U-points
7 207.50 275.00 69.50 138.50 U-points
8 1.00 69.50 138.50 207.50 U-points
9 69.50 138.50 138.50 207.50 U-points
10 138.50 207.50 138.50 207.50 U-points
11 207.50 275.00 138.50 207.50 U-points
12 1.00 69.50 207.50 275.50 U-points
13 69.50 138.50 207.50 275.50 U-points
14 138.50 207.50 207.50 275.50 U-points
15 207.50 275.00 207.50 275.50 U-points

0 0.50 69.50 1.00 69.50 V-points
1 69.50 138.50 1.00 69.50 V-points
2 138.50 207.50 1.00 69.50 V-points
3 207.50 275.50 1.00 69.50 V-points
4 0.50 69.50 69.50 138.50 V-points
5 69.50 138.50 69.50 138.50 V-points
6 138.50 207.50 69.50 138.50 V-points
7 207.50 275.50 69.50 138.50 V-points
8 0.50 69.50 138.50 207.50 V-points
9 69.50 138.50 138.50 207.50 V-points
10 138.50 207.50 138.50 207.50 V-points
11 207.50 275.50 138.50 207.50 V-points
12 0.50 69.50 207.50 275.00 V-points
13 69.50 138.50 207.50 275.00 V-points
14 138.50 207.50 207.50 275.00 V-points
15 207.50 275.50 207.50 275.00 V-points

Maximum halo size in XI and ETA directions:

HaloSizeI(1) = 162
HaloSizeJ(1) = 162
TileSide(1) = 75
TileSize(1) = 5625


Lateral Boundary Conditions: NLM
============================

Variable Grid West Edge South Edge East Edge North Edge
--------- ---- ---------- ---------- ---------- ----------

zeta 1 Chapman Imp Chapman Imp Chapman Imp Chapman Imp

ubar 1 Flather Flather Flather Flather

vbar 1 Flather Flather Flather Flather

u 1 Radiation Radiation Radiation Radiation

v 1 Radiation Radiation Radiation Radiation

temp 1 Clamped Clamped Clamped Clamped

salt 1 Clamped Clamped Clamped Clamped

tke 1 Rad + Nud Rad + Nud Rad + Nud Rad + Nud

Activated C-preprocessing Options:

SSC12 South Sea of China Internal tidal System, 1/12 degree resolution
ANA_BSFLUX Analytical kinematic bottom salinity flux.
ANA_BTFLUX Analytical kinematic bottom temperature flux.
ANA_NUDGCOEF Analytical spatially varying nudging time-scales.
ANA_SPONGE Analytical enhanced viscosity/diffusion sponge.
ASSUMED_SHAPE Using assumed-shape arrays.
AVERAGES Writing out time-averaged nonlinear model fields.
CURVGRID Orthogonal curvilinear grid.
DJ_GRADPS Parabolic Splines density Jacobian (Shchepetkin, 2002).
DOUBLE_PRECISION Double precision arithmetic.
KANTHA_CLAYSON Kantha and Clayson stability function formulation.
MASKING Land/Sea masking.
MIX_GEO_TS Mixing of tracers along geopotential surfaces.
MIX_GEO_UV Mixing of momentum along geopotential surfaces.
MPI MPI distributed-memory configuration.
MY25_MIXING Mellor/Yamada Level-2.5 mixing closure.
NONLINEAR Nonlinear Model.
NONLIN_EOS Nonlinear Equation of State for seawater.
N2S2_HORAVG Horizontal smoothing of buoyancy and shear.
POWER_LAW Power-law shape time-averaging barotropic filter.
PROFILE Time profiling activated .
QCORRECTION Surface net heat flux correction.
K_GSCHEME Third-order upstream advection of TKE fields.
RADIATION_2D Use tangential phase speed in radiation conditions.
RI_SPLINES Parabolic Spline Reconstruction for Richardson Number.
!RST_SINGLE Double precision fields in restart NetCDF file.
SALINITY Using salinity.
SOLVE3D Solving 3D Primitive Equations.
SPLINES_VDIFF Parabolic Spline Reconstruction for Vertical Diffusion.
SPLINES_VVISC Parabolic Spline Reconstruction for Vertical Viscosity.
TS_U3HADVECTION Third-order upstream horizontal advection of tracers.
TS_SVADVECTION Parabolic splines vertical advection of tracers.
TS_DIF2 Harmonic mixing of tracers.
UV_ADV Advection of momentum.
UV_COR Coriolis term.
UV_U3HADVECTION Third-order upstream horizontal advection of 3D momentum.
UV_C4VADVECTION Fourth-order centered vertical advection of momentum.
UV_QDRAG Quadratic bottom stress.
UV_VIS2 Harmonic mixing of momentum.
VAR_RHO_2D Variable density barotropic mode.

Process Information:

Node # 0 (pid= 10246) is active.
Node # 1 (pid= 10247) is active.
Node # 5 (pid= 10251) is active.
Node # 3 (pid= 10249) is active.
Node # 7 (pid= 10253) is active.
Node # 2 (pid= 10248) is active.
Node # 4 (pid= 10250) is active.
Node # 6 (pid= 10252) is active.

INITIAL: Configuring and initializing forward nonlinear model ...
*******

Vertical S-coordinate System, Grid 01:

level S-coord Cs-curve Z at hmin at hc half way at hmax

32 0.0000000 0.0000000 0.000 0.000 0.000 0.000
31 -0.0312500 -0.0001116 -0.188 -0.235 -0.800 -1.136
30 -0.0625000 -0.0004504 -0.377 -0.472 -2.279 -3.630
29 -0.0937500 -0.0010281 -0.567 -0.711 -4.473 -7.555
28 -0.1250000 -0.0018650 -0.757 -0.951 -7.442 -13.031
27 -0.1562500 -0.0029903 -0.949 -1.194 -11.272 -20.233
26 -0.1875000 -0.0044433 -1.143 -1.440 -16.083 -29.396
25 -0.2187500 -0.0062746 -1.338 -1.688 -22.025 -40.823
24 -0.2500000 -0.0085478 -1.534 -1.939 -29.287 -54.895
23 -0.2812500 -0.0113419 -1.733 -2.194 -38.107 -72.083
22 -0.3125000 -0.0147533 -1.934 -2.454 -48.773 -92.966
21 -0.3437500 -0.0188997 -2.138 -2.720 -61.637 -118.248
20 -0.3750000 -0.0239231 -2.346 -2.992 -77.122 -148.778
19 -0.4062500 -0.0299948 -2.557 -3.272 -95.741 -185.581
18 -0.4375000 -0.0373204 -2.774 -3.561 -118.110 -229.889
17 -0.4687500 -0.0461459 -2.997 -3.862 -144.963 -283.172
16 -0.5000000 -0.0567646 -3.227 -4.176 -177.177 -347.187
15 -0.5312500 -0.0695249 -3.466 -4.506 -215.794 -424.018
14 -0.5625000 -0.0848395 -3.714 -4.855 -262.047 -516.135
13 -0.5937500 -0.1031943 -3.975 -5.227 -317.391 -626.446
12 -0.6250000 -0.1251587 -4.251 -5.626 -383.526 -758.359
11 -0.6562500 -0.1513952 -4.543 -6.057 -462.434 -915.837
10 -0.6875000 -0.1826680 -4.856 -6.526 -556.399 -1103.456
9 -0.7187500 -0.2198500 -5.192 -7.039 -668.032 -1326.437
8 -0.7500000 -0.2639239 -5.556 -7.604 -800.270 -1590.663
7 -0.7812500 -0.3159778 -5.951 -8.229 -956.366 -1902.645
6 -0.8125000 -0.3771881 -6.384 -8.923 -1139.838 -2269.423
5 -0.8437500 -0.4487876 -6.858 -9.694 -1354.372 -2698.375
4 -0.8750000 -0.5320097 -7.378 -10.553 -1603.654 -3196.882
3 -0.9062500 -0.6280041 -7.950 -11.507 -1891.123 -3771.824
2 -0.9375000 -0.7377147 -8.576 -12.564 -2219.601 -4428.850
1 -0.9687500 -0.8617150 -9.259 -13.728 -2590.801 -5171.391
0 -1.0000000 -1.0000000 -10.000 -15.000 -3004.710 -5999.420

Time Splitting Weights for Grid 01: ndtfast = 72 nfast = 98
==================================

Primary Secondary Accumulated to Current Step

1-0.0002697221589845 0.0138888888888889-0.0002697221589845 0.0138888888888889
2-0.0005179738466200 0.0138926350299859-0.0007876960056045 0.0277815239188748
3-0.0007447551074202 0.0138998291111890-0.0015324511130247 0.0416813530300638
4-0.0009500661144945 0.0139101729321254-0.0024825172275192 0.0555915259621891
5-0.0011339073179268 0.0139233682948267-0.0036164245454460 0.0695148942570158
6-0.0012962796525070 0.0139391170075756-0.0049127041979530 0.0834540112645914
7-0.0014371848048139 0.0139571208916382-0.0063498890027669 0.0974111321562297
8-0.0015566255396494 0.0139770817917051-0.0079065145424162 0.1113882139479348
9-0.0016546060858249 0.0139987015908669-0.0095611206282412 0.1253869155388017
10-0.0017311325812999 0.0140216822309478-0.0112922532095411 0.1394085977697495
11-0.0017862135776710 0.0140457257390214-0.0130784667872121 0.1534543235087709
12-0.0018198606040138 0.0140705342609335-0.0148983273912259 0.1675248577697044
13-0.0018320887900766 0.0140958101026559-0.0167304161813025 0.1816206678723603
14-0.0018229175488246 0.0141212557802959-0.0185533337301271 0.1957419236526562
15-0.0017923713183374 0.0141465740795851-0.0203457050484645 0.2098884977322413
16-0.0017404803630569 0.0141714681256731-0.0220861854115213 0.2240599658579144
17-0.0016672816343881 0.0141956414640489-0.0237534670459094 0.2382556073219634
18-0.0015728196906507 0.0142187981534154-0.0253262867365601 0.2524744054753787
19-0.0014571476763831 0.0142406428713411-0.0267834344129432 0.2667150483467199
20-0.0013203283609977 0.0142608810335131-0.0281037627739409 0.2809759293802330
21-0.0011624352367880 0.0142792189274159-0.0292661980107289 0.2952551483076488
22-0.0009835536762878 0.0142953638612601-0.0302497516870167 0.3095505121689089
23-0.0007837821489820 0.0143090243289864-0.0310335338359988 0.3238595364978953
24-0.0005632334973686 0.0143199101921667-0.0315967673333673 0.3381794466900619
25-0.0003220362723731 0.0143277328796301-0.0319188036057405 0.3525071795696921
26-0.0000603361281147 0.0143322056056353-0.0319791397338552 0.3668393851753274
27 0.0002217027239766 0.0143330436074147-0.0317574370098786 0.3811724287827420
28 0.0005238960016901 0.0143299644029150-0.0312335410081885 0.3955023931856571
29 0.0008460377792135 0.0143226880695582-0.0303875032289750 0.4098250812552152
30 0.0011878988549600 0.0143109375448469-0.0291996043740150 0.4241360188000621
31 0.0015492250600437 0.0142944389496391-0.0276503793139713 0.4384304577497012
32 0.0019297355074029 0.0142729219349163-0.0257206438065684 0.4527033796846174
33 0.0023291207815727 0.0142461200528690-0.0233915230249958 0.4669494997374865
34 0.0027470410691043 0.0142137711531249-0.0206444819558915 0.4811632708906114
35 0.0031831242296341 0.0141756178049429-0.0174613577262575 0.4953388886955543
36 0.0036369638075999 0.0141314077461980-0.0138243939186576 0.5094702964417523
37 0.0041081169846063 0.0140808943599814-0.0097162769340513 0.5235511908017337
38 0.0045961024724372 0.0140238371796396-0.0051201744616140 0.5375750279813734
39 0.0051003983467181 0.0139600024230780-0.0000197761148959 0.5515350304044513
40 0.0056204398212250 0.0138891635571513 0.0056006637063291 0.5654241939616026
41 0.0061556169628429 0.0138111018929677 0.0117562806691720 0.5792352958545702
42 0.0067052723471717 0.0137256072129282 0.0184615530163437 0.5929609030674984
43 0.0072686986547809 0.0136324784303286 0.0257302516711247 0.6065933814978269
44 0.0078451362081123 0.0135315242823455 0.0335753878792370 0.6201249057801724
45 0.0084337704490309 0.0134225640572328 0.0420091583282679 0.6335474698374053
46 0.0090337293570240 0.0133054283565518 0.0510428876852919 0.6468528981939571
47 0.0096440808080493 0.0131799598932598 0.0606869684933412 0.6600328580872169
48 0.0102638298740300 0.0130460143264814 0.0709507983673713 0.6730788724136982
49 0.0108919160629993 0.0129034611337865 0.0818427144303705 0.6859823335474847
50 0.0115272104998922 0.0127521845218004 0.0933699249302627 0.6987345180692851
51 0.0121685130479867 0.0125920843759686 0.1055384379782495 0.7113266024452537
52 0.0128145493709923 0.0124230772503021 0.1183529873492417 0.7237496796955558
53 0.0134639679357865 0.0122450973979272 0.1318169552850283 0.7359947770934830
54 0.0141153369558014 0.0120580978432635 0.1459322922408297 0.7480528749367465
55 0.0147671412750559 0.0118620514966551 0.1606994335158856 0.7599149264334017
56 0.0154177791928386 0.0116569523122794 0.1761172127087242 0.7715718787456811
57 0.0160655592290373 0.0114428164901566 0.1921827719377615 0.7830146952358378
58 0.0167086968301175 0.0112196837230866 0.2088914687678790 0.7942343789589245
59 0.0173453110157492 0.0109876184893350 0.2262367797836282 0.8052219974482595
60 0.0179734209660816 0.0107467113918941 0.2442102007497098 0.8159687088401536
61 0.0185909425496662 0.0104970805451429 0.2628011432993760 0.8264657893852965
62 0.0191956847920284 0.0102388730097309 0.2819968280914044 0.8367046623950274
63 0.0197853462848869 0.0099722662765083 0.3017821743762913 0.8466769286715357
64 0.0203575115360218 0.0096974698003293 0.3221396859123131 0.8563743984718649
65 0.0209096472597907 0.0094147265845512 0.3430493331721038 0.8657891250564161
66 0.0214390986082931 0.0091243148170541 0.3644884317803970 0.8749134398734703
67 0.0219430853431833 0.0088265495586056 0.3864315171235803 0.8837399894320759
68 0.0224186979481308 0.0085217844843947 0.4088502150717111 0.8922617739164705
69 0.0228628936819303 0.0082104136795596 0.4317131087536414 0.9004721875960301
70 0.0232724925722585 0.0078928734895328 0.4549856013258999 0.9083650610855628
71 0.0236441733500801 0.0075696444260292 0.4786297746759800 0.9159347055115920
72 0.0239744693247019 0.0072412531295003 0.5026042440006819 0.9231759586410923
73 0.0242597641994746 0.0069082743888794 0.5268640082001566 0.9300842330299717
74 0.0244962878281439 0.0065713332194423 0.5513602960283005 0.9366555662494139
75 0.0246801119118487 0.0062311069996069 0.5760404079401492 0.9428866732490209
76 0.0248071456367686 0.0058883276674979 0.6008475535769178 0.9487750009165188
77 0.0248731312524188 0.0055437839780984 0.6257206848293365 0.9543187848946172
78 0.0248736395905937 0.0051983238218148 0.6505943244199301 0.9595171087164319
79 0.0248040655249589 0.0048528566052787 0.6753983899448891 0.9643699653217107
80 0.0246596233712912 0.0045083556952099 0.7000580133161803 0.9688783210169206
81 0.0244353422283667 0.0041658609261642 0.7244933555445470 0.9730441819430847
82 0.0241260612594974 0.0038264811729924 0.7486194168040444 0.9768706631160772
83 0.0237264249147164 0.0034913969888327 0.7723458417187609 0.9803620601049099
84 0.0232308780936106 0.0031618633094617 0.7955767198123714 0.9835239234143716
85 0.0226336612488020 0.0028392122248282 0.8182103810611735 0.9863631356391998
86 0.0219288054300778 0.0025248558185948 0.8401391864912513 0.9888879914577946
87 0.0211101272691677 0.0022202890765104 0.8612493137604190 0.9911082805343050
88 0.0201712239051703 0.0019270928644386 0.8814205376655893 0.9930353733987436
89 0.0191054678506273 0.0016469369768668 0.9005260055162165 0.9946823103756104
90 0.0179060017982467 0.0013815832567192 0.9184320073144632 0.9960638936323296
91 0.0165657333682728 0.0011328887872991 0.9349977406827360 0.9971967824196287
92 0.0150773297965061 0.0009028091571842 0.9500750704792420 0.9980995915768129
93 0.0134332125629703 0.0006934017988994 0.9635082830422124 0.9987929933757124
94 0.0116255519612283 0.0005068294021915 0.9751338350034406 0.9992998227779039
95 0.0096462616083458 0.0003453634027300 0.9847800966117864 0.9996451861806338
96 0.0074869928955037 0.0002113875470585 0.9922670895072901 0.9998565737276923
97 0.0051391293792585 0.0001074015346210 0.9974062188865486 0.9999639752623133
98 0.0025937811134509 0.0000360247376868 0.9999999999999996 1.0000000000000000

ndtfast, nfast = 72 98 nfast/ndtfast = 1.36111

Centers of gravity and integrals (values must be 1, 1, approx 1/2, 1, 1):

1.000000000000 1.025151749473 0.512575874736 1.000000000000 1.000000000000

Power filter parameters, Fgamma, gamma = 0.28400 0.24456

Metrics information for Grid 01:
===============================

Minimum X-grid spacing, DXmin = 8.31956155E+00 km
Maximum X-grid spacing, DXmax = 9.23851059E+00 km
Minimum Y-grid spacing, DYmin = 8.32752431E+00 km
Maximum Y-grid spacing, DYmax = 9.23803707E+00 km
Minimum Z-grid spacing, DZmin = 1.87946450E-01 m
Maximum Z-grid spacing, DZmax = 8.28028490E+02 m

Minimum barotropic Courant Number = 7.65400315E-03
Maximum barotropic Courant Number = 2.03630778E-01
Maximum Coriolis Courant Number = 2.30020631E-02


NLM: GET_STATE - Read state initial conditions, t = 834 00:00:00
(Grid 01, File: ssc12_rst.nc, Rec=0001, Index=1)
- free-surface
(Min = -1.68187836E+00 Max = 7.37968147E-01)
- vertically integrated u-momentum component
(Min = -1.18573319E+00 Max = 1.04792744E+00)
- vertically integrated v-momentum component
(Min = -7.39885390E-01 Max = 1.14603496E+00)
- u-momentum component
(Min = -2.14761646E+00 Max = 2.94321512E+00)
- v-momentum component
(Min = -3.85626957E+00 Max = 1.83796010E+00)
- potential temperature
(Min = -1.72992955E+00 Max = 3.28646731E+01)
- salinity
(Min = 3.24795329E+01 Max = 3.50028380E+01)
- vertical viscosity coefficient
(Min = 0.00000000E+00 Max = 3.20311947E+02)
- temperature vertical diffusion coefficient
(Min = 0.00000000E+00 Max = 4.33677575E+02)
- salinity vertical diffusion coefficient
(Min = 0.00000000E+00 Max = 4.33677575E+02)
GET_2DFLD - surface u-momentum stress, t = 105 00:00:00
(Rec=0000004, Index=1, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = -6.20676099E-05 Max = -1.88522345E-06)
GET_2DFLD - surface v-momentum stress, t = 105 00:00:00
(Rec=0000004, Index=1, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = -9.36735938E-05 Max = 2.28887546E-05)
GET_2DFLD - solar shortwave radiation flux, t = 105 00:00:00
(Rec=0000004, Index=1, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 4.67014091E-05 Max = 7.53452299E-05)
GET_2DFLD - surface net heat flux, t = 105 00:00:00
(Rec=0000004, Index=1, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = -1.61152481E-05 Max = 3.52080195E-05)
GET_2DFLD - sea surface temperature climatology, t = 105 00:00:00
(Rec=0000004, Index=1, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 1.78500004E+01 Max = 3.01499996E+01)
GET_2DFLD - surface net heat flux sensitivity to SST, t = 105 00:00:00
(Rec=0000004, Index=1, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = -1.08742870E-05 Max = -5.57647613E-06)
GET_2DFLD - surface net freswater flux, (E-P), t = 105 00:00:00
(Rec=0000004, Index=1, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = -5.87691981E-08 Max = 5.38352896E-08)
GET_NGFLD - free-surface western boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - free-surface eastern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - free-surface southern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - free-surface northern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D u-momentum western boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D v-momentum western boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D u-momentum eastern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D v-momentum eastern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D u-momentum southern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D v-momentum southern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D u-momentum northern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D v-momentum northern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - temperature western boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 2.58318077E+01 Max = 2.91239976E+01)
GET_NGFLD - salinity western boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.31880350E+01 Max = 3.39559378E+01)
GET_NGFLD - temperature eastern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 1.14744140E+00 Max = 2.87537389E+01)
GET_NGFLD - salinity eastern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.39671503E+01 Max = 3.49988728E+01)
GET_NGFLD - temperature southern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 2.19736436E+00 Max = 2.94572413E+01)
GET_NGFLD - salinity southern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.23207594E+01 Max = 3.50003110E+01)
GET_NGFLD - temperature northern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.08338799E+00 Max = 2.33881566E+01)
GET_NGFLD - salinity northern boundary condition, t = 105 00:00:00
(Rec=0000004, Index=2, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.27818864E+01 Max = 3.48521938E+01)
GET_3DFLD - potential temperature climatology, t = 105 00:00:00
(Rec=0000004, Index=1, File: ssc12_clm.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 1.12952750E+00 Max = 2.93347012E+01)
GET_3DFLD - salinity climatology, t = 105 00:00:00
(Rec=0000004, Index=1, File: ssc12_clm.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.23027001E+01 Max = 3.50098615E+01)

Basin information for Grid 01:

Maximum grid stiffness ratios: rx0 = 1.470108E-01 (Beckmann and Haidvogel)
rx1 = 2.393509E+00 (Haney)

Initial basin volumes: TotVolume = 8.9740602851E+15 m3
MinVolume = 1.3390901769E+07 m3
MaxVolume = 7.0378199510E+10 m3
Max/Min = 5.2556728981E+03

NL ROMS/TOMS: started time-stepping: (Grid: 01 TimeSteps: 00200161 - 00424800)

GET_2DFLD - surface u-momentum stress, t = 135 00:00:00
(Rec=0000005, Index=2, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = -3.82850855E-05 Max = 1.04670723E-05)
GET_2DFLD - surface v-momentum stress, t = 135 00:00:00
(Rec=0000005, Index=2, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = -4.57473183E-05 Max = 2.86107910E-05)
GET_2DFLD - solar shortwave radiation flux, t = 135 00:00:00
(Rec=0000005, Index=2, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 5.37974143E-05 Max = 7.05541445E-05)
GET_2DFLD - surface net heat flux, t = 135 00:00:00
(Rec=0000005, Index=2, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 5.89348978E-06 Max = 3.80686309E-05)
GET_2DFLD - sea surface temperature climatology, t = 135 00:00:00
(Rec=0000005, Index=2, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 2.20245279E+01 Max = 3.10499992E+01)
GET_2DFLD - surface net heat flux sensitivity to SST, t = 135 00:00:00
(Rec=0000005, Index=2, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = -1.03150482E-05 Max = -5.28865044E-06)
GET_2DFLD - surface net freswater flux, (E-P), t = 135 00:00:00
(Rec=0000005, Index=2, File: ssc12_frc.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = -8.59949345E-08 Max = 3.07742399E-08)
GET_NGFLD - free-surface western boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - free-surface eastern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - free-surface southern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - free-surface northern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D u-momentum western boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D v-momentum western boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D u-momentum eastern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D v-momentum eastern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D u-momentum southern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D v-momentum southern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D u-momentum northern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - 2D v-momentum northern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 0.00000000E+00 Max = 0.00000000E+00)
GET_NGFLD - temperature western boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 2.42429398E+01 Max = 3.00986811E+01)
GET_NGFLD - salinity western boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.28348526E+01 Max = 3.40548760E+01)
GET_NGFLD - temperature eastern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 1.14744140E+00 Max = 2.92940963E+01)
GET_NGFLD - salinity eastern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.36917025E+01 Max = 3.50467227E+01)
GET_NGFLD - temperature southern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 2.19736436E+00 Max = 3.01042090E+01)
GET_NGFLD - salinity southern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.28487554E+01 Max = 3.50486625E+01)
GET_NGFLD - temperature northern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.08338799E+00 Max = 2.51038226E+01)
GET_NGFLD - salinity northern boundary condition, t = 135 00:00:00
(Rec=0000005, Index=1, File: ssc12_bry.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.37117703E+01 Max = 3.48547595E+01)
GET_3DFLD - potential temperature climatology, t = 135 00:00:00
(Rec=0000005, Index=2, File: ssc12_clm.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 1.12952750E+00 Max = 3.01322845E+01)
GET_3DFLD - salinity climatology, t = 135 00:00:00
(Rec=0000005, Index=2, File: ssc12_clm.nc)
(Tmin= 15.0000 Tmax= 345.0000)
(Min = 3.27405017E+01 Max = 3.50487220E+01)

STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
C => (i,j,k) Cu Cv Cw Max Speed

200160 834 00:00:00 2.709243E-02 1.908407E+04 1.908410E+04 9.009309E+15
(275,250,27) 1.116631E-01 1.106118E-01 0.000000E+00 3.738429E+00
DEF_HIS - inquiring history file, Grid 01: ssc12_his_0014.nc
DEF_AVG - inquiring average file, Grid 01: ssc12_avg_0000.nc
200161 834 00:06:00 2.709387E-02 1.908407E+04 1.908410E+04 9.009309E+15
(274,250,26) 8.662358E-02 9.662954E-02 2.610078E-01 3.738679E+00

(omit)*********(omit)*****

205303 855 10:18:00 3.526154E-02 1.908405E+04 1.908408E+04 9.009412E+15
(275,253,12) 2.359342E-02 4.247913E-02 5.696632E-01 2.826756E+00
205304 855 10:24:00 3.527035E-02 1.908405E+04 1.908408E+04 9.009412E+15
(275,253,12) 2.475272E-02 4.158159E-02 5.812245E-01 2.823614E+00
205305 855 10:30:00 3.527900E-02 1.908405E+04 1.908408E+04 9.009412E+15
(275,253,12) 2.591119E-02 4.067038E-02 5.919800E-01 2.820217E+00
205306 855 10:36:00 3.528818E-02 1.908405E+04 1.908408E+04 9.009412E+15
(275,253,12) 2.706134E-02 3.970224E-02 6.024620E-01 2.816664E+00
205307 855 10:42:00 3.529777E-02 1.908405E+04 1.908408E+04 9.009411E+15
(275,253,12) 2.819525E-02 3.865272E-02 6.125581E-01 2.817170E+00
205308 855 10:48:00 3.530754E-02 1.908405E+04 1.908408E+04 9.009411E+15
(275,253,12) 2.927960E-02 3.753774E-02 6.226051E-01 2.826378E+00
205309 855 10:54:00 3.531776E-02 1.908405E+04 1.908408E+04 9.009411E+15
(275,253,12) 2.995494E-02 3.636968E-02 6.325719E-01 2.835087E+00
205310 855 11:00:00 3.532825E-02 1.908405E+04 1.908408E+04 9.009411E+15
(275,253,12) 3.099315E-02 3.516521E-02 6.422409E-01 2.843332E+00
205311 855 11:06:00 3.533900E-02 1.908405E+04 1.908408E+04 9.009410E+15
(275,253,12) 3.203498E-02 3.392989E-02 6.524234E-01 2.851146E+00
205312 855 11:12:00 3.534990E-02 1.908405E+04 1.908409E+04 9.009410E+15
(275,253,12) 3.305723E-02 3.272617E-02 6.627129E-01 2.858856E+00
205313 855 11:18:00 3.536101E-02 1.908405E+04 1.908409E+04 9.009410E+15
(275,253,12) 3.405883E-02 3.158176E-02 6.730664E-01 2.866121E+00
205314 855 11:24:00 3.537232E-02 1.908405E+04 1.908409E+04 9.009410E+15
(275,253,12) 3.503133E-02 3.045003E-02 6.835305E-01 2.873413E+00
205315 855 11:30:00 3.538395E-02 1.908405E+04 1.908409E+04 9.009409E+15
(275,253,12) 3.595725E-02 2.926428E-02 6.942394E-01 2.880557E+00
205316 855 11:36:00 3.539564E-02 1.908405E+04 1.908409E+04 9.009409E+15
(275,253,12) 3.683159E-02 2.799544E-02 7.049749E-01 2.889096E+00
205317 855 11:42:00 3.540743E-02 1.908405E+04 1.908409E+04 9.009409E+15
(275,253,12) 3.763728E-02 2.665123E-02 7.154729E-01 2.915840E+00
205318 855 11:48:00 3.541921E-02 1.908405E+04 1.908409E+04 9.009409E+15
(275,253,12) 3.837849E-02 2.526051E-02 7.255248E-01 2.942315E+00
205319 855 11:54:00 3.543090E-02 1.908405E+04 1.908409E+04 9.009408E+15
(275,253,12) 3.903485E-02 2.383714E-02 7.352442E-01 2.968751E+00
205320 855 12:00:00 3.544240E-02 1.908405E+04 1.908409E+04 9.009408E+15
(275,253,12) 3.959201E-02 2.245813E-02 7.445034E-01 2.994961E+00
205321 855 12:06:00 3.545381E-02 1.908405E+04 1.908409E+04 9.009408E+15
(275,253,12) 4.006172E-02 2.117694E-02 7.530285E-01 3.020709E+00
205322 855 12:12:00 3.546518E-02 1.908405E+04 1.908409E+04 9.009408E+15
(275,253,12) 4.041660E-02 1.997603E-02 7.608261E-01 3.046144E+00
205323 855 12:18:00 3.547637E-02 1.908405E+04 1.908409E+04 9.009408E+15
(275,253,13) 4.348721E-02 1.939974E-02 7.636952E-01 3.071278E+00
205324 855 12:24:00 3.548745E-02 1.908405E+04 1.908409E+04 9.009407E+15
(275,253,13) 4.360021E-02 1.811768E-02 7.705619E-01 3.095744E+00
205325 855 12:30:00 3.549807E-02 1.908405E+04 1.908409E+04 9.009407E+15
(275,253,13) 4.368998E-02 1.697693E-02 7.760241E-01 3.119359E+00
205326 855 12:36:00 3.550857E-02 1.908405E+04 1.908409E+04 9.009407E+15
(275,253,13) 4.373112E-02 1.595639E-02 7.796340E-01 3.142272E+00
205327 855 12:42:00 3.551871E-02 1.908405E+04 1.908409E+04 9.009407E+15
(275,253,13) 4.376087E-02 1.508760E-02 7.820757E-01 3.164540E+00
205328 855 12:48:00 3.552841E-02 1.908405E+04 1.908409E+04 9.009406E+15
(275,253,13) 4.377394E-02 1.438975E-02 7.830106E-01 3.186193E+00
205329 855 12:54:00 3.553786E-02 1.908405E+04 1.908409E+04 9.009406E+15
(275,253,13) 4.373087E-02 1.380425E-02 7.824235E-01 3.207101E+00
205330 855 13:00:00 3.554692E-02 1.908405E+04 1.908409E+04 9.009406E+15
(275,253,13) 4.363620E-02 1.326950E-02 7.805699E-01 3.227246E+00
205331 855 13:06:00 3.555572E-02 1.908405E+04 1.908409E+04 9.009406E+15
(275,253,13) 4.347621E-02 1.279885E-02 7.771226E-01 3.246639E+00
205332 855 13:12:00 3.556413E-02 1.908405E+04 1.908409E+04 9.009405E+15
(275,251,11) 8.687961E-03 5.868544E-02 7.606430E-01 3.265299E+00
205333 855 13:18:00 3.557216E-02 1.908405E+04 1.908409E+04 9.009405E+15
(275,251,11) 7.655248E-03 5.796280E-02 7.986381E-01 3.283115E+00
205334 855 13:24:00 3.557991E-02 1.908405E+04 1.908409E+04 9.009405E+15
(275,251,12) 7.625260E-03 5.806786E-02 8.368371E-01 3.299912E+00
205335 855 13:30:00 3.558742E-02 1.908405E+04 1.908409E+04 9.009405E+15
(275,251,12) 6.596264E-03 5.734744E-02 8.772965E-01 3.315893E+00
205336 855 13:36:00 3.559463E-02 1.908405E+04 1.908409E+04 9.009405E+15
(275,251,12) 5.537195E-03 5.650501E-02 9.163505E-01 3.331017E+00
205337 855 13:42:00 3.560156E-02 1.908405E+04 1.908409E+04 9.009404E+15
(275,251,12) 4.468625E-03 5.553002E-02 9.529627E-01 3.345418E+00
205338 855 13:48:00 3.560823E-02 1.908405E+04 1.908409E+04 9.009404E+15
(275,251,13) 3.995803E-03 5.505565E-02 9.875739E-01 3.360165E+00
205339 855 13:54:00 3.561458E-02 1.908405E+04 1.908409E+04 9.009404E+15
(275,251,13) 3.121549E-03 5.403858E-02 1.024790E+00 3.375139E+00
205340 855 14:00:00 3.562080E-02 1.908405E+04 1.908409E+04 9.009404E+15
(275,251,13) 2.427986E-03 5.308032E-02 1.060749E+00 3.389437E+00
205341 855 14:06:00 3.562679E-02 1.908405E+04 1.908409E+04 9.009403E+15
(275,251,13) 1.558878E-03 5.220892E-02 1.093671E+00 3.403209E+00
205342 855 14:12:00 3.563262E-02 1.908405E+04 1.908409E+04 9.009403E+15
(275,251,14) 9.068599E-04 5.085986E-02 1.123449E+00 3.417124E+00
205343 855 14:18:00 3.563828E-02 1.908405E+04 1.908409E+04 9.009403E+15
(275,251,14) 6.136644E-04 4.993357E-02 1.155907E+00 3.431265E+00
205344 855 14:24:00 3.564350E-02 1.908405E+04 1.908409E+04 9.009403E+15
(275,251,14) 8.381187E-05 4.897187E-02 1.184329E+00 3.445476E+00
205345 855 14:30:00 3.564900E-02 1.908405E+04 1.908409E+04 9.009402E+15
(275,251,14) 1.176028E-03 4.829006E-02 1.205966E+00 3.460747E+00
205346 855 14:36:00 3.565424E-02 1.908405E+04 1.908409E+04 9.009402E+15
(275,251,15) 3.141366E-03 4.644630E-02 1.233495E+00 3.475221E+00
205347 855 14:42:00 3.565942E-02 1.908406E+04 1.908409E+04 9.009402E+15
(275,251,15) 2.754590E-03 4.529407E-02 1.254921E+00 3.490049E+00
205348 855 14:48:00 3.566471E-02 1.908406E+04 1.908409E+04 9.009402E+15
(275,251,15) 3.662880E-03 4.342926E-02 1.270466E+00 3.507288E+00
205349 855 14:54:00 3.567121E-02 1.908405E+04 1.908409E+04 9.009401E+15
(275,251,16) 5.686473E-03 4.105372E-02 1.295611E+00 3.530932E+00
205350 855 15:00:00 3.568281E-02 1.908405E+04 1.908409E+04 9.009401E+15
(275,251,16) 9.154510E-03 3.918325E-02 1.337819E+00 3.707565E+00
205351 855 15:06:00 3.572494E-02 1.908405E+04 1.908409E+04 9.009400E+15
(275,251,17) 2.925438E-02 3.822207E-02 1.486118E+00 4.974890E+00
205352 855 15:12:00 3.620864E-02 2.626193E+04 2.626196E+04 9.009397E+15
(275,251,17) 1.398685E-01 3.509188E-02 2.106421E+00 1.220272E+01

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=1,1) into time record = 0000003

Elapsed CPU time (seconds):

Node # 0 CPU: 1878.147
Node # 3 CPU: 1895.773
Node # 1 CPU: 1895.909
Node # 7 CPU: 1896.513
Node # 2 CPU: 1896.583
Node # 4 CPU: 1896.520
Node # 5 CPU: 1896.442
Node # 6 CPU: 1896.219
Node # 9 CPU: 1896.308
Node # 10 CPU: 1896.471
Node # 12 CPU: 1896.603
Node # 13 CPU: 1896.665
Node # 14 CPU: 1896.622
Total: 30322.101
Node # 15 CPU: 1896.529

Nonlinear model elapsed time profile, Grid: 01

Allocation and array initialization .............. 1.099 ( 0.0036 %)
Node # 8 CPU: 1894.255
Ocean state initialization ....................... 0.299 ( 0.0010 %)
Reading of input data ............................ 18.717 ( 0.0617 %)
Processing of input data ......................... 531.208 ( 1.7519 %)
Processing of output time averaged data .......... 0.400 ( 0.0013 %)
Computation of vertical boundary conditions ...... 19.930 ( 0.0657 %)
Computation of global information integrals ...... 371.286 ( 1.2245 %)
Writing of output data ........................... 336.025 ( 1.1082 %)
Model 2D kernel .................................. 14348.311 (47.3196 %)
2D/3D coupling, vertical metrics ................. 788.128 ( 2.5992 %)
Omega vertical velocity .......................... 493.266 ( 1.6268 %)
Equation of state for seawater ................... 660.155 ( 2.1771 %)
My2.5 vertical mixing parameterization ........... 3560.626 (11.7427 %)
Node # 11 CPU: 1896.546
3D equations right-side terms .................... 760.302 ( 2.5074 %)
3D equations predictor step ...................... 1985.028 ( 6.5465 %)
Pressure gradient ................................ 890.511 ( 2.9368 %)
Harmonic mixing of tracers, geopotentials ........ 856.133 ( 2.8235 %)
Harmonic stress tensor, geopotentials ............ 1612.028 ( 5.3163 %)
Corrector time-step for 3D momentum .............. 1045.567 ( 3.4482 %)
Corrector time-step for tracers .................. 1509.167 ( 4.9771 %)
Total: 29788.185 98.2392

Nonlinear model message Passage profile, Grid: 01

Message Passage: 2D halo exchanges ............... 1790.132 ( 5.9037 %)
Message Passage: 3D halo exchanges ............... 817.576 ( 2.6963 %)
Message Passage: 4D halo exchanges ............... 200.821 ( 0.6623 %)
Message Passage: data broadcast .................. 329.280 ( 1.0859 %)
Message Passage: data reduction .................. 21.304 ( 0.0703 %)
Message Passage: data gathering .................. 34.376 ( 0.1134 %)
Message Passage: data scattering.................. 11.133 ( 0.0367 %)
Total: 3204.622 10.5686

All percentages are with respect to total time = 30322.101


ROMS/TOMS - Output NetCDF summary for Grid 01:
number of time records written in HISTORY file = 00000003
number of time records written in RESTART file = 00000008

Analytical header files used:

/vol/home/student/lshw/roms/MYPROJECT/Functionals/ana_btflux.h
/vol/home/student/lshw/roms/MYPROJECT/Functionals/ana_sponge.h
ROMS/Functionals/ana_nudgcoef.h

ROMS/TOMS: DONE... Sunday - June 12, 2016 - 1:04:46 PM
Attachments
figure 2 is the leve 30 of 'Akv',its abnormal value don't at (245,276)
figure 2 is the leve 30 of 'Akv',its abnormal value don't at (245,276)
figure 1 is the level 30 of temp ,I can find that there is a low temperature domain at right boundary.
figure 1 is the level 30 of temp ,I can find that there is a low temperature domain at right boundary.

User avatar
kate
Posts: 4091
Joined: Wed Jul 02, 2003 5:29 pm
Location: CFOS/UAF, USA

Re: Blowing-up: Saving latest model state into RESTART file

#11 Unread post by kate »

Hello! I have a problem when run a spinup with climatology data.
I have run 855 days without error but blowup suddenly. I see the restart file,it has 11 variables 'zeta' 'u' 'v' 'ubar' 'vbar' 'temp' 'salt' 'rho' 'akv' 'akt' 'aks',and I found that there is a abnormal value at right boundary with coordinate(245,276). so I correct the value with its near value. But I just have corrected 9 variables without 'Akt''Akv''Aks',because I don't know how to correct them. And after I correct 9 variable of the restart file. I try to run with restart mode, but failed. What should I do ?
That's not quite how it works. If the temperature is going bad at the edge, you need to think about what is happening in the model at the edge. You can't just correct it in the restart file because if the model wants to go to zero T, it will still do that when you start timestepping again.

As for Akt and friends, they are generated from the other model fields internally. They are allowed to be spiky like that, with large values where things should mix vertically.

Code: Select all

Lateral Boundary Conditions: NLM
============================

Variable Grid West Edge South Edge East Edge North Edge
--------- ---- ---------- ---------- ---------- ----------

zeta 1 Chapman Imp Chapman Imp Chapman Imp Chapman Imp

ubar 1 Flather Flather Flather Flather

vbar 1 Flather Flather Flather Flather

u 1 Radiation Radiation Radiation Radiation

v 1 Radiation Radiation Radiation Radiation

temp 1 Clamped Clamped Clamped Clamped

salt 1 Clamped Clamped Clamped Clamped
You are clamping to monthly climatology for T and S? I prefer RadNud from something like daily HYCOM fields. Same for u,v.

lindandan
Posts: 17
Joined: Tue Nov 03, 2015 12:52 am
Location: School of Space Science of USTC

Re: Blowing-up: Saving latest model state into RESTART file

#12 Unread post by lindandan »

kate wrote:
You are clamping to monthly climatology for T and S? I prefer RadNud from something like daily HYCOM fields. Same for u,v.
sorry applying so late.I run successful with your suggestion.Thank you very much.

Post Reply