Model crashing at a certain baroclinic time step

General scientific issues regarding ROMS

Moderators: arango, robertson

Post Reply
Message
Author
auhgnist
Posts: 4
Joined: Sun Apr 05, 2015 3:35 pm
Location: Tsinghua University

Model crashing at a certain baroclinic time step

#1 Unread post by auhgnist »

I have encountered constant crashing of a ROMS-CICE-WRF coupled model. Interestingly, the model crashes at exactly 10 baroclinic (3D) time steps, no matter how long the time step size is (of coz, within the CFL bound). The time step varies from 240s to 1200s, 1800s and 3600s (with the barotropic step set to 40s for 3600s baroclinic time step to ensure stability).
From 1st to 9th step, the model seems to run without any problem, but on the 10th step, the potential energy witnessed a burst to NaN, but there is no hint what went wrong before the last step (shown in the logs).
I do have WET_DRY enabled in the macros (shown below).
The zeta is obviously too big in the negative direction (-199 m) at certain continental boundaries, but I don't know what has caused it.
The macros and the log is shown below.
Thanks for any suggestions of what might be going wrong with it :)

Macros:

Code: Select all

#define ROMS_MODEL
#undef SWAN_MODEL
#define WRF_MODEL

#define MCT_LIB
# undef MCT_INTERP_WV2AT
# undef MCT_INTERP_OC2AT
# undef MCT_INTERP_OC2WV

#define PERFECT_RESTART

/* Model configuration */
#define SOLVE3D
#define CURVGRID
#define MASKING
#define SPLINES

/* momentum equations */
#define UV_ADV
#define UV_COR
#define UV_VIS2
#define UV_QDRAG

/* tracers equations */
#define TS_U3HADVECTION
#define TS_C4VADVECTION
#define TS_DIF2
#define SALINITY
#define NONLIN_EOS
#define SOLAR_SOURCE

/* pressure gradient algorithm */
#define DJ_GRADPS
#define ATM_PRESS /* use real SLP in  pressure gradient equations */

/* Forcing */
# undef ANA_PSOURCE
# undef ATM2OCN_FLUXES /* surface fluxes from ATM model */
#define ANA_SSFLUX
#define EMINUSP        /* use evap & rain from ATM model */
#define ANA_BTFLUX
#define ANA_BSFLUX
# undef CORE_FORCING
#define BULK_FLUXES
# undef CCSM_FLUXES

/* wave roughness formulation
 * ** if only in MYJSFC and MYNN now
 * */
#undef COARE_TAYLOR_YELLAND
#undef DEEPWATER_WAVES

/* horizontal mixing */
#define VISC_3DCOEF
#define MIX_S_UV
#define VISC_GRID
#define MIX_GEO_TS
#define DIFF_GRID

/* Turbulence closure */
#define GLS_MIXING

/* Sea ice */
# undef  ICE_MODEL
#define CICE_MODEL

/* bottom boundary layer closure */
# undef SSW_BBL

#define RADIATION_2D

/* Wave effort on currents and shallow water */
#define WET_DRY
# undef NEARSHORE_MELLOR /* Nearshore wave driven (radiation stress) momentum terms */


/*** SUB-CONFIGURATIONS ***/

/* Forcing */
#if defined BULK_FLUXES || defined CCSM_FLUXES
#  define SHORTWAVE
#  define LONGWAVE_OUT
#   undef LONGWAVE
#   undef LONGWAVE_ICE
#   undef DIURNAL_SRFLUX
#  define SOLAR_SOURCE
#  define EMINUSP
#   undef ANA_SRFLUX
#  define ALBEDO
#  define ALBEDO_CURVE  /* for water */
#   undef ICE_ALB_EC92  /* for ice */
#   undef ALBEDO_CSIM   /* for ice */
#   undef ALBEDO_FILE  /* for both */
#  if defined ICE_MODEL || defined CICE_MODEL
#    define ALBEDO_ICE
#  endif
#endif

/* Turbulence closure */
#ifdef GLS_MIXING
#  define AKLIMIT
#  define CHARNOK
#  define CRAIG_BANNER
#  define KANTHA_CLAYSON
#  define N2S2_HORAVG
#   undef ZOS_HSIG
#   undef TKE_WAVEDISS
#endif

/* Sea ice */
#ifdef ICE_MODEL
#   undef OUTFLOW_MASK
#   undef FASTICE_CLIMATOLOGY
#  define ICE_THERMO
#  define ICE_MK
#   undef ICE_SMOOTH
#  define ICE_MOMENTUM
#  define ICE_MOM_BULK
#  define ICE_EVP
#  define ICE_ADVECT
#  define ICE_SMOLAR
#  define ICE_UPWIND
#  define ICE_BULK_FLUXES
#   undef ANA_AIOBC
#   undef ANA_HIOBC
#   undef ANA_HSNOBC
#endif

#ifdef CICE_MODEL
#  define ICE_BULK_FLUXES
#endif

/* bottom boundary layer closure */
#ifdef SSW_BBL
#  define SSW_CALC_ZNOT
#  define SSW_LOGINT
#  define SSW_CALC_UB
#  define SSW_ZORIP
#endif
Zeta in the final dumped snapshot (restart) file/state is attached.

(Part of) log file:

Code: Select all

 NL ROMS/TOMS: started time-stepping: (Grid: 01 TimeSteps: 00000001 - 00004320)

    GET_NGFLD   - free-surface western boundary condition,   t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -2.71132774E-01 Max =  1.76061660E-01)
    GET_NGFLD   - free-surface southern boundary condition,  t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -5.81257143E-01 Max = -1.21927726E-01)
    GET_NGFLD   - free-surface northern boundary condition,  t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -1.54453252E-01 Max =  2.08928368E-01)
    GET_NGFLD   - 2D u-momentum western boundary condition,  t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -3.91126597E-02 Max =  6.28961389E-02)
    GET_NGFLD   - 2D v-momentum western boundary condition,  t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -8.56973438E-02 Max =  6.85699264E-02)
    GET_NGFLD   - 2D u-momentum southern boundary condition, t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -2.82746953E-02 Max =  5.11969330E-02)
    GET_NGFLD   - 2D v-momentum southern boundary condition, t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -4.25098229E-02 Max =  6.45392620E-02)
    GET_NGFLD   - 2D u-momentum northern boundary condition, t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -2.06448997E-02 Max =  6.87771481E-02)
    GET_NGFLD   - 2D v-momentum northern boundary condition, t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -1.81647883E-02 Max =  6.91692063E-02)
    GET_NGFLD   - 3D u-momentum western boundary condition,  t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -2.67139584E-01 Max =  3.26596573E-01)
    GET_NGFLD   - 3D v-momentum western boundary condition,  t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -1.46086866E-01 Max =  3.99452947E-01)
    GET_NGFLD   - 3D u-momentum southern boundary condition, t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -2.66644593E-01 Max =  5.03047983E-01)
    GET_NGFLD   - 3D v-momentum southern boundary condition, t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -3.61565101E-01 Max =  3.73100026E-01)
    GET_NGFLD   - 3D u-momentum northern boundary condition, t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -3.14866441E-01 Max =  3.23681000E-01)
    GET_NGFLD   - 3D v-momentum northern boundary condition, t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -2.09041481E-01 Max =  2.32470000E-01)
    GET_NGFLD   - temperature western boundary condition,    t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min =  7.46083388E-01 Max =  2.83218757E+01)
    GET_NGFLD   - salinity western boundary condition,       t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min =  2.72821343E+01 Max =  3.46800718E+01)
    GET_NGFLD   - temperature southern boundary condition,   t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min =  6.17114711E-01 Max =  2.67651677E+01)
    GET_NGFLD   - salinity southern boundary condition,      t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min =  2.98698429E+01 Max =  3.62966581E+01)
    GET_NGFLD   - temperature northern boundary condition,   t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min = -1.19242220E+00 Max =  2.24956871E+01)
    GET_NGFLD   - salinity northern boundary condition,      t =     1 00:00:00
                   (Rec=0000001, Index=1, File: NWP_bry.nc)
                   (Tmin=          1.0000 Tmax=          5.0000)
                   (Min =  3.00333259E+01 Max =  3.46910734E+01)

   STEP   Day HH:MM:SS  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME
          C => (i,j,k)       Cu            Cv            Cw         Max Speed
      0     0 00:00:00  2.696506E-04  1.731806E+04  1.731806E+04  4.967722E+16
          (163,023,40)  7.315241E-02  4.513867E-02  0.000000E+00  6.757293E-01
      DEF_HIS   - creating history file, Grid 01: ocean_joe_tc_coarse_his.nc
      WRT_HIS   - wrote history  fields (Index=1,1) into time record = 0000001
      1     0 01:00:00  6.220933E-04  1.747035E+04  1.747035E+04  4.924581E+16
          (208,036,40)  3.733053E-01  5.822740E-01  5.579508E-04  5.333663E+00
 ## ROMS grid  1 recv data from WRF grid  1
 ## ROMS grid  1  sent data to WRF grid  1
      WRT_HIS   - wrote history  fields (Index=2,2) into time record = 0000002
      2     0 02:00:00  9.446803E-04  1.747077E+04  1.747077E+04  4.924603E+16
          (161,156,40)  2.163557E-03  2.057769E-02  1.055207E+01  2.589992E+00
 ## ROMS grid  1 recv data from WRF grid  1
 ## ROMS grid  1  sent data to WRF grid  1
      WRT_HIS   - wrote history  fields (Index=1,1) into time record = 0000003
      3     0 03:00:00  1.177476E-03  1.747094E+04  1.747094E+04  4.924619E+16
          (054,154,40)  3.322705E-03  1.532365E-02  1.372957E+01  2.567867E+00
 ## ROMS grid  1 recv data from WRF grid  1
 ## ROMS grid  1  sent data to WRF grid  1
      WRT_HIS   - wrote history  fields (Index=2,2) into time record = 0000004
      4     0 04:00:00  1.374695E-03  1.747115E+04  1.747115E+04  4.924631E+16
          (122,122,40)  1.712617E-02  8.448593E-03  1.790937E+01  2.451760E+00
 ## ROMS grid  1 recv data from WRF grid  1
 ## ROMS grid  1  sent data to WRF grid  1
      WRT_HIS   - wrote history  fields (Index=1,1) into time record = 0000005
      5     0 05:00:00  1.547551E-03  1.747128E+04  1.747128E+04  4.924643E+16
          (166,134,40)  2.010454E-02  8.998871E-03  1.833376E+01  1.946859E+00
 ## ROMS grid  1 recv data from WRF grid  1
 ## ROMS grid  1  sent data to WRF grid  1
      WRT_HIS   - wrote history  fields (Index=2,2) into time record = 0000006
      6     0 06:00:00  1.645237E-03  1.747125E+04  1.747125E+04  4.924651E+16
          (107,127,40)  1.132731E-02  2.080250E-02  1.839183E+01  1.817681E+00
 ## ROMS grid  1 recv data from WRF grid  1
 ## ROMS grid  1  sent data to WRF grid  1
      WRT_HIS   - wrote history  fields (Index=1,1) into time record = 0000007
      7     0 07:00:00  1.669672E-03  1.747117E+04  1.747117E+04  4.924653E+16
          (166,134,40)  6.423948E-04  1.342048E-02  2.266929E+01  1.828100E+00
 ## ROMS grid  1 recv data from WRF grid  1
 ## ROMS grid  1  sent data to WRF grid  1
      WRT_HIS   - wrote history  fields (Index=2,2) into time record = 0000008
      8     0 08:00:00  1.684789E-03  1.747108E+04  1.747108E+04  4.924654E+16
          (107,127,40)  6.173031E-03  2.756786E-02  1.689588E+01  2.685525E+00
 ## ROMS grid  1 recv data from WRF grid  1
 ## ROMS grid  1  sent data to WRF grid  1
      WRT_HIS   - wrote history  fields (Index=1,1) into time record = 0000009
      9     0 09:00:00  1.688831E-03  1.747094E+04  1.747094E+04  4.924664E+16
          (107,127,40)  2.407506E-03  1.043648E-02  1.595644E+01  3.744220E+00
 ## ROMS grid  1 recv data from WRF grid  1
 ## ROMS grid  1  sent data to WRF grid  1
      WRT_HIS   - wrote history  fields (Index=2,2) into time record = 0000010
     10     0 10:00:00  1.681953E-03           NaN           NaN  4.924684E+16
          (150,201,40)  1.269356E-01  1.950973E-02  1.188577E+01  3.961068E+00

 Blowing-up: Saving latest model state into  RESTART file

      WRT_RST   - wrote re-start fields (Index=1,1) into time record = 0000001

 Elapsed CPU time (seconds):

 Node   #  0 CPU:     302.568

 ROMS/TOMS - Output NetCDF summary for Grid 01:
             number of time records written in HISTORY file = 00000010
             number of time records written in RESTART file = 00000001

 Analytical header files used:

     ROMS/Functionals/ana_btflux.h
     ROMS/Functionals/ana_stflux.h
Attachments
Zeta in the final snapshot. Note that the large negative values on the continental boundaries.
Zeta in the final snapshot. Note that the large negative values on the continental boundaries.
Top
User avatar
susonic
Posts: 170
Joined: Tue Aug 21, 2007 5:44 pm
Location: UST21 / Korea
Contact:
Contact susonic

Re: Model crashing at a certain baroclinic time step

#2 Unread post by susonic »

What is your minimum depth?
What if you run the model without WET_DRY?
Joonho Lee
Top
User avatar
kate
Posts: 4091
Joined: Wed Jul 02, 2003 5:29 pm
Location: CFOS/UAF, USA

Re: Model crashing at a certain baroclinic time step

#3 Unread post by kate »

This is definitely uncharted territory with running all three models together.

The Budgell ice knows about WET_DRY and CICE does not. I can't see that WET_DRY makes sense at that resolution anyway.
Top
User avatar
kate
Posts: 4091
Joined: Wed Jul 02, 2003 5:29 pm
Location: CFOS/UAF, USA

Re: Model crashing at a certain baroclinic time step

#4 Unread post by kate »

Another thing:

Code: Select all

#ifdef CICE_MODEL
#  define ICE_BULK_FLUXES
#endif
This option is for the Budgell ice only. CICE wants to compute its own bulk fluxes.

Did you try running this with the Budgell ice?
Top
Post Reply

Return to “ROMS Discussion”