starting time is greater than the current model time

[wzhu]

Hi everyone,

When I tried to run the model, I got the error message:

Code: Select all

 GET_CYCLE - starting time for variable: river_time
             is greater than current model time.
             TMIN =          1.0000  TDAYS =     -24855.1348

my river_time is from 1 to 1036, the interval is 15 minutes.

Any reply will be highly appreciated!

Best

[kate]

You have:

Code: Select all

TDAYS =     -24855.1348

What is the time value in your initial conditions file? What is your DSTART and what is your TIME_REF? Do you have the time results from GET_STATE? Here is an example of some times:

Code: Select all

  40907.000  dstart          Time-stamp assigned to model initialization (days).
19000101.00  time_ref        Reference time for units attribute (yyyymmdd.dd)

NLM: GET_STATE - Read state initial conditions,             t = 41059 00:00:00

    GET_2DFLD   - surface u-wind component,                  t = 40906 22:30:00
                   (Rec=0002920, Index=1, File: MERRA_Uwind_3hours_2011.nc)
                   (Tmin=      40542.0625 Tmax=      40906.9375)
                   (Min = -8.09676074E+00 Max =  4.24211923E+00)

ETA: Oops, that's an example of what to watch for. The second reading of winds gives:

Code: Select all

   GET_2DFLD   - surface u-wind component,                  t = 41059 01:30:00
                   (Rec=0001217, Index=2, File: MERRA_Uwind_3hours_2012.nc)
                   (Tmin=      40907.0625 Tmax=      41272.9375)
                   (Min = -8.93241138E+00 Max =  5.45170178E+00)

already 1217 records into the second winds file.

[wzhu]

You have:

Code: Select all

TDAYS =     -24855.1348

What is the time value in your initial conditions file? What is your DSTART and what is your TIME_REF? Do you have the time results from GET_STATE? Here is an example of some times:

Code: Select all

  40907.000  dstart          Time-stamp assigned to model initialization (days).
19000101.00  time_ref        Reference time for units attribute (yyyymmdd.dd)

NLM: GET_STATE - Read state initial conditions,             t = 41059 00:00:00

    GET_2DFLD   - surface u-wind component,                  t = 40906 22:30:00
                   (Rec=0002920, Index=1, File: MERRA_Uwind_3hours_2011.nc)
                   (Tmin=      40542.0625 Tmax=      40906.9375)
                   (Min = -8.09676074E+00 Max =  4.24211923E+00)

ETA: Oops, that's an example of what to watch for. The second reading of winds gives:

Code: Select all

   GET_2DFLD   - surface u-wind component,                  t = 41059 01:30:00
                   (Rec=0001217, Index=2, File: MERRA_Uwind_3hours_2012.nc)
                   (Tmin=      40907.0625 Tmax=      41272.9375)
                   (Min = -8.93241138E+00 Max =  5.45170178E+00)

already 1217 records into the second winds file.

Thank you, Kate.
the Dstart and time_ref in my log is :

Code: Select all

 0.000  dstart          Time-stamp assigned to model initialization (days).
       0.00  time_ref        Reference time for units attribute (yyyymmdd.dd)
 NLM: GET_STATE - Read state initial conditions,             t = ***** **:**:**

what I want to simulate is the time period from 1/1/2006 to 10/31/2008(the field observation is from 2/1/2007 to 10/31/2008). I plan to compute the period from 1/1/2006 to 1/31/2007 as the hindcast simulation.
Do I need to set the starting time of river_time, frc_time, zeta_time,...from 1/1/2006 or 2/1/2007?
What should I set the ocean_time in the initial input file? 0 or 34214400 (from 1/1/2006 to 1/31/2007)?

[kate]

All that matters is that you are completely internally consistent. Pick a time origin and stick with it. Yours can be 1/1/2006 if you want. If you use that as your TIME_REF, then ROMS will know the dates. Also, check your units. Is the time in your initial file in days or seconds? What is the units attribute for your time variable in that file?

[wzhu]

Thank you very much, Kate.

I have taken the 1/1/2006 00:00:00 as the starting time for all the time parameters. The units of ocean_time in my initial file is second, the long name is 'time since initialization' and I set it to 0, the model seemed work, but blew up after one day more time with unclear reason, just ' Blowing-up: Saving latest model state into RESTART file'. How should I find out the reason?
And the Dstart and Time_ref are still 0:

Code: Select all

   0.000  dstart          Time-stamp assigned to model initialization (days).
    0.00  time_ref        Reference time for units attribute (yyyymmdd.dd)

How can I set the Time_ref to 20060101.00?

[kate]

Both DSTART and TIME_REF are set in the ocean.in file.

diag.F checks for out-of-bounds values for both density and velocity and reports what you see when it finds such. If it's a velocity problem, you can probably tell that from the last report of this sort (Max speed):

Code: Select all

   STEP   Day HH:MM:SS  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME
          C => (i,j,k)       Cu            Cv            Cw         Max Speed

       0 40907 00:00:00  6.870926E-03  2.248958E+04  2.248958E+04  2.359222E+16
         (936,1021,35)  1.511051E-02  9.792860E-03  0.000000E+00  1.827056E+00

I suggest setting NINFO to 1 to get this report every timestep. You can also look at the last record of the restart file to see what's going bad.

If this is a new domain, see if using a smaller timestep lets it run longer.

[wzhu]

Thank you so much, Kate.

It seems that we can set both the DSTART and Time_ref to 0 because I ran a model with them set to 0 succesfully.
My own model is just based on that model. It seems there is no report shown it is a velocity or density problem.
Yes, the NINFO is 1. I have three output files: ***_rst.nc, ***_his.nc and ***_avg.nc. The restart file is the ***_rst.nc file, right? what do you mean by the 'last record'? last record of the value of the parameter?
The last records of the log file is:

Code: Select all

   2221     1 03:45:45  1.763320E-02  8.136258E+01  8.138021E+01  3.613824E+11
   2222     1 03:46:30  1.770285E-02  8.134668E+01  8.136438E+01  3.613458E+11
   2223     1 03:47:15  1.777491E-02  8.133072E+01  8.134850E+01  3.613089E+11
   2224     1 03:48:00  1.784953E-02  8.131470E+01  8.133255E+01  3.612717E+11
   2225     1 03:48:45  1.792612E-02  8.129861E+01  8.131654E+01  3.612343E+11
   2226     1 03:49:30  1.800896E-02  8.128246E+01  8.130047E+01  3.611966E+11
   2227     1 03:50:15  1.808512E-02  8.126624E+01  8.128433E+01  3.611587E+11

 Blowing-up: Saving latest model state into  RESTART file

      WRT_RST   - wrote re-start fields (Index=1,2) into time record = 0000001

I have tried a smaller timestep with 30 s (45 s before), but after 2 days, also it blew up.
I attached the .h file and .in file as follows, any more suggestion? looking forward!

Code: Select all

/* Basic physics options */
#define UV_COR
#define UV_ADV
#define UV_VIS2
#define TS_DIF2
#define SOLVE3D
#define SALINITY
#define MIX_GEO_UV        /* mixing on geopotential (constant Z) surfaces */
#define MIX_GEO_TS        /* mixing on geopotential (constant Z) surfaces */
#define NONLIN_EOS

/* Basic numerics options */
/* #define INLINE_2DIO */
#define CURVGRID
#define MASKING
#define SPLINES
#define TS_MPDATA
#undef TS_U3HADVECTION   /* define if 3rd-order upstream horiz. advection */
#undef TS_SVADVECTION    /* define if splines vertical advection */
#define UV_SADVECTION     /* turn ON or OFF splines vertical advection */

/* average options */
#define AVERAGES
#define AVERAGES_AKV
#define AVERAGES_AKT
#define AVERAGES_AKS
#define AVERAGES_FLUXES

/* Basic Diagnostic options */
#undef DIAGNOSTICS_TS    /*compute and write diagnostic terms for tracer equations balances*/
#ifdef DIAGNOSTICS_TS
# define DIAGNOSTICS_TS_split/*compute and write diagnostic terms in XI- and ETA- direction*/
# define DIAGNOSTICS_StoZcor /*Compute and write correction terms to horizontal flux diagnositcs*/
# define DIAGNOSTICS_salt    /*Compute and write salinity diagnositc terms*/
# undef DIAGNOSTICS_UV       /*Compute and write momentum diagnostic terms*/
#endif

/* Outputs */
#define NO_WRITE_GRID   /* define if not writing grid arrays */
#undef WRITE_WATER      /* use if only writing water points data*/
#define INLINE_2DIO

/* Surface and bottom boundary conditions */
#define ANA_BSFLUX      /* analytical bottom salinity flux */
#define ANA_BTFLUX      /* analytical bottom temperature flux */
#define ANA_SSFLUX      /* analytical surface salinity flux */
#define ANA_STFLUX      /* analytical surface temperature flux */
#define UV_LOGDRAG      /* turn ON or OFF logarithmic bottom friction */
#undef QCORRECTION      /* define if net heat flux correction */
#undef SST_RELAXATION
#define BULK_FLUXES
#ifdef BULK_FLUXES
# define LONGWAVE_OUT
# define ANA_CLOUD
# define ANA_RAIN
#else
# define ANA_SMFLUX
# define ANA_STFLUX
#endif
#define ATM_PRESS
#define EMINUSP

/* Vertical subgridscale turbulence closure */
#ifdef SOLVE3D           /*!!!! Masked by Yun Li, Jan.11, 2008!!!!*/
# define GLS_MIXING      /* Activate Generic Length-Scale mixing */
# undef LMD_MIXING
# ifdef MY25_MIXING
#  undef N2S2_HORAVG     /* Activate horizontal smoothing of buoyancy/shear */
#  undef KANTHA_CLAYSON
# endif
# ifdef LMD_MIXING
#  define LMD_RIMIX
#  define LMD_CONVEC
#  define LMD_BKPP
#  define LMD_SKPP
#  undef LMD_NONLOCAL
# endif
# ifdef GLS_MIXING
#  define KANTHA_CLAYSON  /* Kantha and Clayson stability function formulation */
#  undef  N2S2_HORAVG     /* Activate horizontal smoothing of buoyancy/shear */
#  undef  CANUTO_A        /* Canuto A-stability function formulation */
#  undef  CANUTO_B        /* Canuto B-stability function formulation */
# endif
#endif

/* Open boundary conditions */
#define EAST_FSCHAPMAN
#define EAST_M2FLATHER
#define EAST_M3RADIATION
#define EAST_TRADIATION
#define EAST_TNUDGING
#define WESTERN_WALL
#define SOUTHERN_WALL
#define NORTHERN_WALL
#define OBC_DATA
#ifdef OBC_DATA
# define EAST_FSOBC
# define EAST_M2OBC
# define EAST_TOBC
#endif
#define TS_PSOURCE
#define UV_PSOURCE
#ifdef UV_PSOURCE
# define PSOURCE_FSCHAPMAN
#endif

/*  For Restart */
#define PERFECT_RESTART
#ifdef PERFECT_RESTART
# undef ANA_INITIAL
#endif

/* For sediment */
#define SEDIMENT
#ifdef SEDIMENT
# define SUSPLOAD
# define RIVER_SEDIMENT
# undef SED_DENS
# define ANA_SPFLUX
# define ANA_BPFLUX
#endif

Code: Select all

! Grid dimension parameters. See notes below in the Glossary for how to set
! these parameters correctly.

          Lm == 160           ! Number of I-direction INTERIOR RHO-points
          Mm == 240           ! Number of J-direction INTERIOR RHO-points
           N == 20            ! Number of vertical levels

        Nbed =  1             ! Number of sediment bed layers

         NAT =  2             ! Number of active tracers (usually, 2)
         NPT =  0             ! Number of inactive passive tracers
         NCS =  0             ! Number of cohesive (mud) sediment tracers
         NNS =  1             ! Number of non-cohesive (sand) sediment tracers

! Domain decomposition parameters for serial, distributed-memory or
! shared-memory configurations used to determine tile horizontal range
! indices (Istr,Iend) and (Jstr,Jend), [1:Ngrids].

      NtileI == 4                             ! I-direction partition
      NtileJ == 6                             ! J-direction partition

! Time-Stepping parameters.

      NTIMES ==  19200 ! 1985280 ! 1034(2006-1-1 to 2008-10-31)x1920x1 baroclinic steps/day
          DT ==  45.0d0
     NDTFAST ==  30

! Model iteration loops parameters.

       ERstr =  1
       ERend =  1
      Nouter =  1
      Ninner =  1
  Nintervals =  1

! Number of eigenvalues (NEV) and eigenvectors (NCV) to compute for the
! Lanczos/Arnoldi problem in the Generalized Stability Theory (GST)
! analysis. NCV must be greater than NEV (see documentation below).

         NEV =  2                               ! Number of eigenvalues
         NCV =  10                              ! Number of eigenvectors

! Input/Output parameters.

       NRREC == 0                 ! no Restart
   LcycleRST == T                 ! the latest two restart saved every NRST times
        NRST == 19200             ! 10 day x 1920 steps/day rst
        NSTA == 1
        NFLT == 1
       NINFO == 1                 ! display on the screen

! Output history, average, diagnostic files parameters.

     LDEFOUT == T
        NHIS == 320               !  4 hours
     NDEFHIS == 0!57600             !  30 days
      NTSAVG == 1
        NAVG == 1920              ! 24 hr
     NDEFAVG == 0!57600             ! 30 days
      NTSDIA == 1
        NDIA == 1920              ! 24 hours
     NDEFDIA == 0!57600             ! 10 days

! Output tangent linear and adjoint models parameters.

   LcycleTLM == F
        NTLM == 72
     NDEFTLM == 0
   LcycleADJ == F
        NADJ == 72
     NDEFADJ == 0
        NSFF == 72               ! new variables from ROMS 333
        NOBC == 72               ! new variables from ROMS 333

[wzhu]

Both DSTART and TIME_REF are set in the ocean.in file.

diag.F checks for out-of-bounds values for both density and velocity and reports what you see when it finds such. If it's a velocity problem, you can probably tell that from the last report of this sort (Max speed):

Code: Select all

   STEP   Day HH:MM:SS  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME
          C => (i,j,k)       Cu            Cv            Cw         Max Speed

       0 40907 00:00:00  6.870926E-03  2.248958E+04  2.248958E+04  2.359222E+16
         (936,1021,35)  1.511051E-02  9.792860E-03  0.000000E+00  1.827056E+00

I suggest setting NINFO to 1 to get this report every timestep. You can also look at the last record of the restart file to see what's going bad.

If this is a new domain, see if using a smaller timestep lets it run longer.

Hi, Kate, there is a good news: after I set the timestep to 30s and the NDTFAST to 20, it works!
how should I set the NDTFAST to the appropriate value?

Thank you very much, Kate.

[wzhu]

Hi Kate, It again blew up after 148 days, How could I do? use the smaller time steps?
How to restart with the rst file?

[kate]

Before restarting, you need to find a way to look at your solution. Is it otherwise looking reasonable? Where in the domain did the blowup happen? What sort of blowup is it, anyway? Top, bottom, middle? Velocity or density first? How quickly did things go from good to bad?

On restart, check NTIMES, set NRREC to nonzero, set LDEFOUT to F, and change ININAME to the name of your restart file. If a run is not blowing up you can set NRREC to -1, but if it blows up and saves a record right at failure, you don't want to be restarting from that record. Pick the specific record you want to be restarting from and set NRREC to that value. Check the restart file to make sure the record you chose looks good as opposed to showing early signs of the failure.

ETA: If there is trouble with your forcing, for instance, no amount of restarting will save you. However, one of my typical blowups goes from good to bad in just a few timesteps. Simply restarting from the earliest record of my restart file (with LcycleRST on) is good enough, even with PERFECT_RESTART.

[wzhu]

Before restarting, you need to find a way to look at your solution. Is it otherwise looking reasonable? Where in the domain did the blowup happen? What sort of blowup is it, anyway? Top, bottom, middle? Velocity or density first? How quickly did things go from good to bad?

On restart, check NTIMES, set NRREC to nonzero, set LDEFOUT to F, and change ININAME to the name of your restart file. If a run is not blowing up you can set NRREC to -1, but if it blows up and saves a record right at failure, you don't want to be restarting from that record. Pick the specific record you want to be restarting from and set NRREC to that value. Check the restart file to make sure the record you chose looks good as opposed to showing early signs of the failure.

ETA: If there is trouble with your forcing, for instance, no amount of restarting will save you. However, one of my typical blowups goes from good to bad in just a few timesteps. Simply restarting from the earliest record of my restart file (with LcycleRST on) is good enough, even with PERFECT_RESTART.

Thank you so much, Kate.
So I have to look into the rst.nc file to check it out, right? But the restart file seems to be broken. I reduced the timestep to 25s and NDTFAST to 15 and let it run from initial time again. I am not sure if it can work pass the same point, because someone said since it blew up after 148 days, it is probably not the timestep problem, What do you think?

Thanks again.

[kate]

Someone once told me that the later a model gets into trouble, the harder it is to figure out. It could be anything.

I assume you are running the model with a purpose in mind, involving looking at the model output. It is time to work on those skills. Start with ncview - it's really simple.

[wzhu]

Someone once told me that the later a model gets into trouble, the harder it is to figure out. It could be anything.

I assume you are running the model with a purpose in mind, involving looking at the model output. It is time to work on those skills. Start with ncview - it's really simple.

OK, Thank you, I will do that.