why Roms can restart from the step where the model blows up?

[zhouwei]

My model blows up at 1494, but I can't find any problem from rst file. So I rerun this model from the restart file without changing other input files like forcing files and bry files, it can run successfully. Could anyone tell me why this happened?

The First Run:

430248 1493 22:00:00 3.575556E-02 2.083945E+04 2.083949E+04 1.259299E+16
430260 1493 23:00:00 3.574807E-02 2.083945E+04 2.083949E+04 1.259300E+16
430272 1494 00:00:00 3.573948E-02 2.083945E+04 2.083949E+04 1.259301E+16
WRT_HIS - wrote history fields (Index=1,1) into time record = 0000006
GET_2DFLD - surface u-momentum stress, t = 1495 00:00:00
(Rec=1496, Index=2, File: scs-wind_Qscat.nc)
(Tmin= 0.0000 Tmax= 2131.0000)
(Min = -4.63869320E-04 Max = 8.55954701E-05)
GET_2DFLD - surface v-momentum stress, t = 1495 00:00:00
(Rec=1496, Index=2, File: scs-wind_Qscat.nc)
(Tmin= 0.0000 Tmax= 2131.0000)
(Min = -2.46272972E-04 Max = 9.52415337E-05)
430284 1494 01:00:00 3.572706E-02 2.083945E+04 2.083949E+04 1.259302E+16
430296 1494 02:00:00 3.571071E-02 2.083945E+04 2.083948E+04 1.259303E+16
430308 1494 03:00:00 3.569133E-02 2.083944E+04 2.083948E+04 1.259303E+16
430320 1494 04:00:00 3.568990E-02 2.083944E+04 2.083948E+04 1.259304E+16
430332 1494 05:00:00 3.571634E-02 2.083944E+04 2.083948E+04 1.259305E+16
430344 1494 06:00:00 NaN NaN NaN NaN

Blowing-up: Saving latest model state into RESTART file

WRT_RST - wrote re-start fields (Index=1,1) into time record = 0000003

The Second Run:

WRT_HIS - wrote history fields (Index=1,1) into time record = 0000005
430212 1493 19:00:00 3.625512E-02 2.083945E+04 2.083949E+04 1.259294E+16
430224 1493 20:00:00 3.623067E-02 2.083946E+04 2.083949E+04 1.259295E+16
430236 1493 21:00:00 3.618520E-02 2.083945E+04 2.083949E+04 1.259297E+16
430248 1493 22:00:00 3.614200E-02 2.083945E+04 2.083949E+04 1.259298E+16
430260 1493 23:00:00 3.610072E-02 2.083945E+04 2.083949E+04 1.259299E+16
430272 1494 00:00:00 3.605927E-02 2.083945E+04 2.083949E+04 1.259300E+16
WRT_HIS - wrote history fields (Index=1,1) into time record = 0000006
GET_2DFLD - surface u-momentum stress, t = 1495 00:00:00
(Rec=1496, Index=2, File: scs-wind_Qscat.nc)
(Tmin= 0.0000 Tmax= 2131.0000)
(Min = -4.63869320E-04 Max = 8.55954701E-05)
GET_2DFLD - surface v-momentum stress, t = 1495 00:00:00
(Rec=1496, Index=2, File: scs-wind_Qscat.nc)
(Tmin= 0.0000 Tmax= 2131.0000)
(Min = -2.46272972E-04 Max = 9.52415337E-05)
430284 1494 01:00:00 3.601985E-02 2.083945E+04 2.083949E+04 1.259302E+16
430296 1494 02:00:00 3.598046E-02 2.083945E+04 2.083948E+04 1.259303E+16
430308 1494 03:00:00 3.593951E-02 2.083944E+04 2.083948E+04 1.259304E+16
430320 1494 04:00:00 3.589945E-02 2.083944E+04 2.083947E+04 1.259305E+16
430332 1494 05:00:00 3.586047E-02 2.083943E+04 2.083947E+04 1.259306E+16
430344 1494 06:00:00 3.582194E-02 2.083943E+04 2.083946E+04 1.259307E+16

[kate]

I'd call this a case against perfect restart. There's some sort of timestepping instability that's really subtle.