Hi all,
I did quite some trials to compare restarted runs and straight runs of ROMS3.0 foy my application of the Chesapeake Bay, I didn't get perfect restarted results.
I do see there is a CPP option called PERFECT_RESTART, All my tests did not include definition of this CPP option. Do I have to include it? Anybody can tell me when this is needed and a digest of its implementation roughly?
What exactly is the difference between inclusion and exclusion of this option?
Thanks,
Wen
PERFECT_RESTART cpp option
well, if you want to activate it then you need to activate it.
So give it a try:
#define PERFECT_RESTART
I did some tests with this option, and was able to achieve a perfect restart for several applications. I did not get a perfect restart when I activted MY25_MIXING. Still need to investigate why. But I did get perfect restart for GLS_MIXING. I do not have my notes with me right now, and I can not remember exactly which test cases I ran. But we did spend a considerable effort to get this to work, so please try it.
So give it a try:
#define PERFECT_RESTART
I did some tests with this option, and was able to achieve a perfect restart for several applications. I did not get a perfect restart when I activted MY25_MIXING. Still need to investigate why. But I did get perfect restart for GLS_MIXING. I do not have my notes with me right now, and I can not remember exactly which test cases I ran. But we did spend a considerable effort to get this to work, so please try it.
-
longmtm
- Posts: 55
- Joined: Tue Dec 13, 2005 7:23 pm
- Location: Univ of Maryland Center for Environmental Science
Hi
I tried to activate the model PERFECT_RESTART cpp option on SGI openMP system using 8 process. I still met some diffriculty which I beileve is related to netcdf:
First of all, I tried activating both PERFECT_RESTART and OUT_DOUBLE and LARGE=on for 64 bit compilation.
Then the model had problem reading the netcdf restart file with the following error:
//----screen output ----
NLM: GET_STATE - Read state initial conditions, t = 30.0000
(File: chesroms_rst_1mon_x64_PFTRST_2.nc, Rec=0002, Index=1)
NLM: GET_STATE - error while reading variable: zeta at time record = 2
in input NetCDF file: ../forcing/chesroms_rst_1mon_x64_PFTRST_2.nc
ROMS/TOMS - Input error ............. exit_flag: 2
ERROR: Abnormal termination: NetCDF INPUT.
REASON: Unknown Error
//------------
Next, I thought the above problem is due to the 64bit compilation of netcdf on SGI machine, then I went ahead and tried with only: PERFECT_RESTART option, no OUT_DOUBLE and use 32bit netcdf, the model runs but reads the wrong values out of the netcdf restart file:
//---ROMS screen output----
NLM: GET_STATE - Read state initial conditions, t = 30.0000
(File: chesroms_rst_1mon_PFTRST_2.nc, Rec=0002, Index=1)
- free-surface
(Min = 2.09861000-315 Max = 9.98012605-322)
- vertically integrated u-momentum component
(Min = 2.09861000-315 Max = 9.98012605-322)
- vertically integrated v-momentum component
(Min = 2.09861000-315 Max = 9.98012605-322)
- u-momentum component
(Min = 1.00000000E+35 Max = -1.00000000E+35)
- v-momentum component
(Min = 1.00000000E+35 Max = -1.00000000E+35)
- potential temperature
(Min = 1.00000000E+35 Max = -1.00000000E+35)
- salinity
(Min = 1.00000000E+35 Max = -1.00000000E+35)
//---------
All Min and Max in the upper outputs are wrong, but ncdump shows the values are okay. I wonder why???
Is this because ROMS3.0 PERFECT_RESTART on SGI machine requires some newer version of netcdf? Any experience or suggestions? BTW, I'm using netcdf 3.6.1.
The restart works without PERFECT_RESTART except that the restart is not perfectly seamless:
//----ROMS screen output----
NLM: GET_STATE - Read state initial conditions, t = 30.0000
(File: chesroms_rst_1mon_1.nc, Rec=0002, Index=1)
- free-surface
(Min = -3.83674198E-01 Max = 7.77120845E-01)
- vertically integrated u-momentum component
(Min = -9.80383047E-01 Max = 6.85759200E-01)
- vertically integrated v-momentum component
(Min = -7.72161673E-01 Max = 8.78423631E-01)
- u-momentum component
(Min = -1.08687845E+00 Max = 9.23186993E-01)
- v-momentum component
(Min = -9.84046236E-01 Max = 9.46847051E-01)
- potential temperature
(Min = -1.42631364E+01 Max = 6.94150150E+00)
- salinity
(Min = -5.00611365E-01 Max = 2.99097545E+01)
- vertical viscosity coefficient
(Min = 0.00000000E+00 Max = 2.35661786E-01)
- temperature vertical diffusion coefficient
(Min = 0.00000000E+00 Max = 3.19895296E-01)
- salinity vertical diffusion coefficient
(Min = 0.00000000E+00 Max = 3.19895296E-01)
//---------------
I'll have to try this on an other machine and see what is the reason.
Wen
I tried to activate the model PERFECT_RESTART cpp option on SGI openMP system using 8 process. I still met some diffriculty which I beileve is related to netcdf:
First of all, I tried activating both PERFECT_RESTART and OUT_DOUBLE and LARGE=on for 64 bit compilation.
Then the model had problem reading the netcdf restart file with the following error:
//----screen output ----
NLM: GET_STATE - Read state initial conditions, t = 30.0000
(File: chesroms_rst_1mon_x64_PFTRST_2.nc, Rec=0002, Index=1)
NLM: GET_STATE - error while reading variable: zeta at time record = 2
in input NetCDF file: ../forcing/chesroms_rst_1mon_x64_PFTRST_2.nc
ROMS/TOMS - Input error ............. exit_flag: 2
ERROR: Abnormal termination: NetCDF INPUT.
REASON: Unknown Error
//------------
Next, I thought the above problem is due to the 64bit compilation of netcdf on SGI machine, then I went ahead and tried with only: PERFECT_RESTART option, no OUT_DOUBLE and use 32bit netcdf, the model runs but reads the wrong values out of the netcdf restart file:
//---ROMS screen output----
NLM: GET_STATE - Read state initial conditions, t = 30.0000
(File: chesroms_rst_1mon_PFTRST_2.nc, Rec=0002, Index=1)
- free-surface
(Min = 2.09861000-315 Max = 9.98012605-322)
- vertically integrated u-momentum component
(Min = 2.09861000-315 Max = 9.98012605-322)
- vertically integrated v-momentum component
(Min = 2.09861000-315 Max = 9.98012605-322)
- u-momentum component
(Min = 1.00000000E+35 Max = -1.00000000E+35)
- v-momentum component
(Min = 1.00000000E+35 Max = -1.00000000E+35)
- potential temperature
(Min = 1.00000000E+35 Max = -1.00000000E+35)
- salinity
(Min = 1.00000000E+35 Max = -1.00000000E+35)
//---------
All Min and Max in the upper outputs are wrong, but ncdump shows the values are okay. I wonder why???
Is this because ROMS3.0 PERFECT_RESTART on SGI machine requires some newer version of netcdf? Any experience or suggestions? BTW, I'm using netcdf 3.6.1.
The restart works without PERFECT_RESTART except that the restart is not perfectly seamless:
//----ROMS screen output----
NLM: GET_STATE - Read state initial conditions, t = 30.0000
(File: chesroms_rst_1mon_1.nc, Rec=0002, Index=1)
- free-surface
(Min = -3.83674198E-01 Max = 7.77120845E-01)
- vertically integrated u-momentum component
(Min = -9.80383047E-01 Max = 6.85759200E-01)
- vertically integrated v-momentum component
(Min = -7.72161673E-01 Max = 8.78423631E-01)
- u-momentum component
(Min = -1.08687845E+00 Max = 9.23186993E-01)
- v-momentum component
(Min = -9.84046236E-01 Max = 9.46847051E-01)
- potential temperature
(Min = -1.42631364E+01 Max = 6.94150150E+00)
- salinity
(Min = -5.00611365E-01 Max = 2.99097545E+01)
- vertical viscosity coefficient
(Min = 0.00000000E+00 Max = 2.35661786E-01)
- temperature vertical diffusion coefficient
(Min = 0.00000000E+00 Max = 3.19895296E-01)
- salinity vertical diffusion coefficient
(Min = 0.00000000E+00 Max = 3.19895296E-01)
//---------------
I'll have to try this on an other machine and see what is the reason.
Wen
I think CPP option "PERFECT_RESTART" is very important for developing operational forecast systems since forecasts start from a restart file generated from previous nowcasts and we don't expect to get any oscillation at the forecast start time.
I tested this option in my Tampa Bay application with the option "MY25_MIXING". I just compared elevations at some locations, and results show that there is no oscillation at the beginning restart time, and the results of the restart run are consistent with the results of the continuous run. However, one strange thing is that: ROMS blowup after more than 8 day simulation, in order to debug, I reran ROMS from hot_start (restart file was saved at 8th day with perfect_restart option activated), ROMS didn't blewup at that time. So I think this is related with the perfect_restart option.
In terms of WEN's error message, we found there is a bug in initial.F for PERFECT_RESTART. we modified the initial.F as,
#ifdef INI_FILE
IF (nrrec(ng).eq.0) THEN
!
! Read in initial conditions from initial NetCDF file.
!
CALL get_state (ng, iNLM, 1, INIname(ng), IniRec, Tindex)
# ifdef DISTRIBUTE
CALL mp_bcasti (ng, iNLM, exit_flag, 1)
# endif
IF (exit_flag.ne.NoError) RETURN
!
ELSE
!
! If restart, read in initial conditions restart NetCDF file.
!
CALL get_state (ng, 0, 1, INIname(ng), IniRec, Tindex)
# ifdef DISTRIBUTE
CALL mp_bcasti (ng, iNLM, exit_flag, 1)
# endif
IF (exit_flag.ne.NoError) RETURN
!
END IF
#endif
I haven't tried the CPP option "GLS_MIXING" as John did, I would like to try this option with PERFECT_RESTART.
By the way, my ROMS version is ROMS 3.0 downloaded on June 1, 2007.
The CPP options are as,
** Application flag: TAMPA_BAY
** Input script: toms_tampa_bay.in
*/
#define SOLVE3D
#define CURVGRID
#define MASKING
#define WET_DRY
#define SALINITY
#ifdef SALINITY
# define ANA_BSFLUX
# define ANA_SSFLUX
#endif
#define NONLIN_EOS
#define ANA_BTFLUX
#define ANA_STFLUX
#define ANA_SMFLUX
#define UV_ADV
#define UV_COR
#define UV_QDRAG
#define UV_SADVECTION
#define TS_U3HADVECTION
#define TS_SVADVECTION
#define DJ_GRADPS
#define SPLINES
#define RADIATION_2D
#define WEST_M3RADIATION
#define WEST_TRADIATION
#define SOUTH_M3RADIATION
#define SOUTH_TRADIATION
#define ANA_FSOBC
#define ANA_M2OBC
#define SSH_TIDES
#define RAMP_TIDES
#ifdef SSH_TIDES
# define WEST_FSCHAPMAN
# define SOUTH_FSCHAPMAN
#endif
#define UV_TIDES
#ifdef UV_TIDES
# define WEST_M2FLATHER
# define SOUTH_M2FLATHER
#endif
#define MY25_MIXING
#undef GLS_MIXING
#if defined MY25_MIXING || defined GLS_MIXING
# define N2S2_HORAVG
# define KANTHA_CLAYSON
#endif
#define STATIONS
#undef PERFECT_RESTART
#ifdef PERFECT_RESTART
# undef AVERAGES
# undef DIAGNOSTICS_BIO
# undef DIAGNOSTICS_TS
# undef DIAGNOSTICS_UV
# define OUT_DOUBLE
#endif
I tested this option in my Tampa Bay application with the option "MY25_MIXING". I just compared elevations at some locations, and results show that there is no oscillation at the beginning restart time, and the results of the restart run are consistent with the results of the continuous run. However, one strange thing is that: ROMS blowup after more than 8 day simulation, in order to debug, I reran ROMS from hot_start (restart file was saved at 8th day with perfect_restart option activated), ROMS didn't blewup at that time. So I think this is related with the perfect_restart option.
In terms of WEN's error message, we found there is a bug in initial.F for PERFECT_RESTART. we modified the initial.F as,
#ifdef INI_FILE
IF (nrrec(ng).eq.0) THEN
!
! Read in initial conditions from initial NetCDF file.
!
CALL get_state (ng, iNLM, 1, INIname(ng), IniRec, Tindex)
# ifdef DISTRIBUTE
CALL mp_bcasti (ng, iNLM, exit_flag, 1)
# endif
IF (exit_flag.ne.NoError) RETURN
!
ELSE
!
! If restart, read in initial conditions restart NetCDF file.
!
CALL get_state (ng, 0, 1, INIname(ng), IniRec, Tindex)
# ifdef DISTRIBUTE
CALL mp_bcasti (ng, iNLM, exit_flag, 1)
# endif
IF (exit_flag.ne.NoError) RETURN
!
END IF
#endif
I haven't tried the CPP option "GLS_MIXING" as John did, I would like to try this option with PERFECT_RESTART.
By the way, my ROMS version is ROMS 3.0 downloaded on June 1, 2007.
The CPP options are as,
** Application flag: TAMPA_BAY
** Input script: toms_tampa_bay.in
*/
#define SOLVE3D
#define CURVGRID
#define MASKING
#define WET_DRY
#define SALINITY
#ifdef SALINITY
# define ANA_BSFLUX
# define ANA_SSFLUX
#endif
#define NONLIN_EOS
#define ANA_BTFLUX
#define ANA_STFLUX
#define ANA_SMFLUX
#define UV_ADV
#define UV_COR
#define UV_QDRAG
#define UV_SADVECTION
#define TS_U3HADVECTION
#define TS_SVADVECTION
#define DJ_GRADPS
#define SPLINES
#define RADIATION_2D
#define WEST_M3RADIATION
#define WEST_TRADIATION
#define SOUTH_M3RADIATION
#define SOUTH_TRADIATION
#define ANA_FSOBC
#define ANA_M2OBC
#define SSH_TIDES
#define RAMP_TIDES
#ifdef SSH_TIDES
# define WEST_FSCHAPMAN
# define SOUTH_FSCHAPMAN
#endif
#define UV_TIDES
#ifdef UV_TIDES
# define WEST_M2FLATHER
# define SOUTH_M2FLATHER
#endif
#define MY25_MIXING
#undef GLS_MIXING
#if defined MY25_MIXING || defined GLS_MIXING
# define N2S2_HORAVG
# define KANTHA_CLAYSON
#endif
#define STATIONS
#undef PERFECT_RESTART
#ifdef PERFECT_RESTART
# undef AVERAGES
# undef DIAGNOSTICS_BIO
# undef DIAGNOSTICS_TS
# undef DIAGNOSTICS_UV
# define OUT_DOUBLE
#endif
-
longmtm
- Posts: 55
- Joined: Tue Dec 13, 2005 7:23 pm
- Location: Univ of Maryland Center for Environmental Science
Thank god! Oh, no, thanks to the ROMS forum, the restart is working with the PERFECT_RESTART option, GLS_MIXING, and a bug fix pointed out by AiJun and Lyon in NOAA.
Here are the results based on station outputs.
http://131.118.211.30/chesromsweb/inter ... est/zilla/
Wen
Here are the results based on station outputs.
http://131.118.211.30/chesromsweb/inter ... est/zilla/
Wen
-
longmtm
- Posts: 55
- Joined: Tue Dec 13, 2005 7:23 pm
- Location: Univ of Maryland Center for Environmental Science
I want to come back to this bug fix pointed out by Aijun Zhang:
I believe this bug was introduced when ROMS2.2 is upgraded to ROMS3.0 and when PERFECT_RESTART option was introduced.
The fix was in the previous message and also copied here:
1 #ifdef INI_FILE
2 IF (nrrec(ng).eq.0) THEN
3 !
4 ! Read in initial conditions from initial NetCDF file.
5 !
6 CALL get_state (ng, iNLM, 1, INIname(ng), IniRec, Tindex)
7 # ifdef DISTRIBUTE
8 CALL mp_bcasti (ng, iNLM, exit_flag, 1)
9 # endif
10 IF (exit_flag.ne.NoError) RETURN
11 !
12 ELSE
13 !
14 ! If restart, read in initial conditions restart NetCDF file.
15 !
16 CALL get_state (ng, 0, 1, INIname(ng), IniRec, Tindex)
17 # ifdef DISTRIBUTE
18 CALL mp_bcasti (ng, iNLM, exit_flag, 1)
19 # endif
20 IF (exit_flag.ne.NoError) RETURN
21 !
22 END IF
23 #endif
In the upper line number 18, should it be CALL mp_bcasti (ng, 0, exit_flag, 1) ???
Thanks,
Wen
I believe this bug was introduced when ROMS2.2 is upgraded to ROMS3.0 and when PERFECT_RESTART option was introduced.
The fix was in the previous message and also copied here:
1 #ifdef INI_FILE
2 IF (nrrec(ng).eq.0) THEN
3 !
4 ! Read in initial conditions from initial NetCDF file.
5 !
6 CALL get_state (ng, iNLM, 1, INIname(ng), IniRec, Tindex)
7 # ifdef DISTRIBUTE
8 CALL mp_bcasti (ng, iNLM, exit_flag, 1)
9 # endif
10 IF (exit_flag.ne.NoError) RETURN
11 !
12 ELSE
13 !
14 ! If restart, read in initial conditions restart NetCDF file.
15 !
16 CALL get_state (ng, 0, 1, INIname(ng), IniRec, Tindex)
17 # ifdef DISTRIBUTE
18 CALL mp_bcasti (ng, iNLM, exit_flag, 1)
19 # endif
20 IF (exit_flag.ne.NoError) RETURN
21 !
22 END IF
23 #endif
In the upper line number 18, should it be CALL mp_bcasti (ng, 0, exit_flag, 1) ???
Thanks,
Wen