Restarted ROMS does not write out to history files

[jprinehimer]

I was going to submit this as a bug report, but I want to make sure I didn't screw something up in my setup.

I tried to restart a ROMS run by changing these parameters from the original ocean.in file:

Code: Select all

NRREC == -1
LDEFOUT == F   ! to append to existing file
ININAME = (restart file)

The restart would work, but no records would be written to the history files. Records were written to the restart files.

The condition at line 75 of Nonlinear/output.F:

Code: Select all

IF (iic(ng)-1).eq.idefHIS(ng)) THEN

is not met at the beginning of a restart, so LwrtHIS(ng) is not set to true and no history files are written until the next history file is defined.

I was able to solve this (somewhat) by adding the following kludge code below line 106 of Nonlinear/output.F:

Code: Select all

! Kludge code to output data from history files
      IF ((nrrec(ng).ne.0).and.(iic(ng).eq.ntstart(ng)).and.             &
     &     (MOD(iic(ng)-1,ndefHIS(ng)).gt.0)) THEN
         ldef=ldefout(ng)
         ifile=INT((iic(ng)-1)/ndefHIS(ng))+1
         IF (Master) THEN
            lstr=LEN_TRIM(HISbase(ng))
            WRITE (HISname(ng),10) HISbase(ng)(1:lstr-3),ifile
         END IF
         IF (ncHISid(ng).ne.-1) status=nf90_close(ncHISid(ng))

         CALL def_his (ng, ldef)
         IF (exit_flag.ne.NoError) RETURN
         LwrtHIS(ng)=.TRUE.
      END IF

(see also: output.F in jprinehimer_branch/trunk on the CSTM site.)

This could probably be fixed in the code in lines 67-106 by using appropriate conditional statements.

I was unable to get the code to write out to the correct index in the netcdf file. In my case, NRST and NHIS are the same, so the last history record matches the last timestep. Where the restart happened, I'd get two of the same timestep in the history file. I tried altering NrecHIS, but that didn't work and then saw that there was also tHISindx(ng) in wrt_his.F, but couldn't figure out how to set it properly.

[arango]

Well, I checked this logic in the debugger again today and it works for me. I have done this many times in the past; every time a problem has been reported. In most previous reports the logic is fine. The problem is how the user set the parameters in input script ocean.in.

However, I don't have all the information in your posting to reproduce your logic failure. Very important information is missing for me to reproduce the problem. It is possible that I don't have all the possible combinations for this logic. This logic have become complicated through the years to allow different scenarios.

What values are you using for NTIMES, NHIS, NRST, and NDEFHIS? You just mentioned that NHIS=NRST. I assume that NDEFHIS>0, because you mentioned the conditional in line 75. This value is very important. At first look I can see that perhaps additional logic is needed if NDEFHIS is not a multiple of NTIMES. This means that when the execution terminated the multiple history files were not finished and more time records are needed to close that file and define the new one. Or the restart record doesn't correspond to the time that a new history NetCDF needs to be defined.

[jprinehimer]

At first look I can see that perhaps additional logic is needed if NDEFHIS is not a multiple of NTIMES. This means that when the execution terminated the multiple history files were not finished and more time records are needed to close that file and define the new one. Or the restart record doesn't correspond to the time that a new history NetCDF needs to be defined.

I think both are the case. While my NDEFHIS is a multiple of NTIMES, the restart record corresponded to a timestep in the middle of a history file. The run had to be shut down due to queue issues and I wanted to restart it when the queues were back up.

This should only be a problem with NDEFHIS>0. With NDEFHIS=0, def_his should be called appropriately. I don't know but might there also be issues with averages or diagnostics files in these cases as well?

Here are the requested values for my model:

Code: Select all

NTIMES == 115200
NRREC == -1
LcycleRST == T
NRST == 60
LDEFOUT == F
NHIS == 60
NDEFHIS == 3840

In the original input file (pre-restart) they were:

Code: Select all

NRREC == 0 
LDEFOUT == T

The full input file (pre-restart) can be found at:
https://www.myroms.org/svn/cstm/branche ... ud_high.in
Of course, ININAME was changed for the restart input to correspond to RSTNAME

[jprinehimer]

I just did a test using the ESTUARY_TEST case.

I changed these values from the original:

Code: Select all

NRST == 120     !Same as NHIS
NHIS == 120     !Same as original
NDEFHIS == 600  !5 records per file

I then compiled the model with the initial analytical conditions and force quit half-way through ocean_his_0003.nc, around time-step 1580. Three records were written to ocean_his_0003.nc, with the last at timestep 1560 (ocean_time=46800).

I then set:

Code: Select all

NRREC == -1
LDEFOUT == F
ININAME == ocean_rst.nc

and recompiled the code with ANA_INITIAL and ANA_SEDIMENT turned off. I then restarted the run and it ran starting at timestep 1580. No new records were written to ocean_his_0003.nc. The record from step 1680 should have been written out, but was not.

The history file ocean_his_0004.nc was created. The first record in ocean_his_0004.nc was at timestep 1800 (ocean_time = 54000). This is NHIS steps before the file should have been created, had it not been restarted.

[kate]

I too have run into that problem and fought with Hernan over it. I ended up changing output.F to get a version that works the way I think it should. I had to create another logical variable which I think simplifies it a little.

[jprinehimer]

This has been fixed by ticket #132 [ROMS Trac] [CSTMS Trac]
and ticket #134 [ROMS Trac] [CSTMS Trac]

Many thanks to Hernan for fixing the bug.