Climatological run error: temperature vanishing

[Eugen]

Hi,

I have a strange problem with the climatological multi-year simulation in ROMS.

I force the model by a climatological atmospheric forcing, climatological boundary conditions and climatological nudging at the open boundaries. And also with climatological river point sources and tides. So all the forcing are repeated each year, except tides.

However, at the middle of the second (!) year of the simulation, something happens and the temperature is quickly filled with exclusion value 1e+37. The given time step is 12 seconds and the output step - once per day. My frames show, that it happens gradually (see the pictures below). The first frame is the last OK state of simulation, the next frame - the half of the domain is vanished, the last frame - everything disappeared.

======> ======>

The time series of temperature also shows, that there are no more valid temperature values after one year and a half (the figure below).




However, all the other variables continues to be successfully simulated (!). For example, salinity (the picture below) is being simulated during all five years.


Furthermore, although the phenomenon is repeated, I can not reproduce by restarting (even with the same time step) from some initial conditions taken before it happens (which I tried to do to identify the source of this 'NaN' spreading).

So the strange aspects which I don't understand are :

1. Why does this happens during the second year, although the atmospheric and boundary forcing are the same as for the first year?
2. How could it be, that the other variables are still computed and written, then temperature has only NaN values?

The important thing is - without tides there is no such an error. Non-tidal 10-years climatological run went successfully. So, I suspect, it could be due to tides. And tides is also the only forcing, that does not repeat every year.

I built tidal forcing with using of the matlab ROMS script. As a source, I took TPXO Global Tidal Solutions.

I also use WET_DRY, which I suspect, could be involved in the answer to this issue (not sure). The domain, where I'm working, is close to the West Coast of France, where tides are very big.

Thank you in advance!

[kate]

It's true that tides could be responsible. Hard to tell without sneaking up on it as you tried to do. Maybe PERFECT_RESTART would solve that restart issue you had. If not, there could still be bugs in the implementation of PERFECT_RESTART. It's complicated with wetting and drying!

I don't see how it could continue with bad temperature and good salt. I've never had that happen before. What is your NINFO? If temperature goes bad, density should as well, causing the call to diag.F every NINFO steps to crash the model. I use NINFO = 1. I also add more verbose output to diag.F because these things happen. You could add a check on the temperature extrema.

[Eugen]

kate, thank you for the answer!

The error remains again, also during interannual (10 years) real-data run.

We think, that anomalous low temperatures could be a reason of this numerical error. I have many of them occasionally occurring time-by-time along the grid mask. But most of the time, the model can overcome these anomalies. We assume, that finally the model has to deal an extremely unnatural value, which the model can't overcome, so it sets temperature values to NaN. Here below is the picture of the typical grid cell with a crazy temperature value.



Unfortunately, density does not force the model to shut down, because then temperature "disappears", density becomes just twice lower (see the picture below):



So, I checked, that I have NINFO=1. I also enabled the PERFECT_RESTART and made some corrections in diag.f, to trigger the shutdown of the model, if water temperature goes very low (lower than -20 C).

So, we investigated the features of this numerical error. This error can not be reproduced - perfect restart from the state 6-hours before the last recording does not give the same numerical error...

But all these grid cells, that can potentially get crazy temperature values, are shallow (~ 4 m depth - the minimum depth of the domain) , do not have strong bathymetry gradients with surrounding cells, and are located along the grid mask. Their tidal elevations are very big, so WET-DRY subroutine should deal with them during the simulation.

Could you tell, please, it is right, that these very cold temperature values can be in charge of final crucial numerical error (when temperature disappears from the domain)? How it is possible to have such bad values at these points? And what can we do to overcome these problems and finally run a good interannual simulation?

[kate]

I too just had a job blow up after several years of happily running. I think my problem is that I need to turn on WET_DRY because I have zeta=-3.4 where the depth is 2 m. Did you try WET_DRY yet? For me, velocities went nuts, not temperature.

I still don't understand how your job can continue to run with temperatures of NaN. Perhaps there's a compiler setting you can use to change that behavior. I'm debugging something else (tracmass) with ifort using a flag that initializes everything to NaN and another flag to die when operating on NaN - boy, is it smoking out stuff. It's the flags ROMS sets for ifort when USE_DEBUG is on.