I've been trying to set up a run that includes an area with large tides. It's been interesting and it continues to be interesting. We've added the WET_DRY option and fixed the biology blow-ups. However, there are still intermittent problems I don't understand. I put in some stations in order to monitor the previous tidal blow ups and I've managed to capture something odd. Here is the total water column depth for the "bad" station and some of its neighbors:
What follows next are three different surface temperature timeseries for exactly the same executable, three different runs:
The last of these caused the model to blow up while the first two did not. I believe the time and place of the trouble would have a negative water depth without the WET_DRY option. Is there some place in the tracer code that needs to check against the wet-dry mask? I'm not even sure that mask is reading as dry at the time of the 3D timestep. Has anyone else seen anything like this?
We are spending some more time on the wet/dry option. I was having some issues with boundary conditions and need to finalize that and submit to Hernan. But it looks like you are having other issues. I am not sure that it is related to the wet/dry stuff. Right now wet/dry is limited to be a barotropic process. As the water level in a cell becomes shallow, the transport out of the cell will be prevented if the total depth < Dcrit. We think that there should also be an effect on the baroclinic velcoities but there are not any effects right now. We are looking into that. But at very shallow depths the baroclinic flows are typically small. Is there a large wind stress at this time? What does it mean that there are 3 different time series for the same executable?
Good question on the wind stress - yes, it's huge in both u(-) and v(+) directions at that one time and place. Those plots look identical between the first two cases though. Does the wind stress get big when the water depth gets that small?
I don't know what it means that I get different answers for different runs. Those were all with the Sun compiler on Linux. A Pathscale run just now matches the second plot. A PGI run blows up much earlier (twice) and I've got one in the queue on the IBM.
Does the model do anything different after writing a restart record? The only difference between the first two plots is that I asked for a restart record just before the crash on one of them. The other one matches the daily restart frequency of the bad run.
Cutting down to -O1 or -O2 (Pathscale) gives a repeatable temperature spike of 4 degrees. Running on the IBM gives no big spike:
Saving a restart file just before and running from that doesn't give a spike either. Come to think of it, the IBM job was restarted after it ran into a 2 GB file size problem. Here's the 4 degree spike and the corresponding temperature of the under-ice log layer:
I was thinking it might be something to do with the log layer code, but then all the bio tracers get a spike too.
I haven't figured out what is going wrong. The IBM without restart gives a negative spike, by the way. Meanwhile, trying Dcrit = 0.5 instead of 0.1 seems to work so that will be my working solution. The mystery remains, however.
