Blow Up

[bru]

I am wondering what makes a model to blow-up?

Actually, when I run the model, even if the CFL criterium is respected (Cu_max < 1), it happens that the model blows-up at a certain step. How come? Kinetic energy increases too much. This is due to a too big increasing of velocity. Where does it come from ? Is there a way to control this? Should I play with timestepping? I would appreciate if someone could tell me more about this.

[arango]

Well, fortunatelly models blow-up once in a while. Otherwise, we would all be out of jobs and profession.

It is hard to diagnose why a particular applications blows-up. If the model blows-up immidiatelly it is due to a horizontal or vertical CFL violation. You can start playing with the time-step parameters to see if this solves the problem. If the blowing-up persists, the problem is more serious and usually associated with configuration. There are several things to consider:

(1) Inappropriate bathymetry with very steep and tall features. You need to do some smoothing to reduce the r-factor.

(2) Problems with land/sea masking. Try to avoid one-grid bays.

(3) Initial conditions. Unbalanced initial conditions can create havoc during initial time-stepping. As a rule, never extrapolate. The T-S relationship in the deep ocean is very tight. Your initial temperature and salinty must give a neutral density. Otherwise, a lot of convection takes place with huge vertical velocities.

(4) Grid configuration. Avoid placing open boundaries next to steep topographic features and very intense dynamic regions.

(5) Atmopheric forcing. Usually the model blows-up suddenly because of strong atmospheric forcing. Sometimes you can find very powerful storms when using daily atmospheric products. The remedy here is to lower the time-step during these storms events.

Good luck,

[bru]

Thank you for those comments. I'm going to check those points and find out what is going wrong.

[bru]

(4) Grid configuration

For the Mediterranean basin example, an obvious open boundary would be the Gibraltar Strait. The mesh needs to be thin enough so as this district is well represented on the grid. This region is not so steep but really is a dynamic area. By saying:

Avoid placing open boundaries next to steep topographic features and very intense dynamic regions.

Do you mean I should close this boundary? This would make no sense to me, since Mediterranean Sea is at the Gibraltar Straits. The latter is responsible for all entering/outgoing fluxes from/to Atlantic Ocean.

Anyway, I will try to play on this to see how results change.

[arango]

Seems that there is a problem of semantics here. The above statement about grid configuration cannot be taken literally. I am not implying that you need to close your open boundaries Experienced ocean modelers know that open boundary conditions are troublesome. What I meant was to avoid making your application even harder by putting the open boundary in the most inappropriate place. Sometimes, applications improve tremedously by moving the open boundary by few grid points. For example, avoid putting a canyon right at the boundary. You have the choice of including the canyon by making your grid larger or removing it by making your grid smaller. The same applies for dynamic regimes.

To get good results in your application, you need to provide values at the open boundary for the inflow. These values can be taken from a larger scale model. There are options in ROMS for one- and two-way nesting. We have experimented with one-way nesting along the US East coast and ran our applications successfully for several simulation years. This application has open boundary condition at the Gulf Stream and Loop Current.

Good luck

[bru]

I forgot to say something about the blowing-up conditions:

On my example, the model runs perfectly at a resolution of 1/7.
But as soon as I try to increase the resolution, let's say 1/8, the model blows-up at a given time-step (although CFL is ok). After playing with time-stepping, I figured out that it always blows-up at the same exact node of the given mesh.
What's happening? Actually kinetic energy increases suddenly from one time-step to the next one (ex: from 3.3e-3 it turns to 4.6e-2 ... or from 3.4e-3 it becomes 2.1e3) and strangely enough, potential energy becomes negative (ex: from 2.53e2 to -1.6e-1 ... or from 2.53e2 to -3.29e19). The error message returned looks like: Vmax=0.65e21 at i(max)=194, j(max)=56 ...
Well, what is all that suppose to mean?
I understand and I have checked that velocity reaches a too high value at node (194,56), which implies a high kinetic energy. But what is going on with potential energy?

I was just wondering if someone has already had this kind of problem with resolution management. If it is the case for anyone, please let me know.
Thank you

[CMD]

Could someone tell me clearly what kind of condition cause the ROMS blow up ?
On my case , it runs about 140 days and the kinetic energy increase suddenly and then become -nan .
How many velocity will cause the ROMS blow up ? or any other kind of the condition .



thank you

[OcGaBy]

I'm new using ROMS.

I want to configure it for a small region of the South Pacific Coast of Mexico, and force it with tide and wind, the coast line include some bays.

I´m starting from easy to complex. I configure an application with 3 open boundaries and just the northern wall closed. I'm using FSCHAPMAN, M2FLATHER; M3GRADIENT and TGRADIENT in the open boundaries. And for now, I just want to propagate an analytical wave coming from the east open boundary.

I get it to work just fine with an analitical grid, but its blows up at the 3rd time step when I use the real grid. I don't think my bathymetry is steep...


Can some body tell me wich parameters I should check or any hint?

GaBy

[kate]

The model writes out these things during initialization:

Minimum barotropic Courant Number = 2.13066333E-02
Maximum barotropic Courant Number = 3.26983253E-01
Maximum Coriolis Courant Number = 8.30340206E-02

Maximum grid stiffness ratios: rx0 = 3.961050E-01 (Beckmann and Haidvogel)
rx1 = 1.634962E+01 (Haney)

That's about as big as you want the Beckmann and Haidvogel number to get. Both it and the Haney number are measures of bathymetric steepness and pressure gradient troubles. Perhaps others can say sometihng about the Haney number and the Courant numbers.

[OcGaBy]

I got my model to run for a while… It was a grid problem before. But it still blows up after 72 hrs… The kinetic energy grows exponentially do to velocity increase…

I have 3 open boundaries; I’m setting them as _FSCHAPMAN, _M2FLATHER and _M3GRADIENT.

I’m trying to propagate a stationary wave. I set it analytically in one of the open boundaries.

If I close two of the open boundaries it works fine…

Suggestions, please!


GaBy

[OcGaBy]

I quit with my intentions of using ROMS (unless for now) do to the next…

I tried to configure a local application for a very small region of the South Pacific coast of Mexico; I have already given some details about it…

I’m in contact with another ROMS user in South Carolina, she kindly helps me to check my configuration, and after a while we discover that ours ROMS just don’t work in the same way, even when we already interchange all the subroutines…

She can run my application just fine, but when I run it, whit same input files, same everything my one blows up because the kinetic energy goes to the sky after some days do to an increase of the velocity components…

I just want to know it this happens to somebody before, and why it can be…

I’m running roms-2.2 in a laptop Dell-Latitude D510 with Fedora Core 4, and I can happily run the roms examples like upwelling, etc, and some other idealized applications created by me…

Cheers
GaBy

[jcwarner]

I would like to offer a few suggestions:
You said that you were still having troubles after you 'interchanged all the subroutines .." Not sure exactly what that means, but I suggest that you make a zipfile, or tarfile, and send the whole package when you want someone else to try. That way you can be sure they are using excatly the same setup as you are.

I have tried in the past to help people based on exchanging 1,2, or several routines, and it can often end up that they changed something and just forgot to mention it. I have often seen setups with 2 applications defined, and so you can only find some of these things if they send the whole code. So now if someone wants me to see their code, I insist on a full zipfile or tarfile.

When the model stops running for me, and I really want to know why it died, I will restart it and save the restarts more often. It will blow up again, and so I will restart it again saving even more frequently. Eventually you can get to a point where it will restart and blow up in a few time steps. Then you can save every time step in the history file and perhaps see more clearly exactly why it blew up - was it on the boundary, was it the coastline where maybe a cell went dry, was it in the interior at a location of sharp bathy? etc etc etc.

Be careful with the restarts - when it blows up, the restart file saves the last time step before it died. This will be placed at time index 3 in the restart file. Sometimes you can look at those fields to help identify why the model blew up. But be careful when you restart after a blow up. Do NOT use nrrec = -1 to restart it, or it will restart at the bad time step. The value of nrrec =-1 says 'look in the restart file and use the most recent time step.' This will be time index 3 in the restart file. But you will not want this. These values are for the model blowing up. You need to use:
ncks -v ocean_time ocean_rst.nc
, or similar method, to identify the most recent 'good' index, this will be either a 1 or a 2. Then restart with nrrec =1 or nrrec =2.

Seeing that the model blows up after a length of time, there could also be issues of 32 vs 64 bit. I have seen that as well. It will run ok on a 64 bit machine, but die on a 32 bit. But this is often a sign that something else is wrong.

Another option is to deactivate certain features and see if the model still dies. This may help diagnose the problem.

Hope this offers some advice.
john

[kate]

Another question might be that of comilers. Did you try it optimized and not? Do they behave the same? How about limits? (ulimit -a or limit, depending). I set up and ran a small domain for Indonesian waters and it ran on my Mac, but failed to run for my students. I know it was the same code and same forcing files. They had a variety of Linux flavors and compilers and managed to get two different failure modes. I wasn't there long enough to track it down, however. This is why I love having access to more than one system and compiler - you might get more meaningful error messages from one than another, or you might decide that there is a compiler bug. In the five years I've been here, I've reported five compiler bugs for the IBM compiler - they even get fixed eventually.

Also, We use the Flather and Chapman as the 2-D boundary conditions as you do, but for 3-D, we've had better luck with a compination of radiation and boundary nudging. This requres some external values for the fields to be used when the phase speed at the edge is from the outsde to the inside. Daily values from a larger model work better than monthly Levitus climatology.