Boundary forcing error (t3dbc_im)

[skastner]

Hello friendly ROMS forum users,

I am having an issue with my tracer boundary read-in as time stepping is about to start. My simulation is intended to be a realistic representation of Bahia Almirante, a bay in Panama. I've interpolated HYCOM u, v, T, S, and sea surface height onto the boundaries of the modeling domain to use as boundary forcing. My south boundary is closed, while the east, west, and north boundaries use the radiation with nudging condition.

I get the following error:

Code: Select all

[hpc3-19-13:4901 :0:4901] Caught signal 8 (Floating point exception: floating-point invalid operation)
==== backtrace (tid:   4901) ====
 0 0x0000000000054bf5 ucs_debug_print_backtrace()  /export/repositories/buildlibs-admix/BUILD/ucx_1.8.1-1.8.1/ucx-1.8.1/src/ucs/debug/debug.c:653
 1 0x000000000166ad3f __t3dbc_mod_MOD_t3dbc_tile()  /data/homezvol0/skastner/ROMS/trunk/projects/almirante/alpha/Build_romsG/t3dbc_im.f90:301
 2 0x000000000087bd55 __pre_step3d_mod_MOD_pre_step3d_tile()  /data/homezvol0/skastner/ROMS/trunk/projects/almirante/alpha/Build_romsG/pre_step3d.f90:901
 3 0x000000000088c008 __pre_step3d_mod_MOD_pre_step3d()  /data/homezvol0/skastner/ROMS/trunk/projects/almirante/alpha/Build_romsG/pre_step3d.f90:108
 4 0x0000000000778b9b __rhs3d_mod_MOD_rhs3d()  /data/homezvol0/skastner/ROMS/trunk/projects/almirante/alpha/Build_romsG/rhs3d.f90:76
 5 0x000000000041837d main3d_()  /data/homezvol0/skastner/ROMS/trunk/projects/almirante/alpha/Build_romsG/main3d.f90:245
 6 0x000000000040a233 __ocean_control_mod_MOD_roms_run()  /data/homezvol0/skastner/ROMS/trunk/projects/almirante/alpha/Build_romsG/ocean_control.f90:192
 7 0x0000000000406a12 MAIN__()  /data/homezvol0/skastner/ROMS/trunk/projects/almirante/alpha/Build_romsG/master.f90:108
 8 0x0000000000406baf main()  /data/homezvol0/skastner/ROMS/trunk/projects/almirante/alpha/Build_romsG/master.f90:50
 9 0x0000000000022555 __libc_start_main()  ???:0
10 0x0000000000405b89 _start()  ???:0
=================================

after the last few lines of the log give:

Code: Select all

GET_NGFLD   - salinity northern boundary condition,            2019-09-10 00:00:00.00
                   (Grid= 01, Rec=2147, Index=1, File: almirante_bound.nc)
                   (Tmin=       6912.5000 Tmax=       7593.2500)      t =       7192.0000
                   (Min =  3.48235054E+01 Max =  3.68154144E+01)

 TIME-STEP YYYY-MM-DD hh:mm:ss.ss  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME
                     C => (i,j,k)       Cu            Cv            Cw         Max Speed

         0 2019-09-10 00:00:00.00  1.478048E-19  6.262519E+03  6.262519E+03  2.368591E+12
                     (001,350,27)  4.312984E-11  2.302341E-11  0.000000E+00  2.151831E-08
      DEF_HIS     - creating  history      file, Grid 01: /dfs2/davis/skastner/Bocas_Modeling/output/alpha/almirante_his_0001.nc
      WRT_HIS     - wrote history     fields (Index=1,1) in record = 1

There are similar errors in lines 191 (west boundary), 301 (east boundary), and 521 (north boundary) of t3dbc_im that occur simultaneously (I've parallelized the model).

These are all similar lines of code, with varying indexing depending on the boundary:

Code: Select all

IF (LBC(iwest,isTvar(itrc),ng)%nudging) THEN
                  IF (LnudgeTCLM(itrc,ng)) THEN
                    obc_out=CLIMA(ng)%Tnudgcof(Istr-1,j,k,ic)
                    obc_in =obcfac(ng)*obc_out
                  ELSE
                    obc_out=Tobc_out(itrc,ng,iwest)
                    obc_in =Tobc_in (itrc,ng,iwest)
                  END IF
                  IF ((dTdt*dTdx).lt.0.0_r8) THEN
                    tau=obc_in
                  ELSE
                    tau=obc_out
                  END IF
                  tau=tau*dt(ng)
                END IF

The line where the error occurs is: IF ((dTdt*dTdx).lt.0.0_r8) THEN

My initial thought was that I had NaNs in my boundary forcing file, causing dTdt and/or dTdx to contain NaNs, but I have quintuple checked this at this point; there are none.

Has anyone seen anything like this before? I haven't found any similar forum posts. I'm using ROMS v3.9. I'm uploading my log, error, .in, and grid files for reference. The boundary forcing file is large, but I could provide it if needed.

Thanks in advance for your help,

Sam Kastner
Postdoctoral Scholar
University of California-Irvine

[kate]

I would look at this in a debugger, but lacking that, try print statements. You know it's something to do with dTdt or dTdx, so print them and the things that go into them. It's happening early enough in the run that this shouldn't be too insane. Print the i,j,k values so you know which point(s) are bad.

[skastner]

Thanks, Kate. The print commands were helpful; it seems like the tracer fields at nout=3 are all NaNs (at least along the edges), despite no NaNs in any input files. It seems like I may want to turn to a debugger here, and there isn't one on the cluster I'm using as far as I can tell. Would you still recommend xxgdb as a free debugger?

I'm attaching the log file.

[kate]

If you can get xxgdb to work, wonderful! I would expect it to be fine as long as your job fits onto one core/process.

[wilkin]

Your log file indicates Tracer Affection Schemes are undefined. You might want to fix that first.

You have a very large value for OBCFAC > 500. That's an unconventional choice that will lead to very different OBC behavior when the radiation switches sign. I suggest a smaller nudging time scale so that nudging does something on outflow and a smaller OBCFAC so it doesn't do too much on inflow.

Your initial condition temperature minimum is -3. Odd. And open boundary min temps are 0. Also not very physical for this location.

[skastner]

Thanks for catching those abnormalities, John, especially the undefined tracer advection scheme. This was surprising to me, as I include

Code: Select all

#define TS_MPDATA

in my .h file. I'm therefore a little perplexed as to why the advective scheme is undefined in the log file--i'm attaching my .h & .in files here in case you wanted to take a look.

I have adjusted the nudging time scale and OBCFAC by a factor of two in the .in file here, to no change in where the model crashes (unsurprising given the advective scheme issue). They're still rather large, though. My intent was to choose a very long outflow nudging time scale, while still nudging the inflow on the time scale of the 3-hourly HYCOM output I'm using as boundary forcing. Would you suggest further reducing the nudging time scales?

I've also adjusted the boundary temperature fields. Low values are to be expected, as the domain extends far enough offshore to reach 2000m depth, but values less than zero were erroneous interpolations.

Thanks again.

[jivica]

in recent versions of ROMS we have tracer advection scheme defined inside ocean.in file.
the best is to have a look at the example in the Include folder

cheers
Ivica

[skastner]

Thanks, Ivica. That was indeed the problem! Lesson learned.

[pmaccc]

Sam,

I'm having a very similar problem to the one you describe, I think. When ROMS reads my boundary forcing for SSH the min values are zero (in the log) even though I can find no zeros in the bry.nc file. What was it you did to solve your problem?

Thanks,

Parker