segmentation fault when running the COAWST model

[EdwardElric]

hello everyone
In recent day，I use the COAWST model （couple the ROMS and SWAN model）to simulate sediment transport. Before this ，I successfully run the Sed_floc_toy case. But when I actually run the real case, it report a segmentation fault in the very beginning. The problem is described in the below:

Model Input Parameters: ROMS/TOMS version 3.7
Sunday - April 11, 2021 - 6:42:33 PM
--------------------------------------------------------------------------------
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
coawstG 0000000000A8DC4A inp_decode_mod_mp 1629 inp_decode.f90
coawstG 000000000098F3B9 read_phypar_ 1557 read_phypar.f90
coawstG 00000000008FEE01 inp_par_ 66 inp_par.f90
coawstG 000000000040B365 ocean_control_mod 73 ocean_control.f90
coawstG 000000000040B0EF MAIN__ 73 master.f90
coawstG 000000000040AF8C Unknown Unknown Unknown
libc.so.6 0000003D7961ED20 Unknown Unknown Unknown
coawstG 000000000040AE89 Unknown Unknown Unknown

Then I check the inp_decode.F， read_phypar.F and the inp_par.F file.The corresponding code is show as follow：
inp_decode.F line 1629:(which is blank)

!
! If appropriate, increase or reset nested grid counter.
!
IF ((Icont.gt.0).and.(Ngrids.gt.1)) THEN
IF ((iTrcStr.gt.0).and.(iTrcEnd.gt.0)) THEN
IF ((ifield.eq.isTvar(iTrcEnd)).or.(ic.gt.1)) THEN
igrid=igrid+MIN(1,Icont)
END IF
ELSE

read_phypar.F line1557

CASE ('Hout(idVbws)')
IF (idVbws.eq.0) THEN
IF (Master) WRITE (out,280) 'idVbws'
exit_flag=5
RETURN
END IF

inp_par.F line 66

#ifdef ROMS_STDOUT
!
! Change default Fortran standard out unit, so ROMS run information is
! directed to a file. This is advantageous in coupling applications to
! ROMS information separated from other models.
!
stdout=66 ! overwite Fortran default unit 6
!
IF (Master) THEN
OPEN (stdout, FILE='log.roms', FORM='formatted', &
& STATUS='replace')
END IF
#endif

I also try to turn on the idVbws and the related switch (idUbws idVbrs idUbrs) in my_case.in file
but I still report the same segmentation fault
I do have no idea about this, Can anyone give me some suggestions or ideas?
thanks very much!
-Edward

[jcwarner]

couple of things.
the error are asking about the Build/*.f90 files, not the .F.

can you look at the correct lines in the .f90 's ?

1629 inp_decode.f90
1557 read_phypar.f90

[EdwardElric]

hi john
thanks for your reply so quickly！
the line 1629 in inp_decode.f90 is follow:

! Accumulate number of multi-files per each grid.
!
Ncount(ifile,igrid)=Ncount(ifile,igrid)+1

and the line 1557 of read_phypar.f90 is：

CASE ('BRYNAME')
label='BRY - lateral open boundary conditions'
Npts=load_s2d(Nval, Cval, Cdim, line, label, ibcfile, igrid, Ngrids, nBCfiles, NBCcount, max_Ffiles, BRY)

thank you！

[jcwarner]

it probably has to do with reading the names or numbers of boundary files.
make sure you do not have NBCFILES listed twice or something like that.

[kate]

Make sure NBCFILES is defined. Some roms.in files have it, some do not. If using COAWST, you need to have it.

[EdwardElric]

it probably has to do with reading the names or numbers of boundary files.
make sure you do not have NBCFILES listed twice or something like that.

yep， I state the nbcfiles in the .in file and it actually works ！
thank you so much！

[EdwardElric]

Make sure NBCFILES is defined. Some roms.in files have it, some do not. If using COAWST, you need to have it.

hello Kate
I state the nbcfiles in my .in file and it actually works. thank you so much!

But here comes a new error: the output-log reports that it can not fine the grain_diameter in mycase-ini.nc

NLM: CHECKVARS - unable to find model variable: grain_diameter in file: mycase-ini.nc
Found Error: 02 Line: 248 Source: ROMS/Utility/get_state.F
Found Error: 02 Line: 585 Source: ROMS/Nonlinear/initial.F
Found Error: 02 Line: 200 Source: ROMS/Drivers/nl_ocean.h

However, I just want to run a hydrodynamic model so I do not defined SEDIMENT in my .h file and I do not put sediment in my .in file either

Nbed = 0 ! Number of sediment bed layers
NAT = 2 ! Number of active tracers (usually, 2)
NPT = 0 ! Number of inactive passive tracers
NCS = 0 ! Number of cohesive (mud) sediment tracers
NNS = 0 ! Number of non-cohesive (sand) sediment tracers

Why does this happen? Is the sediment distribution must be declared in the initial file in the COAWST? (no matter if it is used or not

[jcwarner]

the sediment is required if you set SSW_BBL or any other BBL.
you can use ana_sediment and set some basic properties in there.
-j

[EdwardElric]

hello John
I do define the SSW_BBL : j. I follow your suggestion and add the # define ANA_SEDIMENT below the cpp option

Code: Select all

#define SSW_BBL
#ifdef SSW_BBL
# define ANA_SEDIMENT
# define SSW_CALC_ZNOT
# undef SSW_LOGINT
# undef SSW_CALC_UB 
# undef SSW_FORM_DRAG_COR 
# undef SSW_ZOBIO 
# undef SSW_ZOBL 
# undef SSW_ZORIP 
#endif

But when I make the bash file again it reports the follow error:

analytical.f90(181): error #5082: Syntax error, found ':' when expecting one of: % . = =>
ana_sediment.h: no values provided for bottom(:,:,isd50) and
--------------------^
analytical.f90(182): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ . = =>
bottom(:,:,idens))

where can I provide these values? or should I prepared a sediment.in?

(By the way, I compared with the Sandy case , and I find it also # define ANA_SEDIMENT and do not declare sediment.in in the ocean_sandy.in file. How does this case work?

thank you
-Edward

[jcwarner]

so the ana_sediment and sediment.in are different files.

If you need sediment, then you can provide initial information in a netcdf file, or by using ana_sediment.
- Suggest you copy ana_sediment from the Sandy or inlet_test case, and move that file to the location of your project.h file.
-Edit the ana_sediment and include your project definition in there. for example, search the inlet test one for inlet_test and see how that sets the initial values.
# if defined INLET_TEST
DO j=JstrR,JendR
DO i=IstrR,IendR
bottom(i,j,isd50)=0.0005_r8
...
but you will change 'INLET_TEST' to your project name.
-In your coawst. bash, you need to set the locations of your analytical files.
export MY_HEADER_DIR=${MY_PROJECT_DIR}/Projects/Inlet_test/Coupled
export MY_ANALYTICAL_DIR=${MY_PROJECT_DIR}/Projects/Inlet_test/Coupled

-when it builds it will find the files and include the correct sections of code.
-if you have #undef SEDIMENT you should be done.
- if you have #define SEDIMENT then you need to include a sediment.in file, with all that info.

hope that helps
-j

[EdwardElric]

thank you john , it really helps a lot！
I copy ana_sediment.h to my Project and edit it. The model can run a bit, but it still report an error
I use oceanM and the output.log reports as follow:

NLM: GET_STATE - Reading state initial conditions, 2010-01-01 00:00:00.00
(Grid 01, t = 0.0000, File: mycase-ini.nc, Rec=0001, Index=1)
- free-surface
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated u-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated v-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
--------------------------------------------------------------------------
mpirun noticed that process rank 10 with PID 31701 on node 168 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
4 total processes killed (some possibly by mpirun during cleanup)

it seems that there is still some error in reading the mycase-ini.nc
thank you!
-Edward

[jcwarner]

good that you are making progress. but at this point, there is not much information there to help.
look through that entire log file, are there any NaN's? does the data look correct?
use ncview or ncdump or matlab or anything to look at the fields in your "mycase-ini.nc" file.

maybe start out with a setup that has very limited number of process options. then add physics later. just get a base case to run.
-j

[EdwardElric]

good that you are making progress. but at this point, there is not much information there to help.
look through that entire log file, are there any NaN's? does the data look correct?
use ncview or ncdump or matlab or anything to look at the fields in your "mycase-ini.nc" file.

maybe start out with a setup that has very limited number of process options. then add physics later. just get a base case to run.
-j

hi john
I check my log file and do not see any NaN in it. All things go well before it reading state initial conditions, and the model broke up suddenly

The input files（including the initial file has been used in ROMS model for a long time. So it's almost impossible that my initial file goes wrong. So I supposed that the problem may be the difference between ROMS and COAWST model ?

In case I miss anything, I put my log files in the attachment.If you have time, you could check it and see what is going wrong
thank you
-Edward

[jcwarner]

not sure.
if you are using COAWST, make sure you start with a COAWST ocean input file. Compare yours to the one in Inlet Test or Sandy or something like that. We have a few differences to Rutgers, such as NBCfiles and NCLmfiles, etc.

Another thing to try is to compile in debug=on and that gives a "G" file to run.

[EdwardElric]

not sure.
if you are using COAWST, make sure you start with a COAWST ocean input file. Compare yours to the one in Inlet Test or Sandy or something like that. We have a few differences to Rutgers, such as NBCfiles and NCLmfiles, etc.

Another thing to try is to compile in debug=on and that gives a "G" file to run.

Hello John
So you mean that I should try to create a new input file in COAWST type and test if it can work?
Is there a website with scripts that transform our original data to input files? (just like //www.myroms.org/wiki/Matlab_Scripts
First I think it could find these in Sandy case. But I check the file below Sandy and can not find such scripts
thank you
-Edward

[jcwarner]

it is not that big of an effort. Just do a diff of your ocean.in to lets say the Sandy ocean.in. that is all. It will probably not be the issue.

Suggest you compile in debug mode, and run that.

[EdwardElric]

it is not that big of an effort. Just do a diff of your ocean.in to lets say the Sandy ocean.in. that is all. It will probably not be the issue.

Suggest you compile in debug mode, and run that.

hello john
I follow your idea and run it in debug mode. However, weird things happen:

when I run in coawstM the oouput log is

NLM: GET_STATE - Reading state initial conditions, 2010-01-01 00:00:00.00
(Grid 01, t = 0.0000, File: mycase-ini.nc, Rec=0001, Index=1)
- free-surface
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated u-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated v-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
--------------------------------------------------------------------------
mpirun noticed that process rank 15 with PID 5358 on node 168 exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
5 total processes killed (some possibly by mpirun during cleanup)
"op3.log" 533L, 29905C

I once thought that it may be the problem of mycase-ini.nc
but when I run in coastG, the model can read the mycase-ini.nc smoothly

NLM: GET_STATE - Reading state initial conditions, 2010-01-01 00:00:00.00
(Grid 01, t = 0.0000, File: mycase-ini.nc, Rec=0001, Index=1)
- free-surface
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated u-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated v-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- u-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- v-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- potential temperature
(Min = 1.59813415E-02 Max = 2.52891937E+01)
- salinity
(Min = 2.90392214E+01 Max = 3.49541981E+01)

that means my ini.nc have no problems. but what's the difference between two cases?
thank you
-Edward