DYNAMICS OPTION: Eulerian Mass Coordinate forrtl: severe (174): SIGSEGV, segmentation fault occurred

[agpc]

Hi everyone,
When I run the coawst model it reported that:

Code: Select all

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source
coawstM            0000000003C4696A  Unknown               Unknown  Unknown
libpthread-2.17.s  00002B4CA7C955D0  Unknown               Unknown  Unknown
coawstM            000000000091D1D2  read_phypar_             3082  read_phypar.f90
coawstM            000000000078E749  inp_par_                   87  inp_par.f90
coawstM            00000000004426D9  ocean_control_mod          90  ocean_control.f90
coawstM            000000000041E93B  MAIN__                    373  master.f90
coawstM            0000000000418AA2  Unknown               Unknown  Unknown
libc-2.17.so       00002B4CA83CA495  __libc_start_main     Unknown  Unknown
coawstM            00000000004189A9  Unknown               Unknown  Unknown
   alloc_space_field: domain            1 ,               89554972  bytes allocated
Fatal error in MPI_Send: Other MPI error, error stack:
MPI_Send(173)..................: MPI_Send(buf=0xa06f380, count=4, MPI_INTEGER, dest=0, tag=2, MPI_COMM_WORLD) failed
MPIDI_EagerContigShortSend(262): failure occurred while attempting to send an eager message
MPIDI_CH3_iStartMsg(36)........: Communication error with rank 0

Can anyone help me to solve this problem?
The log files are included in this post.
I'm really appreciate that.
-Manh

[kate]

3082 read_phypar.f90

What is on this line in your code? You say you are using COAWST, but where did you get your roms.in file? Does it have lines for how many boundary and climatology files you have? NCLMFILES and NBCFILES, also NSSFFILES. These need to be non-zero.

[agpc]

Hi Kate,
The line in my code is "fname=BRY(i,ng)%files(ifile)", it is displayed clearly below:

Code: Select all

          IF (ObcData(ng)) THEN
            DO i=1,nBCfiles(ng)
              DO ifile=1,BRY(i,ng)%Nfiles
                fname=BRY(i,ng)%files(ifile)
                IF (.not.find_file(ng, fname, 'BRYNAME')) GO TO 30
                IF (ifile.eq.1) THEN
                  WRITE (out,230) '             Input Boundary File:  ',&

After I add the HOTFILE command in SWAN's namelist, it reported that:

Code: Select all

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 1 PID 67104 RUNNING AT compute-1-17
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

I aslo attach all of my namelist files of my project in this post.
Can you help me to address this problem?
Many thanks,
-manh

[jcwarner]

ok. you have this

NCLMFILES == 2 ! number of climate files
CLMNAME == Projects/Sarika4/coawst_clm_20161016.nc |
Projects/Sarika4/coawst_clm_20161017.nc
NBCFILES == 2 ! number of boundary files
! BRYNAME == Projects/Sandy/Sarika_bdy.nc
BRYNAME == Projects/Sarika4/coawst_bdy_20161016.nc |
Projects/Sarika4/coawst_bdy_20161017.nc |

But I think there are 2 problems here.
- IF the information in *clm_20161016 and *clm_20161017 are the same fields but with different dates, then you list as 1 file with a continuation symbol |.
- Also, for the BRY files, there should not be a | or any symbol after the last file name.
Give this a try:

NCLMFILES == 1 ! number of climate files
CLMNAME == Projects/Sarika4/coawst_clm_20161016.nc |
Projects/Sarika4/coawst_clm_20161017.nc
NBCFILES == 1 ! number of boundary files
! BRYNAME == Projects/Sandy/Sarika_bdy.nc
BRYNAME == Projects/Sarika4/coawst_bdy_20161016.nc |
Projects/Sarika4/coawst_bdy_20161017.nc

[agpc]

Hi John,
Thank you for your help. I changed my ROMS's namelist file following your suggestion, however it still errored:

Code: Select all

DYNAMICS OPTION: Eulerian Mass Coordinate

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 1 PID 75646 RUNNING AT compute-1-17
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   RANK 2 PID 75647 RUNNING AT compute-1-17
=   KILLED BY SIGNAL: 9 (Killed)
===================================================================================

[jcwarner]

there is not enough information for me to determine what is going wrong.
You need to dig into the full output being written.
Did SWAN give Err* files? Maybe try to run the system with one model at a time, to see if each model can handle your new application. then slowly add each component.

[agpc]

Hi John,
In this post, I give you all of my output files.
I hope it is enough to determine what is going wrong.
Thank you very much!

[jcwarner]

in the *.err file we have:
mage PC Routine Line Source
coawstM 0000000003C4696A Unknown Unknown Unknown
libpthread-2.17.s 00002AAF329CD5D0 Unknown Unknown Unknown
coawstM 000000000092471A edit_file_struct_ 212 edit_multifile.f90
coawstM 000000000090A307 read_phypar_ 2198 read_phypar.f90
coawstM 000000000078E749 inp_par_ 87 inp_par.f90
coawstM 00000000004426D9 ocean_control_mod 90 ocean_control.f90
coawstM 000000000041E93B MAIN__ 373 master.f90
coawstM 0000000000418AA2 Unknown Unknown Unknown
libc-2.17.so 00002AAF33102495 __libc_start_main Unknown Unknown
coawstM 00000000004189A9 Unknown Unknown Unknown

so this is a problem ROMS reading the ocean.in file, and maybe opening up some files. i think this was an error earlier, so it has not been solved.
Can you step through read_phypar.f90 line 2198 and see what files it is trying to open, and see what edit_multifile.f90 line 212 is trying to do? this will help solve the problem.

[agpc]

HI John,
The line 2198 of read_phypar.f90 is:

Code: Select all

CALL edit_file_struct (ng, OutFiles, QCK)

The line 212 of edit_multifile.f90 is "end if" which located below:

Code: Select all

        IF (associated(S(ng)%files))    deallocate (S(ng)%files)
!
        allocate ( S(ng)%Nrec(Nfiles) )
        allocate ( S(ng)%time_min(Nfiles) )
        allocate ( S(ng)%time_max(Nfiles) )
        allocate ( S(ng)%files(Nfiles) )
      END IF

[jcwarner]

ok. so this is looking for the Quick output file writes.
Your ocean.in has
! NQCK == 0
! NDEFQCK == 0
lets try it with this:
NQCK == 0
NDEFQCK == 0
and see if that works.

[agpc]

Hi John,
Thank you for your suggestion. I changed my ROMS namelist file and the problem was solved.
However, I had another issue is that:

Code: Select all

 MULTIFILE_INFO_S2D - Error while processing lateral boundary multi-files: 
                      data does not include initialization time = 2174-09-15 00:00:00.00

   ****-**-** 00:00:00.00  ****-**-** 00:00:00.00     Projects/Sarika/merged_coawst_bdy.nc
 Found Error: 04   Line: 484      Source: ROMS/Utility/check_multifile.F
 Found Error: 04   Line: 111      Source: ROMS/Utility/check_multifile.F
 Found Error: 04   Line: 846      Source: ROMS/Nonlinear/initial.F
 Found Error: 04   Line: 211      Source: ROMS/Drivers/nl_ocean.h

Code: Select all

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

 ROMS/TOMS - Output NetCDF summary for Grid 01:
 Found Error: 05   Line: 443      Source: ROMS/Utility/close_io.F

 ROMS/TOMS - Configuration error ..... exit_flag:   5


 ERROR: Illegal model configuration.
 == SWAN grid  1 sent wave data to ROMS grid  1

I check the line 443 of the close_io.F program is " IF (Master.and.(FoundError(exit_flag, NoError, __LINE__, &"
This line is located in the below:

Code: Select all

      IF (Master.and.(FoundError(exit_flag, NoError, __LINE__,          &
     &               __FILE__))) THEN
        WRITE (stdout,40) Rerror(exit_flag), exit_flag
      END IF
      IF (blowup.ne.0) THEN
        IF (Master) WRITE (stdout,50) TRIM(blowup_string)
      ELSE IF (exit_flag.eq.NoError) THEN

Please help me to tackle this problem. I'm really appreciate that!
-Manh,

[jcwarner]

looks to me like something is wrong with "time". the error is:

"MULTIFILE_INFO_S2D - Error while processing lateral boundary multi-files:
data does not include initialization time = 2174-09-15 00:00:00.00
****-**-** 00:00:00.00 ****-**-** 00:00:00.00 Projects/Sarika/merged_coawst_bdy.nc"

Your input files for lateral boundary conditions are starting with a time in the year 2174, .... that is probably wrong.
Then, the model time is not even being displayed accurately, it says ****-**-** 00:00:00.00 . that means the number is too big to be displayed with that formatting.

So first i suggest you look at the information for the time variables in the boundary netcdf files. you can do a
ncdump -h merged_coawst_bdy.nc
and see what that shows for the v3dtime etc.
Then you need to compare that with the way you are running ROMS, using DSTART and TIME_REF. there is information at the bottom of the ocean.in to help with those settings.

[agpc]

Hi John,
Here is my bdy.nc file:

Code: Select all

[cloud@igplogin Sarika]$ ncdump -c merged_coawst_bdy.nc 
netcdf merged_coawst_bdy {
dimensions:
	xpsi = 299 ;
	xrho = 300 ;
	xu = 299 ;
	xv = 300 ;
	epsi = 219 ;
	erho = 220 ;
	eu = 220 ;
	ev = 219 ;
	s_rho = 16 ;
	zeta_time = 2 ;
	v2d_time = 2 ;
	v3d_time = 2 ;
	salt_time = 2 ;
	temp_time = 2 ;
variables:
	double zeta_time(zeta_time) ;
		zeta_time:long_name = "zeta_time" ;
		zeta_time:units = "days" ;
		zeta_time:field = "zeta_time, scalar, series" ;
	double v2d_time(v2d_time) ;
		v2d_time:long_name = "v2d_time" ;
		v2d_time:units = "days" ;
		v2d_time:field = "v2d_time, scalar, series" ;
	double v3d_time(v3d_time) ;
		v3d_time:long_name = "v3d_time" ;
		v3d_time:units = "days" ;
		v3d_time:field = "v3d_time, scalar, series" ;
	double salt_time(salt_time) ;
		salt_time:long_name = "salt_time" ;
		salt_time:units = "days" ;
		salt_time:field = "salt_time, scalar, series" ;
	double temp_time(temp_time) ;
		temp_time:long_name = "temp_time" ;
		temp_time:units = "days" ;
		temp_time:field = "temp_time, scalar, series" ;
	double zeta_south(zeta_time, xrho) ;
		zeta_south:long_name = "free-surface southern boundary condition" ;
		zeta_south:units = "meter" ;
		zeta_south:field = "zeta_south, scalar, series" ;
	double zeta_east(zeta_time, erho) ;
		zeta_east:long_name = "free-surface eastern boundary condition" ;
		zeta_east:units = "meter" ;
		zeta_east:field = "zeta_east, scalar, series" ;
	double zeta_west(zeta_time, erho) ;
		zeta_west:long_name = "free-surface western boundary condition" ;
		zeta_west:units = "meter" ;
		zeta_west:field = "zeta_west, scalar, series" ;
	double zeta_north(zeta_time, xrho) ;
		zeta_north:long_name = "free-surface northern boundary condition" ;
		zeta_north:units = "meter" ;
		zeta_north:field = "zeta_north, scalar, series" ;
	float ubar_south(v2d_time, xu) ;
		ubar_south:long_name = "2D u-momentum southern boundary condition" ;
		ubar_south:units = "meter second-1" ;
		ubar_south:field = "ubar_south, scalar, series" ;
	float ubar_east(v2d_time, eu) ;
		ubar_east:long_name = "2D u-momentum eastern boundary condition" ;
		ubar_east:units = "meter second-1" ;
		ubar_east:field = "ubar_east, scalar, series" ;
	float ubar_west(v2d_time, eu) ;
		ubar_west:long_name = "2D u-momentum western boundary condition" ;
		ubar_west:units = "meter second-1" ;
		ubar_west:field = "ubar_west, scalar, series" ;
	float ubar_north(v2d_time, xu) ;
		ubar_north:long_name = "2D u-momentum northern boundary condition" ;
		ubar_north:units = "meter second-1" ;
		ubar_north:field = "ubar_north, scalar, series" ;
	float vbar_south(v2d_time, xv) ;
		vbar_south:long_name = "2D v-momentum southern boundary condition" ;
		vbar_south:units = "meter second-1" ;
		vbar_south:field = "vbar_south, scalar, series" ;
	float vbar_east(v2d_time, ev) ;
		vbar_east:long_name = "2D v-momentum eastern boundary condition" ;
		vbar_east:units = "meter second-1" ;
		vbar_east:field = "vbar_east, scalar, series" ;
	float vbar_west(v2d_time, ev) ;
		vbar_west:long_name = "2D v-momentum western boundary condition" ;
		vbar_west:units = "meter second-1" ;
		vbar_west:field = "vbar_west, scalar, series" ;
	float vbar_north(v2d_time, xv) ;
		vbar_north:long_name = "2D v-momentum northern boundary condition" ;
		vbar_north:units = "meter second-1" ;
		vbar_north:field = "vbar_north, scalar, series" ;
	float u_south(v3d_time, s_rho, xu) ;
		u_south:long_name = "3D u-momentum southern boundary condition" ;
		u_south:units = "meter second-1" ;
		u_south:field = "u_south, scalar, series" ;
	float u_east(v3d_time, s_rho, eu) ;
		u_east:long_name = "3D u-momentum eastern boundary condition" ;
		u_east:units = "meter second-1" ;
		u_east:field = "u_east, scalar, series" ;
	float u_west(v3d_time, s_rho, eu) ;
		u_west:long_name = "3D u-momentum western boundary condition" ;
		u_west:units = "meter second-1" ;
		u_west:field = "u_west, scalar, series" ;
	float u_north(v3d_time, s_rho, xu) ;
		u_north:long_name = "3D u-momentum northern boundary condition" ;
		u_north:units = "meter second-1" ;
		u_north:field = "u_north, scalar, series" ;
	float v_south(v3d_time, s_rho, xv) ;
		v_south:long_name = "3D v-momentum southern boundary condition" ;
		v_south:units = "meter second-1" ;
		v_south:field = "v_south, scalar, series" ;
	float v_east(v3d_time, s_rho, ev) ;
		v_east:long_name = "3D v-momentum eastern boundary condition" ;
		v_east:units = "meter second-1" ;
		v_east:field = "v_east, scalar, series" ;
	float v_west(v3d_time, s_rho, ev) ;
		v_west:long_name = "3D v-momentum western boundary condition" ;
		v_west:units = "meter second-1" ;
		v_west:field = "v_west, scalar, series" ;
	float v_north(v3d_time, s_rho, xv) ;
		v_north:long_name = "3D v-momentum northern boundary condition" ;
		v_north:units = "meter second-1" ;
		v_north:field = "v_north, scalar, series" ;
	float temp_south(temp_time, s_rho, xrho) ;
		temp_south:long_name = "3D temperature southern boundary condition" ;
		temp_south:units = "C" ;
		temp_south:field = "temp_south, scalar, series" ;
	float temp_east(temp_time, s_rho, erho) ;
		temp_east:long_name = "3D temperature eastern boundary condition" ;
		temp_east:units = "C" ;
		temp_east:field = "temp_east, scalar, series" ;
	float temp_west(temp_time, s_rho, erho) ;
		temp_west:long_name = "3D temperature western boundary condition" ;
		temp_west:units = "C" ;
		temp_west:field = "temp_west, scalar, series" ;
	float temp_north(temp_time, s_rho, xrho) ;
		temp_north:long_name = "3D temperature northern boundary condition" ;
		temp_north:units = "C" ;
		temp_north:field = "temp_north, scalar, series" ;
	float salt_south(salt_time, s_rho, xrho) ;
		salt_south:long_name = "3D salinity southern boundary condition" ;
		salt_south:units = "psu" ;
		salt_south:field = "salt_south, scalar, series" ;
	float salt_east(salt_time, s_rho, erho) ;
		salt_east:long_name = "3D salinity eastern boundary condition" ;
		salt_east:units = "psu" ;
		salt_east:field = "salt_east, scalar, series" ;
	float salt_west(salt_time, s_rho, erho) ;
		salt_west:long_name = "3D salinity western boundary condition" ;
		salt_west:units = "psu" ;
		salt_west:field = "salt_west, scalar, series" ;
	float salt_north(salt_time, s_rho, xrho) ;
		salt_north:long_name = "3D salinity northern boundary condition" ;
		salt_north:units = "psu" ;
		salt_north:field = "salt_north, scalar, series" ;

// global attributes:
		:history = "Created by updatclim on 16-Dec-2020 23:39:24" ;
		:type = "climate forcing file from http://hycom.coaps.fsu.edu:8080/thredds/dodsC/glb_analysis" ;
data:

 zeta_time = 4983379200, 4983465600 ;

 v2d_time = 4983379200, 4983465600 ;

 v3d_time = 4983379200, 4983465600 ;

 salt_time = 4983379200, 4983465600 ;

 temp_time = 4983379200, 4983465600 ;

I want to forecast during 48h of lead time begin at 20161016, why is my set-up going wrong?

Code: Select all

      DSTART =   2.0d0         ! days
  TIDE_START =   0.0d0                  ! days
    TIME_REF =  20161016.0d0               ! yyyymmdd.dd

Can you help me to address this issue.
Thank you.

[jcwarner]

what does this give

ncks -v v2d_time Projects/Sarika/merged_coawst_bdy.nc

[agpc]

Hi John,
This is my result:

Code: Select all

[cloud@igplogin Sarika]$ ncks -v v2d_time merged_coawst_bdy.nc
v2d_time: type NC_DOUBLE, 1 dimension, 3 attributes, compressed? no, chunked? no, packed? no
v2d_time size (RAM) = 2*sizeof(NC_DOUBLE) = 2*8 = 16 bytes
v2d_time dimension 0: v2d_time, size = 2 NC_DOUBLE (Coordinate is v2d_time)
v2d_time attribute 0: long_name, size = 8 NC_CHAR, value = v2d_time
v2d_time attribute 1: units, size = 4 NC_CHAR, value = days
v2d_time attribute 2: field, size = 24 NC_CHAR, value = v2d_time, scalar, series

v2d_time[0]=4983379200 days
v2d_time[1]=4983465600 days

[jcwarner]

so the problem is that the forcing time is at 4983379200 days.
that number is probably in seconds (not days),
4983379200/3600/24=57678
and that number is probably
julian(2016,10,1)-2400000=57678
So you need to change the values of time in that forcings file from
v2d_time[0]=4983379200 days
v2d_time[1]=4983465600 days
to
v2d_time[0]=57678 days
v2d_time[1]=57679 days
for all the vars in the bdy and init and clm.

For the ocean model iteself, i am not sure what that is doing. you need to look closely at what it writes in the output file.

[agpc]

Hi John,
Thank you for your suggestion.
I had changed my time vars in my clm.nc file to:

Code: Select all

 ocean_time = 57678, 57679 ;

 zeta_time = 57678, 57679 ;

 v2d_time = 57678, 57679 ;

 v3d_time = 57678, 57679 ;

 salt_time = 57678, 57679 ;

 temp_time = 57678, 57679 ;

and my bdy.nc file to:

Code: Select all

 zeta_time = 57678, 57679 ;

 v2d_time = 57678, 57679 ;

 v3d_time = 57678, 57679 ;

 salt_time = 57678, 57679 ;

 temp_time = 57678, 57679 ;

The ocean_time in Sarika_ini.nc file to:

Code: Select all

ocean_time = 57678 ;

The result is that my problem is solved.
Although I saw the line

NLM: GET_STATE - Reading state initial conditions, 2174-09-16 00:00:00.00
(Grid 01, t = 57678.0000, File: Sarika_ini.nc, Rec=0001, Index=1)

my model still ran, isn't it?
I also attached my log file in this post.
thank you very much!

[jcwarner]

getting close, but it still looks like something is wrong with the ocean times.
what is the ocean_time in the ocean ini netcdf file?
and what are the attributes for that (ncdump and ncks the ocean_time in the ini file).
these need to be consistent with the dstart and time_ref.

try to figure it out.
-j

[agpc]

Hi John,
Finally, I can fix it.
Thank you very much!
-Manh