Segmentation Fault Upwelling Example

[aryansafaie]

Dear ROMS community,

I recently installed and compiled ROMS on my Mac (64-bit Yosemite, GNU 3.8.1);
I followed (mostly) the tutorial here: https://www.myroms.org/wiki/index.php/ROMS_Mac

After compiling ROMS with ./build.bash and getting an oceanS file in my Projects/Upwelling directory,
I entered

Code: Select all

./oceanS < ocean_upwelling.in

.
Ultimately, the terminal ended with the following:

DEF_HIS - creating history file, Grid 01: ocean_his.nc
Segmentation fault: 11

I've checked the other forums for a solution to this topic, but I would GREATLY appreciate any help.

Thank you very much.

[kate]

Many things can cause a seg fault. The first thing I would do is check your system limits to make sure the stacksize isn't ridiculously small. If it is, change it and try again. If it isn't, recompile with USE_DEBUG then run in the gdb debugger to see what you see. That should give you a line number and file name for where it gets into trouble and should be at least possible since you are running serially.

[aryansafaie]

Hi Kate,

Thanks for getting back to me.
My stack size is 65532 KB;

when I enter the following in the terminal:

Code: Select all

gdb ./oceanS < ocean_upwelling.in

the terminal prints out many (worrisome) messages. I've attached them as a txt file if that helps.
However, there doesn't seem to be any mention of a segfault,
and I'm not sure what to address next.

Any further guidance will be greatly appreciated. Thanks again!

Best,
-aryan

[kate]

Ah, you have to type:

gdb ./oceanS

and then at the prompt:

run < ocean_upwelling.in

[aryansafaie]

Hi Kate,

Thanks again for being so speedy!
Pardon my novice ineptitude running gdb.

Ok, this time, after running it properly,
I received the following (I'm pasting the last few lines from the terminal)

Code: Select all

   STEP   Day HH:MM:SS  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME
          C => (i,j,k)       Cu            Cv            Cw         Max Speed

      0     0 00:00:00  0.000000E+00  6.585677E+02  6.585677E+02  3.884376E+11
            (00,00,00)  0.000000E+00  0.000000E+00  0.000000E+00  0.000000E+00
      DEF_HIS   - creating history file, Grid 01: ocean_his.nc

Program received signal SIGSEGV, Segmentation fault.
0x000000010047399c in ?? ()

Does that last row of 0.000000E+00 have anything to do with it, or
is the problem something to do with my ocean_upwelling file?

Thanks again,
-aryan

[kate]

Can you make your stacksize limit larger (like unlimited) and try again?

The line of zeroes is from your motionless initial condition.

If you are still in gdb at the end, try the "where" command.

I got this gem:

Code: Select all

 NETCDF_CREATE - unable to create output NetCDF file:
                 
                 call from:  wrt_his.F
 NetCDF: Memory allocation (malloc) failure

and "where" didn't work because the program had already exited.

[kate]

One thing you can do to the UPWELLING case to take up less memory is to change Lm in the ocean.in file to something smaller, like 10. It shouldn't change the results since things are invariant in the along-channel direction.

[aryansafaie]

Hi again,

Using the where command gives me the following:

Code: Select all

#0  0x000000010047399c in ?? ()
#1  0x000000005fbfca90 in ?? ()
#2  0x00007fff00000001 in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x00007fff5fbfcbbf in ?? ()
#5  0x0000000000000000 in ?? ()

I also tried lowering Lm to 10, but it made no difference.
Should I be using the -traceback flag (and where do I insert that), or is that already activated once USE_DEBUG is on?

Thanks
-aryan

[kate]

How about running:

Code: Select all

gdb ./oceanG

[aryansafaie]

Doing that results in:

Code: Select all

0x0000000100fd299c in ?? ()
(gdb) where
#0  0x0000000100fd299c in ?? ()
#1  0x000000005fbfa200 in ?? ()
#2  0x00007fff00000000 in ?? ()
#3  0x0000000000000001 in ?? ()
#4  0x0000000000000000 in ?? ()

I don't understand why I'm getting the ??; perhaps it's an issue with my directories?

[kate]

It should be giving you filenames and line numbers:

Code: Select all

(gdb) where
#0  0x000000000099581c in pre_step3d_mod::pre_step3d_tile (ng=1, tile=0, 
    lbi=-2, ubi=43, lbj=0, ubj=82, imins=-2, imaxs=44, jmins=-2, jmaxs=83, 
    nrhs=1, nstp=1, nnew=2, pm=<incomplete type>, pn=<incomplete type>, 
    hz=<incomplete type>, huon=<incomplete type>, hvom=<incomplete type>, 
    z_r=<incomplete type>, z_w=<incomplete type>, btflx=<incomplete type>, 
    bustr=<incomplete type>, bvstr=<incomplete type>, stflx=<incomplete type>, 
    sustr=<incomplete type>, svstr=<incomplete type>, akt=<incomplete type>, 
    akv=<incomplete type>, w=<incomplete type>, ru=<incomplete type>, 
    rv=<incomplete type>, diatwrk=<incomplete type>, 
    diau3wrk=<incomplete type>, diav3wrk=<incomplete type>, 
    diaru=<incomplete type>, diarv=<incomplete type>, t=<incomplete type>, 
    u=<incomplete type>, v=<incomplete type>) at pre_step3d.f90:374
#1  0x00000000009ad14f in pre_step3d_mod::pre_step3d (ng=1, tile=0)
    at pre_step3d.f90:110
#2  0x00000000005c06ef in rhs3d_mod::rhs3d (ng=1, tile=0) at rhs3d.f90:74
#3  0x000000000056e31f in main3d (runinterval=432000) at main3d.f90:241
#4  0x0000000000403ac6 in ocean_control_mod::roms_run (runinterval=432000)
    at ocean_control.f90:160
#5  0x00000000004030e5 in ocean () at master.f90:86
#6  0x0000000000403120 in main (argc=1, 
    argv=0x7fffffffd33a '/import/u/u1/uaf/kate/feedme/oceanG\000')
    at master.f90:50
#7  0x00002aaaabd4fd5d in __libc_start_main () from /lib64/libc.so.6
#8  0x0000000000402dd9 in _start ()

Can you recompile from a clean state and give us the final link command? I have:

Code: Select all

/usr/local/pkg/gcc/gcc-4.7.3/bin/gfortran -frepack-arrays -g -fbounds-check  Build_up/esmf_roms.o Build_up/master.o Build_up/ocean_control.o Build_up/ocean_coupler.o Build_up/propagator.o Build_up/roms_export.o Build_up/roms_import.o -o oceanG Build_up/libMODS.a Build_up/libNLM.a Build_up/libNLM_bio.a Build_up/libNLM_sed.a Build_up/libANA.a Build_up/libUTIL.a Build_up/libMODS.a -L/usr/local/pkg/netcdf/netcdf-4.3.0.gnu-4.7.3/lib -lnetcdff -L/usr/local/pkg/hdf5/hdf5-1.8.10-p1.gnu-4.7.3/lib -L/usr/local/pkg/szip/szip-2.1.gnu/lib -L/usr/local/pkg/udunits/udunits-2.1.24.gnu-4.7.3/lib -L/usr/local/pkg/netcdf/netcdf-4.3.0.gnu-4.7.3/lib -lnetcdf -lnetcdf

Make sure the "-g" is in there.

[aryansafaie]

Code: Select all

cd /Users/aryansafaie/ROMS/Projects/Upwelling/Build; /opt/local/bin/gfortran -c -frepack-arrays -mcmodel=medium -g -fbounds-check master.f90
/opt/local/bin/gfortran -frepack-arrays -mcmodel=medium -g -fbounds-check  /Users/aryansafaie/ROMS/Projects/Upwelling/Build/esmf_roms.o /Users/aryansafaie/ROMS/Projects/Upwelling/Build/master.o /Users/aryansafaie/ROMS/Projects/Upwelling/Build/ocean_control.o /Users/aryansafaie/ROMS/Projects/Upwelling/Build/ocean_coupler.o /Users/aryansafaie/ROMS/Projects/Upwelling/Build/propagator.o /Users/aryansafaie/ROMS/Projects/Upwelling/Build/roms_export.o /Users/aryansafaie/ROMS/Projects/Upwelling/Build/roms_import.o -o /Users/aryansafaie/ROMS/Projects/Upwelling/oceanG /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libUTIL.a /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM.a /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_bio.a /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libANA.a /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libUTIL.a /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libMODS.a -L/opt/local/lib -lnetcdf -lnetcdff
rm -f -r /Users/aryansafaie/make_macros.mk

I added that last line in there too.

[kate]

If you are running /Users/aryansafaie/ROMS/Projects/Upwelling/oceanG, it should have the symbols in there. I don't know. You can tell from:

Code: Select all

rv=<incomplete type>, diatwrk=<incomplete type>,

that gdb isn't smart about f90 structures, but it should know filenames. Perhaps it died in a non-Fortran part of the code (system calls or such).

[aryansafaie]

Hi Kate,

I was worried it would come to this; I have a feeling I did something iffy earlier on (in terms of preparing netcdf and fortran), and I remember some of the other forums making mention of different compilers giving different segfault errors.
I'll dig around some more and hopefully make a big step towards resolving this.

Thank you very much for all of your help so far! I certainly appreciate it, and my apologies for all the novice and annoying questions.

Sincerely,
-aryan

[aryansafaie]

Hi again,

This time I'm trying to run ROMS on my university's HPC cluster;
It compiled, and I have an oceanS executable.
Now when I run it with ocean_upwelling.in,

I get the following error: (I've copied and pasted the relevant part of the output):

Code: Select all

   STEP   Day HH:MM:SS  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME
          C => (i,j,k)       Cu            Cv            Cw         Max Speed

      0     0 00:00:00  0.000000E+00  6.585677E+02  6.585677E+02  3.884376E+11
            (00,00,00)  0.000000E+00  0.000000E+00  0.000000E+00  0.000000E+00
      DEF_HIS   - creating history file, Grid 01: ocean_his.nc

 NETCDF_CREATE - unable to create output NetCDF file:
                 ocean_his.nc
                 call from:  def_his.F

 DEF_HIS - unable to create history NetCDF file: ocean_his.nc

 Elapsed CPU time (seconds):

 Thread #  0 CPU:       0.060
 Total:                 0.060

 Nonlinear model elapsed time profile:

  Allocation and array initialization ..............         0.046  (76.6665 %)
  Ocean state initialization .......................         0.008  (13.3337 %)
  Computation of global information integrals ......         0.001  ( 1.6669 %)
  2D/3D coupling, vertical metrics .................         0.002  ( 3.3338 %)
  Omega vertical velocity ..........................         0.002  ( 3.3338 %)
                                              Total:         0.059   98.3348

 All percentages are with respect to total time =            0.060

 ROMS/TOMS - Output NetCDF summary for Grid 01:

 Analytical header files used:

     ROMS/Functionals/ana_btflux.h
     ROMS/Functionals/ana_grid.h
     ROMS/Functionals/ana_initial.h
     ROMS/Functionals/ana_smflux.h
     ROMS/Functionals/ana_stflux.h
     ROMS/Functionals/ana_vmix.h

 ROMS/TOMS - Output error ............ exit_flag:   3


 ERROR: Abnormal termination: NetCDF OUTPUT.
 REASON: Invalid argument 

I used "where" and "bt" in gdb, but received the "No stack." message:

Code: Select all

 [Inferior 1 (process 12627) exited normally]
(gdb) where
No stack.
(gdb) bt
No stack.
(gdb) 

I've browsed around the forums, and others have similar issues, but I'm not sure exactly what I can do.
Is it related to how running the model's initial step uses 60% of the total time?

Thank you again!
-aryan

[kate]

It is trying and failing to create the file ocean_his.nc. This happens before any actually time-stepping is done, so don't worry about the profile output. Why can't it create the file? Are you running in some directory where you don't have write permission for some reason? Is there a disk quota system where you already filled the quota?

Also, gdb is for when you don't know why a job is failing. In this case, ROMS does it's own checking and reporting, then it dies cleanly. You can set breakpoints to watch this happen in gdb, but I don't think it would be very helpful.

[aryansafaie]

Hello again Kate,

You were exactly right; the NetCDF directory on the HPC was in a directory I wasn't able to access yet, so I changed some things around and resolved that issue. Apologies for the side-issue; perhaps I should have posted that in another thread.

Getting back to the main issue at hand (Segmentation fault: 11), I've noticed a few things that may be related to the problem. If I can't resolve it after these, then I will try checking out ROMS again and starting over:

1) During the the ./build.bash process, I receive many output messages like the following (though the command executes to completion without exiting, and results in an oceanS or oceanG (if USE_DEBUG is on):

Code: Select all

ar: creating archive /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a
/opt/local/bin/ranlib: file: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a(sed_bed.o) has no symbols
/opt/local/bin/ranlib: file: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a(sed_bedload.o) has no symbols
/opt/local/bin/ranlib: file: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a(sed_fluxes.o) has no symbols
/opt/local/bin/ranlib: file: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a(sed_settling.o) has no symbols
/opt/local/bin/ranlib: file: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a(sed_surface.o) has no symbols
/opt/local/bin/ranlib: file: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a(sediment.o) has no symbols
warning: /opt/local/bin/ranlib: warning for library: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a the table of contents is empty (no object file members in the library define global symbols)
ranlib: file: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a(sed_bed.o) has no symbols
ranlib: file: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a(sed_bedload.o) has no symbols
ranlib: file: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a(sed_fluxes.o) has no symbols
ranlib: file: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a(sed_settling.o) has no symbols
ranlib: file: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a(sed_surface.o) has no symbols
ranlib: file: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a(sediment.o) has no symbols
warning: ranlib: warning for library: /Users/aryansafaie/ROMS/Projects/Upwelling/Build/libNLM_sed.a the table of contents is empty (no object file members in the library define global symbols)

It also outputs similar messages for libUTIL.a, libMODS.a, libNLM_bio.a, and libNLM.a; I don't believe libNLM_bio.a and libNLM_sed.a are relevant for the Upwelling example, but libUTIL.a and libMODS.a probably are?
To handle this, I've passed the flag "-s" to ARFLAGS and RANLIB in /trunk/Compilers/Darwin-gfortran.mk :

Code: Select all

  FC := gfortran
           FFLAGS := -frepack-arrays
              CPP := /usr/bin/cpp
         CPPFLAGS := -P -traditional
          LDFLAGS :=
               AR := ar
          ARFLAGS := -r -s
            MKDIR := mkdir -p
               RM := rm -f
           RANLIB := ranlib -s
             PERL := perl
             TEST := test

But I still receive all the "has no symbols" messages during ./build.bash;
Am I supposed to add or change something in the Module.mk files in the trunk/ROMS/Adjoint, trunk/ROMS/Nonlinear, trunk/ROMS/Utility, and trunk/ROMS/Modules directories?

2) When I run the executable oceanS by itself, and then enter the escape key (which shows up as ^[ in the Mac terminal) I still get the segmentation fault:

Code: Select all

Aryan-Safaies-MacBook-Pro:Upwelling aryansafaie$ ./oceanS

 Model Input Parameters:  ROMS/TOMS version 3.7  
                          Monday - March 9, 2015 - 11:24:35 AM
 -----------------------------------------------------------------------------


^[
Segmentation fault: 11
Aryan-Safaies-MacBook-Pro:Upwelling aryansafaie$ 

I'm not sure if this is relevant or not, but I noticed somebody in another thread had the same issue.

Thank you so much for your diligence and assistance.
-aryan

[kate]

You can check the intermediate files, for instance sed_bedload.f90. Mine contains:

Code: Select all

      MODULE sed_bedload_mod
      END MODULE sed_bedload_mod

You can look for symbols in the corresponding .o or .a files with nm:

Code: Select all

% nm sed_bedload.o
% nm libNLM_sed.a

sed_bed.o:

sed_bedload.o:

sed_fluxes.o:

sediment.o:

sed_settling.o:

sed_surface.o:

I didn't get any warnings on the Linux cluster, but did on the Mac. Not to worry, they are for things like set_masks.o, where MASKING is off in UPWELLING.

On the other thing, I don't know. Why are you hitting escape? Are you running "oceanS < ocean_upwelling.in"?

[aryansafaie]

Hi Kate,

I was being foolish in hitting escape...I saw it on another forum post, and thought it might lead me to a clue as to what's going on in my situation. So never mind that.

Now in ocean_upwelling.h I've tried setting NHIS, NAVG, and NDIA each form 72 (their defaults) to something large like 2000, and when running oceanG < ocean_upwelling.in, I still get the same segmentation fault before any time stepping has occurred, but there is no creation of history, average, or diagnostic files yet. So I'm wondering if my problem is something other than an NetCDF issue.

Thanks,
-aryan

[kate]

It could be that you need to increase your limits. The way to see and change it depends on your interactive shell, so your mileage may vary:

Code: Select all

bash-3.2$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 2560
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited
bash-3.2$ ulimit -s unlimited
bash: ulimit: stack size: cannot modify limit: Operation not permitted

This stack size is quite small and would be the trouble for a larger domain, but it is big enough for me to run the default UPWELLING case on my Mac. What is the last thing ROMS prints for you before it dies?

[aryansafaie]

Hi Kate

I've been running it with the following limits:

Code: Select all

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 65532
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited
Aryan-Safaies-MacBook-Pro:Upwelling aryansafaie$ 

I've even tried setting my core to unlimited, with no luck.

The last few output lines are:

Code: Select all

 Basin information for Grid 01:

 Maximum grid stiffness ratios:  rx0 =   6.931666E-02 (Beckmann and Haidvogel)
                                 rx1 =   8.661243E-01 (Haney)

 Initial basin volumes: TotVolume =  3.8843755884E+11 m3
                        MinVolume =  8.4521383562E+05 m3
                        MaxVolume =  2.5612332106E+07 m3
                          Max/Min =  3.0302783777E+01

 NL ROMS/TOMS: started time-stepping: (Grid: 01 TimeSteps: 00000001 - 00001440)


   STEP   Day HH:MM:SS  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME
          C => (i,j,k)       Cu            Cv            Cw         Max Speed

      0     0 00:00:00  0.000000E+00  6.585677E+02  6.585677E+02  3.884376E+11
            (00,00,00)  0.000000E+00  0.000000E+00  0.000000E+00  0.000000E+00
      DEF_HIS   - creating history file, Grid 01: ocean_his.nc
Segmentation fault: 11
Aryan-Safaies-MacBook-Pro:Upwelling aryansafaie$ 

And depending on how large NHIS, NAVG, and NDIA are, it will proceed to define the appropriate .nc file. For example, if NHIS is 2000 and NAVG is 72, that last line before the Segmentation fault: 11 statement is:

Code: Select all

       DEF_AVG   - creating average file, Grid 01: ocean_avg.nc

Thanks again.

[kate]

You've already compiled it with USE_DEBUG. The next step is to fire up gdb and run it in there. Or perhaps it left you a core file you can investigate with gdb:

Code: Select all

gdb oceanG core

then do "where".
ETA to add: https://www.cs.rochester.edu/~nelson/co ... w/gdb.html

[aryansafaie]

Running a backtrace or where gives me question marks:

Code: Select all

0x0000000100e2799c in ?? ()
(gdb) bt
#0  0x0000000100e2799c in ?? ()
#1  0x000000005fbf8920 in ?? ()
#2  0x00007fff00000000 in ?? ()
#3  0x0000000100000001 in ?? ()
#4  0x0000000000000000 in ?? ()
(gdb) where
#0  0x0000000100e2799c in ?? ()
#1  0x000000005fbf8920 in ?? ()
#2  0x00007fff00000000 in ?? ()
#3  0x0000000100000001 in ?? ()
#4  0x0000000000000000 in ?? ()
(gdb) 

And core gives me: "No core file now."

I tried a few things following the link you sent me (thanks):

-typing list, I get:

Code: Select all

(gdb) list
1	      PROGRAM ocean
2	!
3	!svn $Id: ocean.h 751 2015-01-07 22:56:36Z arango $
4	!================================================== Hernan G. Arango ===
5	!  Copyright (c) 2002-2015 The ROMS/TOMS Group                         !
6	!    Licensed under a MIT/X style license                              !
7	!    See License_ROMS.txt                                              !
8	!=======================================================================
9	!                                                                      !
10	!  Regional Ocean Model System (ROMS)                                  !
(gdb) 

-just typing break and then run gives me:

Code: Select all

(gdb) break
Breakpoint 1 at 0x100e2799c
(gdb) run
The program being debugged has been started already.
Start it from the beginning? (y or n) y

Starting program: /Users/aryansafaie/ROMS/Projects/Upwelling/oceanG < ocean_upwelling.in
Warning:
Cannot insert breakpoint 1.
Cannot access memory at address 0x100e2799c

Thanks again

[aryansafaie]

What I'm guessing is that the problem is either:
-I have a hardware issue
-I need to use NetCDF 4 instead of 3
-Somewhere along the way, using MacPorts went wrong.

But that still doesn't explain why gdb won't give me line numbers.

[kate]

Which compiler are you using? Can you try again with gfortran?

[aryansafaie]

I'm using gfortran.

Should I try using any of the others in build.bash?

[kate]

I don't know anything about the computer you are running on. What does it offer for compilers and debuggers?

[aryansafaie]

The compilers listed in build.bash are ifort, pgi, and gfortran.
I don't know if this helps, but I also have gcc and g++ installed:

Code: Select all

Aryan-Safaies-MacBook-Pro:Upwelling aryansafaie$ which gcc
/opt/local/bin/gcc
Aryan-Safaies-MacBook-Pro:Upwelling aryansafaie$ which gcc43
/opt/local/bin/gcc43
Aryan-Safaies-MacBook-Pro:Upwelling aryansafaie$ which g++
/usr/local/bin/g++
Aryan-Safaies-MacBook-Pro:Upwelling aryansafaie$ which g++43
/opt/local/bin/g++43

I don't know how to check what debuggers are available to me.

Thanks

[kate]

Let's stick with gfortran. Just because a compiler is mentioned in build.bash, that doesn't mean it's on your system.

You didn't delete the Build directory, did you? Or the .f90 files in it?

[aryansafaie]

No, the directory and .f90 files are still there. The build folder gets replaced everytime I run a new build though.

[aryansafaie]

An update:
This time I tried running the oceanG executable with lldb instead of gdb.
The last few lines of the output are:

Code: Select all

NL ROMS/TOMS: started time-stepping: (Grid: 01 TimeSteps: 00000001 - 00001440)


   STEP   Day HH:MM:SS  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME
          C => (i,j,k)       Cu            Cv            Cw         Max Speed

      0     0 00:00:00  0.000000E+00  6.585677E+02  6.585677E+02  3.884376E+11
            (00,00,00)  0.000000E+00  0.000000E+00  0.000000E+00  0.000000E+00
      DEF_AVG   - creating average file, Grid 01: ocean_avg.nc
Process 525 stopped
* thread #1: tid = 0x4536, 0x0000000100e2799c libnetcdff.5.dylib`__netcdf_MOD_nf90_create + 236, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xc)
    frame #0: 0x0000000100e2799c libnetcdff.5.dylib`__netcdf_MOD_nf90_create + 236
libnetcdff.5.dylib`__netcdf_MOD_nf90_create:
->  0x100e2799c <+236>: movl   (%r14), %eax
    0x100e2799f <+239>: jmp    0x100e279a5               ; <+245>
    0x100e279a1 <+241>: movl   0x48(%rsp), %eax
    0x100e279a5 <+245>: testl  %r9d, %r9d

Also, running a backtrace gives:

Code: Select all

(lldb) bt
error: unable to find CIE at 0x00000018 for cie_id = 0x00000004 for entry at 0x00000018.
error: unable to find CIE at 0x00000048 for cie_id = 0x00000004 for entry at 0x00000048.
error: unable to find CIE at 0x00000078 for cie_id = 0x00000004 for entry at 0x00000078.
error: unable to find CIE at 0x000000a8 for cie_id = 0x00000004 for entry at 0x000000a8.
error: unable to find CIE at 0x000000d8 for cie_id = 0x00000004 for entry at 0x000000d8.
error: unable to find CIE at 0x00000108 for cie_id = 0x00000004 for entry at 0x00000108.
error: unable to find CIE at 0x00000138 for cie_id = 0x00000004 for entry at 0x00000138.
error: unable to find CIE at 0x00000168 for cie_id = 0x00000004 for entry at 0x00000168.
error: unable to find CIE at 0x00000198 for cie_id = 0x00000004 for entry at 0x00000198.
error: unable to find CIE at 0x000001c8 for cie_id = 0x00000004 for entry at 0x000001c8.
error: unable to find CIE at 0x000001f8 for cie_id = 0x00000004 for entry at 0x000001f8.
error: unable to find CIE at 0x00000228 for cie_id = 0x00000004 for entry at 0x00000228.
error: unable to find CIE at 0x00000258 for cie_id = 0x00000004 for entry at 0x00000258.
error: unable to find CIE at 0x00000288 for cie_id = 0x00000004 for entry at 0x00000288.
error: unable to find CIE at 0x000002b8 for cie_id = 0x00000004 for entry at 0x000002b8.
error: unable to find CIE at 0x000002e8 for cie_id = 0x00000004 for entry at 0x000002e8.
error: unable to find CIE at 0x00000318 for cie_id = 0x00000004 for entry at 0x00000318.
error: unable to find CIE at 0x00000348 for cie_id = 0x00000004 for entry at 0x00000348.
error: unable to find CIE at 0x00000378 for cie_id = 0x00000004 for entry at 0x00000378.
error: unable to find CIE at 0x000003a8 for cie_id = 0x00000004 for entry at 0x000003a8.
error: unable to find CIE at 0x000003d8 for cie_id = 0x00000004 for entry at 0x000003d8.
error: unable to find CIE at 0x00000408 for cie_id = 0x00000004 for entry at 0x00000408.
error: unable to find CIE at 0x00000438 for cie_id = 0x00000004 for entry at 0x00000438.
error: unable to find CIE at 0x00000468 for cie_id = 0x00000004 for entry at 0x00000468.
error: unable to find CIE at 0x00000498 for cie_id = 0x00000004 for entry at 0x00000498.
error: unable to find CIE at 0x000004c8 for cie_id = 0x00000004 for entry at 0x000004c8.
error: unable to find CIE at 0x000004f8 for cie_id = 0x00000004 for entry at 0x000004f8.
error: unable to find CIE at 0x00000528 for cie_id = 0x00000004 for entry at 0x00000528.
error: unable to find CIE at 0x00000558 for cie_id = 0x00000004 for entry at 0x00000558.
error: unable to find CIE at 0x00000588 for cie_id = 0x00000004 for entry at 0x00000588.
error: unable to find CIE at 0x000005b8 for cie_id = 0x00000004 for entry at 0x000005b8.
error: unable to find CIE at 0x000005e8 for cie_id = 0x00000004 for entry at 0x000005e8.
error: unable to find CIE at 0x00000618 for cie_id = 0x00000004 for entry at 0x00000618.
error: unable to find CIE at 0x00000648 for cie_id = 0x00000004 for entry at 0x00000648.
error: unable to find CIE at 0x00000678 for cie_id = 0x00000004 for entry at 0x00000678.
error: unable to find CIE at 0x000006a8 for cie_id = 0x00000004 for entry at 0x000006a8.
error: unable to find CIE at 0x000006d8 for cie_id = 0x00000004 for entry at 0x000006d8.
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: libMODS.a(mod_netcdf.o) {0x00000876}: unhandled type tag 0x0012 (DW_TAG_string_type), please file a bug and attach the file at the start of this error message
error: unable to find CIE at 0x00000018 for cie_id = 0x00000004 for entry at 0x00000018.
error: unable to find CIE at 0x00000018 for cie_id = 0x00000004 for entry at 0x00000018.
error: unable to find CIE at 0x00000018 for cie_id = 0x00000004 for entry at 0x00000018.
error: unable to find CIE at 0x00000018 for cie_id = 0x00000004 for entry at 0x00000018.
error: unable to find CIE at 0x00000048 for cie_id = 0x00000004 for entry at 0x00000048.
error: unable to find CIE at 0x00000078 for cie_id = 0x00000004 for entry at 0x00000078.
error: unable to find CIE at 0x00000018 for cie_id = 0x00000004 for entry at 0x00000018.
error: unable to find CIE at 0x00000018 for cie_id = 0x00000004 for entry at 0x00000018.
* thread #1: tid = 0x4536, 0x0000000100e2799c libnetcdff.5.dylib`__netcdf_MOD_nf90_create + 236, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xc)
  * frame #0: 0x0000000100e2799c libnetcdff.5.dylib`__netcdf_MOD_nf90_create + 236
    frame #1: 0x0000000100af6498 oceanG`netcdf_create(ng=0x00007fff5fbff8a4, model=0x00007fff5fbfeae8, ncname=0x0000000104a15cc0, ncid=0x0000000104a0c00c, _ncname=12) + 135 at mod_netcdf.f90:4283
    frame #2: 0x000000010001b5f0 oceanG`def_avg(ng=0x00007fff5fbff8a4, ldef=0x0000000104a12600) + 1136 at def_avg.f90:69
    frame #3: 0x000000010054571e oceanG`output(ng=0x00007fff5fbff8a4) + 26863 at output.f90:189
    frame #4: 0x00000001004fcdbf oceanG`main3d(runinterval=0x0000000100da8f00) + 13793 at main3d.f90:214
    frame #5: 0x0000000100001ab5 oceanG`roms_run(runinterval=0x0000000100da8f00) + 896 at ocean_control.f90:151
    frame #6: 0x0000000100000df9 oceanG`ocean + 655 at master.f90:86
    frame #7: 0x0000000100b7a296 oceanG`main(argc=1, argv=0x00007fff5fbffb20) + 37 at fmain.c:21
    frame #8: 0x00007fff911935c9 libdyld.dylib`start + 1

I think I'll try rebuilding NetCDF and downloading an older version of GNU to fix this...I'll report back.