Segmentation Fault with forcing files

Message

LRasmuson · Fri Mar 15, 2013 11:07 pm

Hi All,

My goal is to develop an idealized upwelling model. I have created a grid using seagrid and been able to get the model up and running with forcing being provided by the ANA files. My next goal was to develop an forcing file that I can use to provide more realistic forcing values. After generating the file in matlab I changed the *.in file to recognize the location of the forcing file and undefined the ANA variables in the *.h file (leaving the ANA_initial defined since I did not create a initial conditions file). The file builds with no error messages but after attempting to initiate an MPI run with 8 processors I get the following error message

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
oceanM 000000000068D8E2 Unknown Unknown Unknown
oceanM 0000000000620912 Unknown Unknown Unknown
oceanM 000000000061541D Unknown Unknown Unknown
oceanM 00000000005AF6A1 Unknown Unknown Unknown
oceanM 00000000004A4224 Unknown Unknown Unknown
oceanM 0000000000424767 Unknown Unknown Unknown
oceanM 0000000000424223 Unknown Unknown Unknown
oceanM 00000000004240EC Unknown Unknown Unknown
libc.so.6 00007FC601F1E30D Unknown Unknown Unknown
oceanM 0000000000423FE9 Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
oceanM 000000000068D8E2 Unknown Unknown Unknown
oceanM 0000000000620912 Unknown Unknown Unknown
oceanM 000000000061541D Unknown Unknown Unknown
oceanM 00000000005AF6A1 Unknown Unknown Unknown
oceanM 00000000004A4224 Unknown Unknown Unknown
oceanM 0000000000424767 Unknown Unknown Unknown
oceanM 0000000000424223 Unknown Unknown Unknown
oceanM 00000000004240EC Unknown Unknown Unknown
libc.so.6 00007F7D397F430D Unknown Unknown Unknown
oceanM 0000000000423FE9 Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
oceanM 000000000068D8E2 Unknown Unknown Unknown
oceanM 0000000000620912 Unknown Unknown Unknown
oceanM 000000000061541D Unknown Unknown Unknown
oceanM 00000000005AF6A1 Unknown Unknown Unknown
oceanM 00000000004A4224 Unknown Unknown Unknown
oceanM 0000000000424767 Unknown Unknown Unknown
oceanM 0000000000424223 Unknown Unknown Unknown
oceanM 00000000004240EC Unknown Unknown Unknown
libc.so.6 00007F6EB2CE530D Unknown Unknown Unknown
oceanM 0000000000423FE9 Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
oceanM 000000000068D8E2 Unknown Unknown Unknown
oceanM 0000000000620912 Unknown Unknown Unknown
oceanM 000000000061541D Unknown Unknown Unknown
oceanM 00000000005AF6A1 Unknown Unknown Unknown
oceanM 00000000004A4224 Unknown Unknown Unknown
oceanM 0000000000424767 Unknown Unknown Unknown
oceanM 0000000000424223 Unknown Unknown Unknown
oceanM 00000000004240EC Unknown Unknown Unknown
libc.so.6 00007F10B845730D Unknown Unknown Unknown
oceanM 0000000000423FE9 Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
oceanM 000000000068D8E2 Unknown Unknown Unknown
oceanM 0000000000620912 Unknown Unknown Unknown
oceanM 000000000061541D Unknown Unknown Unknown
oceanM 00000000005AF6A1 Unknown Unknown Unknown
oceanM 00000000004A4224 Unknown Unknown Unknown
oceanM 0000000000424767 Unknown Unknown Unknown
oceanM 0000000000424223 Unknown Unknown Unknown
oceanM 00000000004240EC Unknown Unknown Unknown
libc.so.6 00007FB14CAB630D Unknown Unknown Unknown
oceanM 0000000000423FE9 Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
oceanM 000000000068D8E2 Unknown Unknown Unknown
oceanM 0000000000620912 Unknown Unknown Unknown
oceanM 000000000061541D Unknown Unknown Unknown
oceanM 00000000005AF6A1 Unknown Unknown Unknown
oceanM 00000000004A4224 Unknown Unknown Unknown
oceanM 0000000000424767 Unknown Unknown Unknown
oceanM 0000000000424223 Unknown Unknown Unknown
oceanM 00000000004240EC Unknown Unknown Unknown
libc.so.6 00007FF4436B330D Unknown Unknown Unknown
oceanM 0000000000423FE9 Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
oceanM 000000000068D8E2 Unknown Unknown Unknown
oceanM 0000000000620912 Unknown Unknown Unknown
oceanM 000000000061541D Unknown Unknown Unknown
oceanM 00000000005AF6A1 Unknown Unknown Unknown
oceanM 00000000004A4224 Unknown Unknown Unknown
oceanM 0000000000424767 Unknown Unknown Unknown
oceanM 0000000000424223 Unknown Unknown Unknown
oceanM 00000000004240EC Unknown Unknown Unknown
libc.so.6 00007FD4C1D5630D Unknown Unknown Unknown
oceanM 0000000000423FE9 Unknown Unknown Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
oceanM 000000000068D8E2 Unknown Unknown Unknown
oceanM 0000000000620912 Unknown Unknown Unknown
oceanM 000000000061541D Unknown Unknown Unknown
oceanM 00000000005AF6A1 Unknown Unknown Unknown
oceanM 00000000004A4224 Unknown Unknown Unknown
oceanM 0000000000424767 Unknown Unknown Unknown
oceanM 0000000000424223 Unknown Unknown Unknown
oceanM 00000000004240EC Unknown Unknown Unknown
libc.so.6 00007F1A8FC8230D Unknown Unknown Unknown
oceanM 0000000000423FE9 Unknown Unknown Unknown
--------------------------------------------------------------------------
mpirun has exited due to process rank 4 with PID 43982 on
node david-H8QG6 exiting improperly. There are two reasons this could occur:

1. this process did not call "init" before exiting, but others in
the job did. This can cause a job to hang indefinitely while it waits
for all processes to call "init". By rule, if one process calls "init",nanoi
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--------------------------------------------------------------------------

My initial thought was that I had made a mistake in creating the forcing files so I went back to the test cases and noticed that the test_head case uses forcing files that are included with ROMs and tried to run the test case. I received the attached error message when running the make command.

My next thought was that since the test_head case was a sediment application it could have been some secondary issue relating to this but I attempted to run the sed_toy application and it compiled and ran fine which made me think that the error may be in the reading of the netcdf file. Unfortunately I am running this over a server and any attempts to change the build and make file to allow me to run the model in debug return the following line

-bash: syntax error near unexpected token `newline'

I spoke to the folks who got the server up and running and they have never run any models in debug so they weren't sure what the cause of this was. Any advice on the cause and a repair of the segmentation faults would be greatly appreciated.
Thanks
Leif