Using OPENMP on Sun AMD (intel) for solaris 10

Message

jklinck · #1 Unread post by **jklinck** » Thu May 08, 2008 5:17 pm

I am attempting to run the Upwelling test case on a Sun computer with 2 dual core opteron (AMD) chips with openmp. The calculation blows up in the first long step. Has anyone run ROMS with openmp on SUN opteron using the SunStudio 12 compilers? The OS is solaris 10.

Thanks in advance for your help.

John

Details:

I can run this case in serial mode with no problem. I copied SunOS-pgi.mk to SunOS-f95.mk and changed FC to f95. I also had to remove the -U option from FFLAGS. Otherwise, the compiler does not find the right netcdf variables in the modules (this is a persistent problem from almost all earlier versions of ROMS with Sun compilers).

With these changes, serial upwelling runs to completion (about 400 s).

I turned on the openmp option and recompiled everything. I also changed the compiler flags to -xopenmp=parallel in both FFLAGS and LDFLAGS (as recommended by the f95 man page). I set OMP_NUM_THREADS to 2. I change the tiling in the input file to 2x2.

The model starts, writes the initial state to the output files, then hits a blowup condition. The head and tail of the run log are below.

START of RUN LOG:

Process Information:

Thread # 1 (pid= 18151) is active.
Thread # 0 (pid= 18151) is active.

Model Input Parameters: ROMS/TOMS version 3.0
Thursday - May 8, 2008 - 12:54:00 PM
-----------------------------------------------------------------------------

Wind-Driven Upwelling/Downwelling over a Periodic Channel

Operating system : SunOS
CPU/hardware : i86pc
Compiler system : f95
Compiler command : f95
Compiler flags : -u -xopenmp=parallel -fast -free

SVN Root URL : https://www.myroms.org/svn/src/trunk
SVN Revision :

Local Root : /net/longterm/klinck/ROMS
Header Dir : ./ROMS/Include
Header file : upwelling.h
Analytical Dir: /net/longterm/klinck/ROMS/ROMS/Functionals

Resolution, Grid 01: 0041x0080x016, Parallel Threads: 2, Tiling: 002x002

[deleted stuff]

ROMS/TOMS - Output NetCDF summary for Grid 01:
number of time records written in HISTORY file = 00000001
number of time records written in RESTART file = 00000001

Analytical header files used:

ROMS/Functionals/ana_btflux.h
ROMS/Functionals/ana_grid.h
ROMS/Functionals/ana_initial.h
ROMS/Functionals/ana_smflux.h
ROMS/Functionals/ana_stflux.h
ROMS/Functionals/ana_vmix.h

MAIN: Abnormal termination: BLOWUP.

ROMS/TOMS: DONE... Thursday - May 8, 2008 - 12:54:03 PM

robertson · #2 Unread post by **robertson** » Thu May 08, 2008 5:28 pm

I am not familiar with OpenMP but I would start by setting OMP_NUM_THREADS to 4 to match your 2x2 tiling. With 2x2 tiling you are telling ROMS to make 4 tiles but OpenMP will only give ROMS 2 threads if you have OMP_NUM_THREADS set to 2.

jklinck · #3 Unread post by **jklinck** » Thu May 08, 2008 5:48 pm

I changed the tiling to by 1x2 and ran with 2 threads. (I have another longrunning job and don't want to interfere).

I get the same result-- abnormal termination during the first step.

John

kate · #4 Unread post by **kate** » Mon May 12, 2008 5:50 pm

Have you tried backing off on the compiler optimization? I've had to with the Sun compiler. If you have WET_DRY, then the compiler issues are even worse.