A problem about building OpenMP ROMS

[zhouwei]

Hi,
I turned on shared-memory parallelization in ROMS using the OpenMP compiler directives in SGI workstation with ifort compiler in the upwelling test. It runs fine on SERIAL.

I typed the following command :
make -j 8 > compile.txt
then I got the following warnings:

makefile:230: INCLUDING FILE Compilers/make_macros.mk WHICH CONTAINS APPLICATION-DEPENDENT MAKE DEFINITIONS
make: svnversion: Command not found
makefile:226: INCLUDING FILE Build/make_macros.mk WHICH CONTAINS APPLICATION-DEPENDENT MAKE DEFINITIONS
make: svnversion: Command not found
make: Warning: File `Build' has modification time 1 s in the future
ROMS/Modules/mod_nesting.F:579: warning: extra tokens at end of #endif directive
ar: creating Build/libICE.a
ar: creating Build/libMODS.a
ar: creating Build/libANA.a
ar: creating Build/libUTIL.a
ar: creating Build/libNLM.a
make: warning: Clock skew detected. Your build may be incomplete.

Does anyone have ideas on steps to be taken to successfully build OpenMP ROMS on this sort of platform?
Any information is greatly appreciated.

[jcwarner]

well, i think the problem is not the open mp stuff. I think it is
make - j 8
the "8" is probably the culprit. Try
make -j 2
or soemthing and see if that works.
-j

[zhouwei]

Hi,jcwarner
Thanks for your reply.
When I typed this command:

make -j > compile.txt

I got the following message:

zw@a330:~/roms> make -j >compile.txt
makefile:230: INCLUDING FILE Compilers/make_macros.mk WHICH CONTAINS APPLICATION-DEPENDENT MAKE DEFINITIONS
make: svnversion: Command not found
makefile:226: INCLUDING FILE Build/make_macros.mk WHICH CONTAINS APPLICATION-DEPENDENT MAKE DEFINITIONS
make: svnversion: Command not found
ROMS/Modules/mod_nesting.F:579: warning: extra tokens at end of #endif directive
ar: creating Build/libICE.a
ar: creating Build/libMODS.a
ar: creating Build/libANA.a
ar: creating Build/libUTIL.a
ar: creating Build/libNLM.a


Then I attempted to run 'oceanO' by typing the command :

mpirun -np 2 oceanO < ROMS/External/ocean_upwelling.in


I got the following errors:




Process Information:

Thread # 1 (pid= 29363) is active.
Thread # 0 (pid= 29363) is active.

Model Input Parameters: ROMS/TOMS version 3.0
Wednesday - August 8, 2007 - 8:06:38 AM
-----------------------------------------------------------------------------

Wind-Driven Upwelling/Downwelling over a Periodic Channel

Operating system : Linux
CPU/hardware : ia64
Compiler system : ifort
Compiler command : ifort
Compiler flags : -openmp -fpp -ip -O3 -free

SVN Root URL : https://www.myroms.org/svn/src/trunk
SVN Revision :

Local Root : /lun/home/zw/roms
Header Dir : ./ROMS/Include
Header file : upwelling.h
Analytical Dir: /lun/home/zw/roms/ROMS/Functionals

Resolution, Grid 01: 0041x0080x016, Parallel Threads: 2, Tiling: 001x001

ROMS/TOMS: Wrong choice of domain 001 partition or number of parallel threads.
NtileI*NtileJ must be a positive multiple of the number of threads.
Change number of threads (environment variable) or
change domain partition in input script.

Tile partition information for Grid 01: 0041x0080x0016 tiling: 001x001

tile Istr Iend Jstr Jend Npts

0 1 41 1 80 52480

Activated C-preprocessing Options:

UPWELLING Wind-Driven Upwelling/Downwelling over a Periodic Channel
ANA_BSFLUX Analytical kinematic bottom salinity flux.
ANA_BTFLUX Analytical kinematic bottom temperature flux.
ANA_GRID Analytical grid set-up.
ANA_INITIAL Analytical initial conditions.
ANA_SMFLUX Analytical kinematic surface momentum flux.
ANA_SSFLUX Analytical kinematic surface salinity flux.
ANA_STFLUX Analytical kinematic surface temperature flux.
ANA_VMIX Analytical vertical mixing coefficients.
ASSUMED_SHAPE Using assumed-shape arrays.
AVERAGES Writing out time-averaged fields.
DIAGNOSTICS_TS Computing and writing tracer diagnostic terms.
DIAGNOSTICS_UV Computing and writing momentum diagnostic terms.
DJ_GRADPS Parabolic Splines density Jacobian (Shchepetkin, 2002).
DOUBLE_PRECISION Double precision arithmetic.
EW_PERIODIC East-West periodic boundaries.
MIX_S_TS Mixing of tracers along constant S-surfaces.
MIX_S_UV Mixing of momentum along constant S-surfaces.
NONLINEAR Nonlinear Model.
!NONLIN_EOS Linear Equation of State for seawater.
_OPENMP OpenMP parallel shared-memory directives.
POWER_LAW Power-law shape time-averaging barotropic filter.
PROFILE Time profiling activated .
!RST_SINGLE Double precision fields in restart NetCDF file.
SALINITY Using salinity.
SOLVE3D Solving 3D Primitive Equations.
SPLINES Conservative parabolic spline reconstruction.
TS_U3HADVECTION Third-order upstream bias horizontal advection of tracers.
TS_C4VADVECTION Fourth-order centered vertical advection of tracers.
TS_DIF2 Harmonic mixing of tracers.
UV_ADV Advection of momentum.
UV_COR Coriolis term.
UV_U3HADVECTION Third-order upstream bias advection of momentum.
UV_LDRAG Linear bottom stress.
UV_VIS2 Harmonic mixing of momentum.
VAR_RHO_2D Variable density barotropic mode.

ROMS/TOMS - Partition error ......... exit_flag: 6


Elapsed CPU time (seconds):

Thread # 1 CPU: 0.035
Thread # 0 CPU: 0.030
Total: 0.065

Nonlinear model elapsed time profile:

Total: 0.000 0.0000

All percentages are with respect to total time = 0.065

ROMS/TOMS - Output NetCDF summary for Grid 01:

ERROR: PARAM - Illegal domain partition.

[m.hadfield]

I can see a couple of problems.

First you seem to be trying to run an OpenMP executable with the MPI command mpirun. This seems an odd thing to do, though I can't say for sure it wouldn't work. You should be able to run the executable with

Code: Select all

./oceanO < ROMS/External/ocean_upwelling.in

Second, in the input file, ROMS/External/ocean_upwelling.in, the number of tiles is set to

Code: Select all

      NtileI == 1  ! I-direction partition
      NtileJ == 1  ! J-direction partition

At least, this is the default and the model output confirms you haven't changed it.

Code: Select all

 Resolution, Grid 01: 0041x0080x016, Parallel Threads: 2, Tiling: 001x001

As the model is telling you, this is invalid. Try setting it to something like

Code: Select all

      NtileI == 1  ! I-direction partition
      NtileJ == 4  ! J-direction partition

Re the preprocessor warning:

Code: Select all

ROMS/Modules/mod_nesting.F:579: warning: extra tokens at end of #endif directive

This implies an error in this file, but errors of this sort are often benign.

The error reported by make:

Code: Select all

make: svnversion: Command not found

is definitely benign.

[zhouwei]

Thank you very much for your constructions, m.hadfield.
When I compiled the OMP ROMS and typed this command:

make -j

I got the following warning:

make: warning: Clock skew detected. Your build may be incomplete.

I don't know what caused this problem in general.

I know few about OpenMP .Could you please give me some message about how to use
the OpenMP. Thank you again for your help.

[kate]

The clock skew problem is due to mounting a file system from one computer to another, where the clocks don't match. The file lives on computer A and gets created at some time, which computer B thinks is in the future.

[arango]

Yes, the file mod_nesting.F has NESTING after the #endif. This is a syntax error but it shouldn't affect compilation.

You have two problems here. You can never compile with OpenMP and run with the mpirun script. There is a lot of special things done during c-preprocessing that affect the code that you compile with OpenMP. The array allocation and all the shared-memory loops are very different

The second problem that you have is clock synchronization problem during make. This can be dangerous because you don't know what version of a particular routine you are compiling. You need to synchronize your computer and disk clock. This happen quite frequently in computers. There are several tools out there to synchronize clocks.

[zhouwei]

Hi ,arango ,Kate
Thank you very much for your replies,which really help me a lot

[gerardo]

Hernan,

Perhaps a check could be inserted somewhere in the make scripts
that terminates a build if USE_MPI and USE_OpenMP are both set?

Saludos,
--
Gerardo Cisneros