Compiling ROMS in parallel on 4 processors (2*Intel Core2)

[thomas.roc]

Hi everyone,

I'm trying to compile ROMS on 4 processors with the command "make -j 4". My exploitation system is Fedora 9 on a 64 bits architecture and I'm using intel fortran compiler and intel MPI library. In the makefile I've only switched on the "USE_MPI" and "USE_LARGE" options and define the compiler I'm using.

According to the latter discussions, these errors might come from my MPI library. But I've cheked if the environment variables related to my Mpi library are defined properly and they are. Does it mean that the intel MPI library are not sufficient to compile ROMS in parallel (if it's the case does anybody know where I can find complementary libraries ?) or am I missing the point ?

Has anybody come across the same kind of troubles ?

Cheers

P.S: Following my error message...it mights help

Build/master.o: In function `MAIN__':
master.f90:(.text+0x3c): undefined reference to `mpi_init_'
master.f90:(.text+0x8f): undefined reference to `mpi_comm_rank_'
master.f90:(.text+0x125): undefined reference to `mpi_finalize_'
Build/ocean_control.o: In function `ocean_control_mod_mp_roms_initialize_':
ocean_control.f90:(.text+0x264): undefined reference to `mpi_comm_rank_'
ocean_control.f90:(.text+0x27a): undefined reference to `mpi_comm_size_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_gather3d_':
distribute.f90:(.text+0xa66): undefined reference to `mpi_irecv_'
distribute.f90:(.text+0xabe): undefined reference to `mpi_isend_'
distribute.f90:(.text+0xad9): undefined reference to `mpi_wait_'
distribute.f90:(.text+0xc38): undefined reference to `mpi_wait_'
distribute.f90:(.text+0xef1): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x1007): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_aggregate_':
distribute.f90:(.text+0x1b63): undefined reference to `mpi_irecv_'
distribute.f90:(.text+0x1bb9): undefined reference to `mpi_isend_'
distribute.f90:(.text+0x1bd4): undefined reference to `mpi_wait_'
distribute.f90:(.text+0x1f93): undefined reference to `mpi_wait_'
distribute.f90:(.text+0x2241): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x2357): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_scatter_state_':
distribute.f90:(.text+0x26f1): undefined reference to `mpi_irecv_'
distribute.f90:(.text+0x270d): undefined reference to `mpi_wait_'
distribute.f90:(.text+0x28f2): undefined reference to `mpi_isend_'
distribute.f90:(.text+0x290a): undefined reference to `mpi_wait_'
distribute.f90:(.text+0x293e): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x2a6e): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x2ba6): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x2cb6): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_scatter3d_':
distribute.f90:(.text+0x313b): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x3408): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_scatter2d_':
distribute.f90:(.text+0x3833): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x3a12): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_reduce_v_':
distribute.f90:(.text+0x3c38): undefined reference to `mpi_allgather_'
distribute.f90:(.text+0x3ed4): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_reduce_s_':
distribute.f90:(.text+0x405f): undefined reference to `mpi_allgather_'
distribute.f90:(.text+0x41ed): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_bcasti_v_':
distribute.f90:(.text+0x4314): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x435e): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_ncwrite_':
distribute.f90:(.text+0x4616): undefined reference to `mpi_gather_'
distribute.f90:(.text+0x4dfb): undefined reference to `mpi_irecv_'
distribute.f90:(.text+0x4e13): undefined reference to `mpi_wait_'
distribute.f90:(.text+0x5227): undefined reference to `mpi_isend_'
distribute.f90:(.text+0x5249): undefined reference to `mpi_wait_'
distribute.f90:(.text+0x52ab): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x54e6): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x566b): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x577a): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x5893): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_ncread_':
distribute.f90:(.text+0x5ba0): undefined reference to `mpi_gather_'
distribute.f90:(.text+0x5c58): undefined reference to `mpi_irecv_'
distribute.f90:(.text+0x5c70): undefined reference to `mpi_wait_'
distribute.f90:(.text+0x6568): undefined reference to `mpi_isend_'
distribute.f90:(.text+0x6580): undefined reference to `mpi_wait_'
distribute.f90:(.text+0x6603): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x6725): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x6869): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x6ae0): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x6bfd): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_gather_state_':
distribute.f90:(.text+0x6dc5): undefined reference to `mpi_allgather_'
distribute.f90:(.text+0x6e74): undefined reference to `mpi_irecv_'
distribute.f90:(.text+0x6e8c): undefined reference to `mpi_wait_'
distribute.f90:(.text+0x6f5d): undefined reference to `mpi_isend_'
distribute.f90:(.text+0x6f75): undefined reference to `mpi_wait_'
distribute.f90:(.text+0x6fa6): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x7001): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x70f0): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x7200): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x7310): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_gather2d_':
distribute.f90:(.text+0x7c9f): undefined reference to `mpi_irecv_'
distribute.f90:(.text+0x7cf9): undefined reference to `mpi_isend_'
distribute.f90:(.text+0x7d14): undefined reference to `mpi_wait_'
distribute.f90:(.text+0x7e32): undefined reference to `mpi_wait_'
distribute.f90:(.text+0x8054): undefined reference to `mpi_error_string_'
distribute.f90:(.text+0x816a): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_collect_':
distribute.f90:(.text+0x8326): undefined reference to `mpi_allreduce_'
distribute.f90:(.text+0x8389): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_boundary_':
distribute.f90:(.text+0x87d2): undefined reference to `mpi_allreduce_'
distribute.f90:(.text+0x88ba): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_bcasts_v_':
distribute.f90:(.text+0x8a04): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x8a5c): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_bcasts_s_':
distribute.f90:(.text+0x8ba8): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x8bfe): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_bcastl_v_':
distribute.f90:(.text+0x8d34): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x8d7e): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_bcastl_s_':
distribute.f90:(.text+0x8ea1): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x8ee9): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_bcasti_m_':
distribute.f90:(.text+0x9027): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x9073): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_bcasti_s_':
distribute.f90:(.text+0x91a1): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x91e9): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_bcastf_m_':
distribute.f90:(.text+0x9317): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x9363): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_bcastf_v_':
distribute.f90:(.text+0x9494): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x94de): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_bcastf_s_':
distribute.f90:(.text+0x9601): undefined reference to `mpi_bcast_'
distribute.f90:(.text+0x9649): undefined reference to `mpi_error_string_'
Build/libUTIL.a(distribute.o): In function `distribute_mod_mp_mp_barrier_':
distribute.f90:(.text+0x973d): undefined reference to `mpi_barrier_'
Build/libUTIL.a(mp_exchange.o): In function `mp_exchange_mod_mp_mp_exchange4d_':
mp_exchange.f90:(.text+0x1928): undefined reference to `mpi_irecv_'
mp_exchange.f90:(.text+0x1971): undefined reference to `mpi_irecv_'
mp_exchange.f90:(.text+0x19b2): undefined reference to `mpi_send_'
mp_exchange.f90:(.text+0x19f3): undefined reference to `mpi_send_'
mp_exchange.f90:(.text+0x2539): undefined reference to `mpi_irecv_'
mp_exchange.f90:(.text+0x2584): undefined reference to `mpi_irecv_'
mp_exchange.f90:(.text+0x25c2): undefined reference to `mpi_send_'
mp_exchange.f90:(.text+0x2605): undefined reference to `mpi_send_'
mp_exchange.f90:(.text+0x263a): undefined reference to `mpi_wait_'
mp_exchange.f90:(.text+0x5363): undefined reference to `mpi_error_string_'
mp_exchange.f90:(.text+0x5475): undefined reference to `mpi_wait_'
mp_exchange.f90:(.text+0x61f6): undefined reference to `mpi_error_string_'
mp_exchange.f90:(.text+0x6308): undefined reference to `mpi_wait_'
mp_exchange.f90:(.text+0x6d62): undefined reference to `mpi_error_string_'
mp_exchange.f90:(.text+0x6e74): undefined reference to `mpi_wait_'
mp_exchange.f90:(.text+0x7977): undefined reference to `mpi_error_string_'
Build/libUTIL.a(mp_exchange.o): In function `mp_exchange_mod_mp_mp_exchange3d_':
mp_exchange.f90:(.text+0xab2e): undefined reference to `mpi_irecv_'
mp_exchange.f90:(.text+0xab71): undefined reference to `mpi_irecv_'
mp_exchange.f90:(.text+0xabaf): undefined reference to `mpi_send_'
mp_exchange.f90:(.text+0xabed): undefined reference to `mpi_send_'
mp_exchange.f90:(.text+0xd983): undefined reference to `mpi_irecv_'
mp_exchange.f90:(.text+0xd9c8): undefined reference to `mpi_irecv_'
mp_exchange.f90:(.text+0xda08): undefined reference to `mpi_send_'
mp_exchange.f90:(.text+0xda48): undefined reference to `mpi_send_'
mp_exchange.f90:(.text+0xda82): undefined reference to `mpi_wait_'
mp_exchange.f90:(.text+0xf55f): undefined reference to `mpi_error_string_'
mp_exchange.f90:(.text+0xf671): undefined reference to `mpi_wait_'
mp_exchange.f90:(.text+0x10f1f): undefined reference to `mpi_error_string_'
mp_exchange.f90:(.text+0x11031): undefined reference to `mpi_wait_'
mp_exchange.f90:(.text+0x12279): undefined reference to `mpi_error_string_'
mp_exchange.f90:(.text+0x1238b): undefined reference to `mpi_wait_'
mp_exchange.f90:(.text+0x1360e): undefined reference to `mpi_error_string_'
Build/libUTIL.a(mp_exchange.o): In function `mp_exchange_mod_mp_mp_exchange2d_':
mp_exchange.f90:(.text+0x14ddb): undefined reference to `mpi_irecv_'
mp_exchange.f90:(.text+0x14e1a): undefined reference to `mpi_irecv_'
mp_exchange.f90:(.text+0x14e52): undefined reference to `mpi_send_'
mp_exchange.f90:(.text+0x14e8c): undefined reference to `mpi_send_'
mp_exchange.f90:(.text+0x15ff5): undefined reference to `mpi_irecv_'
mp_exchange.f90:(.text+0x1603a): undefined reference to `mpi_irecv_'
mp_exchange.f90:(.text+0x16077): undefined reference to `mpi_send_'
mp_exchange.f90:(.text+0x160b7): undefined reference to `mpi_send_'
mp_exchange.f90:(.text+0x160ee): undefined reference to `mpi_wait_'
mp_exchange.f90:(.text+0x1a60d): undefined reference to `mpi_error_string_'
mp_exchange.f90:(.text+0x1a71f): undefined reference to `mpi_wait_'
mp_exchange.f90:(.text+0x1be57): undefined reference to `mpi_error_string_'
mp_exchange.f90:(.text+0x1bf69): undefined reference to `mpi_wait_'
mp_exchange.f90:(.text+0x1cdd6): undefined reference to `mpi_error_string_'
mp_exchange.f90:(.text+0x1cee8): undefined reference to `mpi_wait_'
mp_exchange.f90:(.text+0x1e0d4): undefined reference to `mpi_error_string_'
Build/libMODS.a(mod_parallel.o): In function `mod_parallel_mp_initialize_parallel_':
mod_parallel.f90:(.text+0x30): undefined reference to `mpi_comm_size_'
make: *** [/home/troc/Desktop/idealized_case/oceanM] Error 1

[kate]

Do you have something like an mpif90 command? Most of us are using a script of that ilk which knows how to find mpi.h and all its declarations. Well, it's not complaining about not finding mpi.h, so it might be that the mpi library was compiled with different underscore options. Do you know where the mpi libraries are on your system? You can query them with the nm command to find out what underscores they have:

nm libmpixxx.a | grep mpi_init

[robertson]

I believe for Intel's MPI the executable is named mpiifort. What you'll want to do is turn on USE_MPIF90 Then edit your Compilers/Linux-ifort.mk file of you ROMS code. Around line 70 you will see:

Code: Select all

FC := mpif90

change this to:

Code: Select all

FC := mpiifort

Then of course you'll want to make sure your PATH is set properly. You can do this via the build script (there's a section for modifying the path to get the proper mpif90) or in your environment before you build ROMS.

As Kate mentioned you can use nm to check the contents of a library. However, since you not don't have USE_MPIF90 set, my guess is that the libraries libfmpi-pgi and libmpi-pgi (specified around line 72 of the Linux-ifort.mk file) don't exist on your system and that fact is likely reported above the errors you put in your post.

Good luck!

[thomas.roc]

Many thanks for the piece of advice, now it's compiling and running correctly.
But a new error has popped up, from now I can use the "make clean" command neither recompiling the code.
It can be read:
"
makefile:229: INCLUDING FILE Build/make_macros.mk WHICH CONTAINS APPLICATION-DEPENDENT MAKE DEFINITIONS
ROMS/Nonlinear/Module.mk:15: *** commands commence before first target. Stop.
"
When I'm trying to re-compile the code and:
"
ROMS/Nonlinear/Module.mk:15: *** commands commence before first target. Stop.
"
When I'm using the make-clean command.

Anybody has any thoughts on this one ?

Cheers