Running ROMS-2.2 on multi-processor

Message

nageswararao · Tue Apr 29, 2008 2:08 pm

Actually I dont much about multi-processor and mpi options.
My code ran well with single processor. Now I want to run on multi-processor architecture.
My system is RHEL-3.0 with AMD 64-bit processors. 40 nodes individually are there and connected through MPICH2.
I ran the model with MPI on with 2 processors and I got the error like below.

/shared/nagesh/IntelFort/bin/ifort -mp -recursive -g -check bounds -Vaxlib master.o ocean_control.o -o /shared/nagesh/ROMS-2.2/roms_io_r_adv4 libNLM.a libUTIL.a libMODS.a -L/shared/netcdf3/ifort/lib -lnetcdf -lfmpi-pgi -lmpi-pgi
master.o(.text+0x21): In function `ocean':
/shared/nagesh/ROMS-2.2/master.f90:61: undefined reference to `mpi_init_'
master.o(.text+0x93):/shared/nagesh/ROMS-2.2/master.f90:71: undefined reference to `mpi_comm_rank_'
master.o(.text+0x132):/shared/nagesh/ROMS-2.2/master.f90:102: undefined reference to `mpi_finalize_'
libUTIL.a(distribute.o)(.text+0x1e): In function `mp_barrier':
/shared/nagesh/ROMS-2.2/distribute.f90:29: undefined reference to `mpi_barrier_'
libUTIL.a(distribute.o)(.text+0xc3): In function `mp_bcastf':
/shared/nagesh/ROMS-2.2/distribute.f90:80: undefined reference to `mpi_bcast_'
libUTIL.a(distribute.o)(.text+0x255): In function `mp_bcasti':
/shared/nagesh/ROMS-2.2/distribute.f90:144: undefined reference to `mpi_bcast_'
libUTIL.a(distribute.o)(.text+0x3e7): In function `mp_bcastl':
/shared/nagesh/ROMS-2.2/distribute.f90:208: undefined reference to `mpi_bcast_'
libUTIL.a(distribute.o)(.text+0x57e): In function `mp_bcasts':
/shared/nagesh/ROMS-2.2/distribute.f90:272: undefined reference to `mpi_bcast_'
libUTIL.a(distribute.o)(.text+0x11ab): In function `mp_boundary':
/shared/nagesh/ROMS-2.2/distribute.f90:374: undefined reference to `mpi_allgather_'
libUTIL.a(distribute.o)(.text+0x1f2c): In function `mp_collect':
/shared/nagesh/ROMS-2.2/distribute.f90:452: undefined reference to `mpi_allgather_'
libUTIL.a(distribute.o)(.text+0x66f5): In function `mp_exchange':
/shared/nagesh/ROMS-2.2/distribute.f90:702: undefined reference to `mpi_irecv_'
libUTIL.a(distribute.o)(.text+0x6779):/shared/nagesh/ROMS-2.2/distribute.f90:706: undefined reference to `mpi_irecv_'
libUTIL.a(distribute.o)(.text+0x67f1):/shared/nagesh/ROMS-2.2/distribute.f90:710: undefined reference to `mpi_send_'
libUTIL.a(distribute.o)(.text+0x6869):/shared/nagesh/ROMS-2.2/distribute.f90:714: undefined reference to `mpi_send_'
libUTIL.a(distribute.o)(.text+0x68a0):/shared/nagesh/ROMS-2.2/distribute.f90:718: undefined reference to `mpi_wait_'
libUTIL.a(distribute.o)(.text+0x68da):/shared/nagesh/ROMS-2.2/distribute.f90:720: undefined reference to `mpi_error_string_'
libUTIL.a(distribute.o)(.text+0x697a):/shared/nagesh/ROMS-2.2/distribute.f90:725: undefined reference to `mpi_wait_'
libUTIL.a(distribute.o)(.text+0x69b4):/shared/nagesh/ROMS-2.2/distribute.f90:727: undefined reference to `mpi_error_string_'
libUTIL.a(distribute.o)(.text+0x997e):/shared/nagesh/ROMS-2.2/distribute.f90:858: undefined reference to `mpi_irecv_'
libUTIL.a(distribute.o)(.text+0x9a02):/shared/nagesh/ROMS-2.2/distribute.f90:862: undefined reference to `mpi_irecv_'
libUTIL.a(distribute.o)(.text+0x9a7a):/shared/nagesh/ROMS-2.2/distribute.f90:866: undefined reference to `mpi_send_'
libUTIL.a(distribute.o)(.text+0x9af2):/shared/nagesh/ROMS-2.2/distribute.f90:870: undefined reference to `mpi_send_'
libUTIL.a(distribute.o)(.text+0x9b29):/shared/nagesh/ROMS-2.2/distribute.f90:874: undefined reference to `mpi_wait_'
libUTIL.a(distribute.o)(.text+0x9b63):/shared/nagesh/ROMS-2.2/distribute.f90:876: undefined reference to `mpi_error_string_'
libUTIL.a(distribute.o)(.text+0x9c03):/shared/nagesh/ROMS-2.2/distribute.f90:881: undefined reference to `mpi_wait_'
libUTIL.a(distribute.o)(.text+0x9c3d):/shared/nagesh/ROMS-2.2/distribute.f90:883: undefined reference to `mpi_error_string_'
libUTIL.a(distribute.o)(.text+0xe5a1): In function `mp_gather':
/shared/nagesh/ROMS-2.2/distribute.f90 undefined reference to `mpi_irecv_'
libUTIL.a(distribute.o)(.text+0xe9a2):/shared/nagesh/ROMS-2.2/distribute.f90 undefined reference to `mpi_wait_'
libUTIL.a(distribute.o)(.text+0xf285):/shared/nagesh/ROMS-2.2/distribute.f90 undefined reference to `mpi_isend_'
libUTIL.a(distribute.o)(.text+0xf2ae):/shared/nagesh/ROMS-2.2/distribute.f90 undefined reference to `mpi_wait_'
libUTIL.a(distribute.o)(.text+0xfdd9): In function `mp_reduce':
/shared/nagesh/ROMS-2.2/distribute.f90 undefined reference to `mpi_allgather_'
libUTIL.a(distribute.o)(.text+0x130dc): In function `mp_scatter':
/shared/nagesh/ROMS-2.2/distribute.f90 undefined reference to `mpi_bcast_'
libMODS.a(mod_parallel.o)(.text+0x35): In function `initialize_parallel':
/shared/nagesh/ROMS-2.2/mod_parallel.f90:94: undefined reference to `mpi_comm_size_'
gmake: *** [/shared/nagesh/ROMS-2.2/roms_io_r_adv4] Error 1
rm mod_kinds.f90

Actually it ran upto end but at the end it had given me this error. I used the same ifort compiler and netcdf compiled with ifort which I had used for single processor (serial mode).
As preliminary step I did this.
Can anybody tell me about what we have to do to work with multiple processors like my problem.
Please help in this aspect.
Thanks in advance.

With best regards,
G.NageswaraRao.

kate · #2 Unread post by **kate** » Tue Apr 29, 2008 5:28 pm

Your system needs to have the MPI library installed and you need to be linking to it. It would have things like mpi_barrier and all the other routines you are missing there. There are a couple free ones out there to download and compile, but I've never done that since the system guys take care of it here. Some versions create a shell wrapper called mpif90 that knows about the mpif.h include file and the MPI library. Do you have mpif90 on your system?

On our Sun Linux cluster, we have three Fortran compilers. The guys complain because each one needs its own MPI library. Also, there's a piece that works with the switch (Voltaire infiniband in our case) and would depend on how your processors communicate with each other.

nageswararao · Fri May 02, 2008 6:21 am

Thanks Kate.
But I could compile the code with HP-MPI. But its not running since I am not having licensed version. So again back to the problem. Can anybody explain me how to use MPICH (which I am having license here) to run ROMS-2.2 on multi-processors.

Thanks in advance.

Wtih best regards,
G.NageswaraRao.

kate · #4 Unread post by **kate** » Fri May 02, 2008 6:11 pm

This isn't really a ROMS topic, more of an MPI topic. You need to get to where a simple "hello, world" sort of program compiles and runs first. There are several free implementations of MPI out there. MPICH, LAM, OpenMPI are ones I've heard of, with MPICH being the first but not necessarily the best. There are probably whole forums out there about MPI.