Some problems about using MPICH2 compile ROMS

[s091334]

Dear all:
I'm a new user about ROMS(Region Ocean Modeling System).
And I used WindosXP, Cygwin and g95 on my personal computers (Intel Core 2 Quad).
Before I installed MPICH2, I can use g95 to compile ROMS. And can produce oceanS.
I could use oceanS running my case, but spent to many time, so tried to install MPICH2.

In my Compilers/CYGWIN-g95.mk (I send it in appendix.)
ifdef USE_MPI
MPI_INCDIR ?= /MPICH2/include
MPI_LIBDIR ?= /MPICH2/lib
LIBS += "-L$(MPI_LIBDIR)/fmpich2.lib "
CPPFLAGS += -DMPI -I$(MPI_INCDIR)
FFLAGS += -I$(MPI_INCDIR)
endif

Because need "fmpich2.lib", so I chose WinXP version (mpich2-1.4.1p1-win-ia32.msi) to install.
And I had added right path on my cygwin/etc/profile
MPI_BINDIR=/MPICH2/bin
export MPI_BINDIR
MPI_INCDIR=/MPICH2/include
export MPI_INCDIR
MPI_LIBDIR=/MPICH2/lib
export MPI_LIBDIR

In my makefile
USE_MPI = on
USE_MPIF90 =
FORT = g95

When I type "make", I get some error about MPI. (detail saved in make_sh_mpif90_off_20120920.txt)
Build/libICE.a Build/libUTIL.a Build/libMODS.a -L/usr/local/lib -lnetcdf "-L/MPICH2/lib/fmpich2.lib "
Build/master.o:master.f90:(.text+0xe): undefined reference to `mpi_init_'
Build/master.o:master.f90:(.text+0x34): undefined reference to `mpi_comm_rank_'
Build/master.o:master.f90:(.text+0x5d): undefined reference to `mpi_finalize_'

(1) Can anyone give me some advices to solve the problem?
(2) If I should change to use g95 to install MPICH2 source code.
But it will not found "fmpich2.lib", so how to modify CYGWIN-g95.mk?

Best regards,
Chi-Che Hsieh

[m.hadfield]

This is more a MPICH2 on Windows problem than a ROMS problem. There might be an MPICH2 forum somewhere wher people could help you.

That said, I have run MPI on Cygwin successfully in the past (though for the past year or two it seems to have been broken) so I can offer a few pieces of advice.

The first is that you will probably be better off using OpenMP with Gfortran. This can be installed with Cygwin setup. G95 used to be the best free compiler for ROMS on Cygwin, but that stopped being true several years ago. Gfortran does everything you want, is easy to install and supports OpenMP out of the box.

The second is that if you want MPI on Cygwin, I think you would be better off building the package from source in the Cygwin environment rather than using pre-built Windows binaries. I can't say for user that the Windows binaries wouldn't work, but I doubt they would.

The third piece of advice is that Cygwin is not a great platform for MPI, because it's not very good at inter-process communication.

Finally, if you are building and running ROMS with MPI, use the mpif90 script rather than trying to find the right combination of compiler and linker options by yourself.

[m.hadfield]

I was inspired by your question to take another look at the status of MPI on my system (PC with dual-core Intel, Windows 7, Cygwin). I have MPICH2 1.4.1p1 installed. I did this by building from source. The commands were

Code: Select all

CC=gcc FC=gfortran ./configure --prefix=/usr/local
nice make
nice make install

I got screeds of errors when I tried "make testing" but I seem to be able to build and ROMS with it nonetheless. I set USE_MPIF90 to on so that the mpif90 script is invoked.

There is one significant irritant: the Windows Firewall insists I enter an administrator account name and password whenever I run a ROMS executable for the first time. I haven't bothered to find a cure for this as I don't actually use MPI on this machine in any serious way. (I normally run on a Unix host. If I want to run on the desktop machine I use serial or OpenMP.)

[s091334]

Thanks your advices, Mark
And I'm sorry about too late to replied.

I found another version CYGWIN-g95.mk supported MPICH2 from this page(Modifications to platform-specific make fileshttps://www.myroms.org/projects/src/ticket/212) that you shared.

Now, I can use this version compiling oceanS by Cygwin and g95 in WinXP. And it run well.
My instructor hoped me adding SWAN model and still working on my persional computer using WinXP.

When I installed MPICH2-1.4.1p1, I used command
$./configure F77=g95 FC=g95 --prefix=/MPICH2/ 2>&1 | tee c.txt
$make 2>&1 | tee m.txt
$make install 2>&1 | tee mi.txt
I added right path in cygwin/etc/prfile
MPI_BINDIR=/MPICH2/bin
export MPI_BINDIR
MPI_INCDIR=/MPICH2/include
export MPI_INCDIR
MPI_LIBDIR=/MPICH2/lib
export MPI_LIBDIRThen,
I can see mpif.h and mpi.mod in /MPICH2/include.
And try to run the examples hellw and cpi
User@980622-05 /cygdrive/c/cygwin/mpich2-1.4.1p1/examples
$ make hellow
CC hellow.c
../bin/mpicc -o hellow hellow.o

User@980622-05 /cygdrive/c/cygwin/mpich2-1.4.1p1/examples
$ mpirun -n 2 ./hellow
Hello world from process 0 of 5
Hello world from process 1 of 5

User@980622-05 /cygdrive/c/cygwin/mpich2-1.4.1p1/examples
$ mpirun -n 2 ./cpi
Process 0 of 5 is on 980622-05
Process 1 of 5 is on 980622-05
pi is approximately 3.14159265442311310, Error is 0.0000000008333387
wall clock time = 0.00000

I thought that I finish installing.
And then, I set makefile
USE_MPI = on
USE_MPIF90 = on
FORT = g95
When I typed "make",I got some errors undefined reference about MPI (detail saved in make_sh_mpif90_on_0928.txt)

cd /cygdrive/d/ROMS_for_Square_Harbor/trunk/Build;
/MPICH2/bin/mpif90 -c -fno-second-underscore -O3 -ffast-math master.f90
mpif90 -fno-second-underscore -O3 -ffast-math
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/master.o
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/ocean_control.o
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/ocean_coupler.o
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/propagator.o -o
/cygdrive/d/ROMS_for_Square_Harbor/trunk/oceanM.exe
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/libNLM.a
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/libANA.a
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/libICE.a
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/libUTIL.a
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/libMODS.a -L/usr/local/lib -lnetcdf
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/master.o:master.f90:(.text+0xe): undefined reference to `mpi_init_'
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/master.o:master.f90:(.text+0x34): undefined reference to `mpi_comm_rank_'
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/master.o:master.f90:(.text+0x5d): undefined reference to `mpi_finalize_'
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/ocean_control.o:ocean_control.f90:(.text+0x3e6): undefined reference to `mpi_comm_rank_'
/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/ocean_control.o:ocean_control.f90:(.text+0x3fd): undefined reference to `mpi_comm_size_'

Finally, I had trying Inlet Test Case(https://www.myroms.org/wiki/index.php/INLET_TEST_CASE) just couple ROMS and SWAN.
And when I installed MCT, using command
$./configure FC=mpif90 F90=g95 MPIHEADER=-I/MPICH2/include MPILIBS=-L/MPICH2/lib -prefix=/MCT/ 2>&1 | tee config.txt
$make 2>&1 | tee m.txt
$make install 2>&1 | tee mi.txt

I set Build.bash
USE_MPI = on
USE_MPIF90 = on
FORT = g95

I still got errors and warning messages (detail saved in build_inlet_mpif90_on_0928.txt)
cd /cygdrive/d/ROMS_for_Square_Harbor/trunk/Build; /MPICH2/bin/mpif90 -c -fno-second-underscore -O3 -ffast-math -I/MCT/include ocean_coupler.f90
In file ocean_coupler.f90:111

integer :: OCN_COMM_WORLD, MPI_COMM_WORLD
1
Error: Symbol 'ocn_comm_world' at (1) already has basic type of INTEGER
In file ocean_coupler.f90:337

integer :: OCN_COMM_WORLD
1
Error: Symbol 'ocn_comm_world' at (1) already has basic type of INTEGER
make: *** [/cygdrive/d/ROMS_for_Square_Harbor/trunk/Build/ocean_coupler.o] Error 1

Were the MPICH2 installation problem, bring about these errors?
Can anyone give some advices to solve these problems?

[jcwarner]

the make_sh file shows some problems with incompatible variable types (real vs int) in distribute.f90. If you can not figure this out, and if you are truly going to use the coupled system, it may be better to use our COAWST code than what is here on rutgers site. we have advanced the coupling over there, and have updated swan code, etc.
-john

[m.hadfield]

Hi s091334

I think your problem is with the -fno-second-underscore option. In the past I have recommended using this for building ROMS. (It controls symbol name-mangling. Try a Google search on "g95 -fno-second-underscore Hadfield" for some history.) However this recommendation comes with the proviso that you have to build all other libraries (netCDF, MPICH2) with the same option.

According to my records, the last time I built MPICH2 successfully for use with ROMS/G95/Cygwin was in June 2010 with version 1.2.1p1 and here is the configure command I used

Code: Select all

CC=gcc FC=g95 F90=g95 FFLAGS="-fno-second-underscore" F90FLAGS="-fno-second-underscore" ./configure --prefix=/usr/local/mpich2-g95

The same notes say that I tried to upgrade MPICH2 in June 2011 with 1.3.2p1, that the configure command no longer recognised or accepted FFLAGS or F90FLAGS, that what I used was

Code: Select all

CC=gcc FC=g95 FFLAGS="-fno-second-underscore" ./configure --prefix=/usr/local/mpich2-g95

and that it built OK but wouldn't work with ROMS for reasons I didn't bother elucidating, because I gave up on G95 around that time.

I still think your life would be easier if you switched to Gfortran; avoidance of all this -fno-second-underscore nonsense is one of the reasons why.

Cheers, Mark

[s091334]

Thanks, John and Mark.

I changed to use the mpich2-1.2.1p1 version.
Used the command to install .
$CC=gcc FC=g95 F90=g95 FFLAGS="-fno-second-underscore" F90FLAGS="-fno-second-underscore" ./configure --prefix=/MPICH2/
$make 2>&1 | tee m.txt
$make install 2>&1 | tee mi.txt
$cd $HOME
$touch .mpd.conf
$echo "secretword=abcdef" > .mpd.conf
$chmod 600 .mpd.conf

Also, trying examples hellow and cpi, and it can well done.

And then, I set makefile
USE_MPI = on
USE_MPIF90 = on
FORT = g95
This time, I could compile finish, and produce oceanM, just like Mark said about.
After,I'll check the values that between oceanS and oceanM run my case.

The other things, I tried to run Inlet Test Case(https://www.myroms.org/wiki/index.php/INLET_TEST_CASE)just couple ROMS and SWAN.
I used the command to install MCT.
$CC=gcc FC=mpif90 F90=g95 FFLAGS="-fno-second-underscore" F90FLAGS="-fno-second-underscore" ./configure MPIHEADER=-I/MPICH2/include MPILIBS=-L/MPICH2/lib -prefix=/MCT/ 2>&1 | tee config.txt

Finally, I tried to type "make examples" in MCT files, it also could finish compiling.

In coupling_inlet_test.in set
Nthreads(ocean) = 1
Nthreads(waves) = 1

I used the build.bash to compile Inlet Test case, and it could finish.
./build.bash -j 2

When I used the command to run, .
$mpd &
$mpiexec -np 2 ./oceanM User/External/coupling_inlet_test.in > oceaM_n2_inlettest_20121006.log

I got some error messages
Coupled Input File name = User/External/coupling_inlet_test.in
rank 1 in job 2 980622-05_3343 caused collective abort of all ranks
exit status of rank 1: killed by signal 9
rank 0 in job 2 980622-05_3343 caused collective abort of all ranks
exit status of rank 0: return code 0

How can I solve the problems?

[jcwarner]

i have not been using the coupled part of the Rutgers code. We have dramatically advanced the coupling as part of a separate effort. So if you really want to use a coupled system, you should use our coawst modeling system.
-john

[s091334]

Thanks John.
Now, I think to use ROMS and coupling SWAN model.
then, if it can finish, I will try to use the more complete modeling system(COAWST).