Compiling using MPI anf ifort -> mpiifort

[c.drinkorn]

Hi everyone,

after having tried to find the source of error myself during the whole day, I resign and ask for help here:

I want to compile a test case for parallel processing and use openmpi and ifort. I already switched on USE_MPI, of course, and also pointed to mpiifort instead of mpif90 in the Compiler file due to the default gfortan in mpif90. In the build file I point to the mpi library (parallel fortran netcdf open mpi) and using Netcdf4, I also point to the respective nf-config.
After a while of compilation I receive tons of "undefined reference" errors regarding MPI, mpi or ompi variables.
The errors occur regarding four library links:
libhdf5.so
libnetcdf.so
libpnetcdf.so
libmpi_mpifh.so.12

I am already so confused about where to put which library link that it may well be that I just completely lost track on where I put what. So if someone could supply me with a "map" of where to put which pointer in the build and the Compiler file, or a "receipe" for mpiifort usage, he or she would save my week.

[kate]

The supercomputer I use has a "modules" feature. Depending on which modules I load, I can either be in an ifort universe or a gfortran universe. Both want the mpi compiler to be called mpif90. For one:

chinook04 674% which mpif90
/opt/scyld/openmpi/1.10.7/intel/bin/mpif90

and for the other:

chinook04 329% which mpif90
/usr/local/pkg/mpi/OpenMPI/1.10.3-GCC-5.4.0-2.26/bin/mpif90

Either way, I want

USE_MPIF90 ?= on

Anyway, if your computer doesn't have these modules, you want to give the full path for the appropriate mpif90.

[c.drinkorn]

Hi Kate!
Your hint was very valuable, I managed to find the module which provides the openmpi mpif90 environment for intel! Thank you!
(For users having the same issue: typing something like "module whatis" gives additonal descriptions of the modules and makes it easier to identify the right one. )
This ...almost saved my week because now I receive these three library conflict warnings:

ld: warning: libmpi_usempi_ignore_tkr.so.6, needed by /sw/rhel6-x64/netcdf/netcdf_fortran-4.4.3-parallel-openmpi-intel14/lib/libnetcdff.so, may conflict with libmpi_usempi_ignore_tkr.so.0
ld: warning: libmpi_mpifh.so.12, needed by /sw/rhel6-x64/netcdf/netcdf_fortran-4.4.3-parallel-openmpi-intel14/lib/libnetcdff.so, may conflict with libmpi_mpifh.so.2
ld: warning: libmpi.so.12, needed by /sw/rhel6-x64/netcdf/netcdf_fortran-4.4.3-parallel-openmpi-intel14/lib/libnetcdff.so, may conflict with libmpi.so.1

I figured that libmpi_usempi_ignore_tkr.so.0, libmpi_mpifh.so.2 and libmpi.so.1 are set by the openmpi library. I have this one now (the only appropiate one I found):

which mpif90
/sw/rhel6-x64/mpi/openmpi-1.8.4-intel14/bin/mpif90

So apparently mpif90 and fortran netcdf for openmpi do not match. I don't really have many options to choose here for this combination, though. Maybe I am still misunderstanding something...

[kate]

They are only warnings. Did it build an executable? Can you try running it?

Also, you want to make sure netcdf-fortran was compiled with the same compiler you are using to compile ROMS. Always.

[c.drinkorn]

I just solved it myself. I saw that you have a much newer openmpi so I looked again and found openmpi-1.10.7_hpcx-intel14. I thought it would be a wrong one since it comes with the hpcx suffix but the compiling worked without error. So thank you again!
HOWEVER, I only have an oceanG and no oceanM. I want to run the DAMEE_4 test case. I know this one is old but it's only for computation estimates... This one is made for parallel processing, too, right? It comes with default 2 by 2 tiles domain decomposition so I thought it should work in parallel mode, too...

[c.drinkorn]

They are only warnings. Did it build an executable? Can you try running it?

Also, you want to make sure netcdf-fortran was compiled with the same compiler you are using to compile ROMS. Always.

I missed this one... Ok, well it built an oceanG. But I was expecting an oceanM.

Regarding the netcdf fortran library: Yes, I use the openmpi parallel netcdf fortran library.

[kate]

The oceanG means you compiled it with USE_DEBUG=on. You get oceanG with or without MPI when in debug mode.

[c.drinkorn]

Yes, I know. But I did comment out USE_DEBUG=on...
However, I found that in the makefile it was not. When I did it I got an oceanM. Happy end.
Thank you for your help, Kate!