a problem in modeling ROMS with SWAN by using MCT

[chenyf]

Hi,

I'm trying modeling ROMS(3.0) with SWAN but came across the following problem:

"mpif90 -c -fast -fixed -I/opt/MCT/include waves_coupler.f90
fortcom: Error: waves_coupler.f90, line 17: Error in opening the Library module file. [M_MCTWORLD]
use m_MCTWorld, only: MCTWorld_init => init
............"

I have make some change in the Linux-mpif90.mk as following:

"ifdef SWAN_COUPLE
FFLAGS = -fast -fixed -I/opt/MCT/include
MCT_LIBDIR ?= /opt/MCT/lib
LIBS += -L$(MCT_LIBDIR) -lmct -lmpeu
endif"

(MCT is installed at /opt/MCT on my machine)

But it does not work.

Is there anybody who can give me some suggestion of this problem? Thank you so much!

[jcwarner]

In the makefile, did you also activate
SWAN_COUPLE := on

[chenyf]

Hi, jcwarner

I had turned that to be ON. I think my problem is mainly related with the MCT, but I've no idea what else should i do with the MCT.

Anyway, thank you so much for your kind reminder.

[jcwarner]

what is returned for:
ls /opt/MCT/include
and for
ls /opt/MCT/lib

[chenyf]

HI,jcwarner,

the return for ls /opt/MCT/lib is:

libmct.a libmpeu.a

and the return for ls /opt/MCT/include is:

M_ACCUMULATORCOMMS.mod M_GLOBALTOLOCAL.mod M_REALKINDS.mod
M_ACCUMULATOR.mod M_INDEXBIN_CHAR.mod M_REARRANGER.mod
M_ATTRVECTCOMMS.mod M_INDEXBIN_INTEGER.mod M_ROUTER.mod
M_ATTRVECT.mod M_INDEXBIN_LOGICAL.mod M_SORTINGTOOLS.mod
M_ATTRVECTREDUCE.mod M_INPAK90.mod M_SPARSEMATRIXCOMMS.mod
M_CHARS.mod M_IOUTIL.mod M_SPARSEMATRIXDECOMP.mod
M_CONVERTMAPS.mod M_LIST.mod M_SPARSEMATRIX.mod
M_DIE.mod M_MALL.mod M_SPARSEMATRIXPLUS.mod
M_DROPDEAD.mod M_MATATTRVECTMUL.mod M_SPARSEMATRIXTOMAPS.mod
M_EXCHANGEMAPS.mod M_MCTWORLD.mod M_SPATIALINTEGRAL.mod
M_FILENAME.mod M_MERGE.mod M_SPATIALINTEGRALV.mod
M_FILERESOLV.mod M_MERGESORTS.mod M_STDIO.mod
M_FLOW.mod M_MPIF90.mod M_STRING.mod
M_GENERALGRIDCOMMS.mod M_MPIF.mod M_STRTEMPLATE.mod
M_GENERALGRID.mod M_MPOUT.mod M_TRACEBACK.mod
M_GLOBALMAP.mod M_NAVIGATOR.mod M_TRANSFER.mod
M_GLOBALSEGMAPCOMMS.mod M_PERMUTER.mod M_ZEIT.mod
M_GLOBALSEGMAP.mod M_RANKMERGE.mod

[jcwarner]

the contents of those directories look ok.
Here are some thoughts:
1)Try this:
make clean
make depend
make

2) One needs to compile MCT with the same flags as compiling ROMS. These issues are similar to another recent thread about compiling netcdf. I noticed that your FFLAGS are
FFLAGS = -fast -fixed -I/opt/MCT/include
This really should be something like:

ifdef SWAN_COUPLE
FFLAGS += -Mfixed -I/usr/local/mct/include
LIBS += -L$(MCT_LIBDIR) -lmct -lmpeu
endif

Notice the +=, so that these flags are added when swan_couple is activated. Did you build MCT with the same flags as you are using for ROMS?

[chenyf]

I've tried "make depend" and it showed something like:

"Can't find file: rp_prsgrd42.h
Can't find file: tl_fasham.h
Can't find file: tl_prsgrd44.h
Can't find file: tl_prsgrd42.h
Can't find file: netcdf.inc
Can't find file: propagator_so.h
Can't find file: propagator_so_trace.h
Don't know where module ad_balance_mod lives.
Don't know where module m_sparsematrix lives."

----------BUT I also tried "make depend" in the folder where I run successfully ROMS(without SWAN) with MPI and it showed the same information.

As for FFLAGS, the original one was

"FFLAGS = -fast -Mfixed -I/usr/local/mct/include"
(and I first changed it to be: "FFLAGS = -fast -Mfixed -I/opt/MCT/include"
But it failed very soon when I tried to compile it and then i changed it to be:

"FFLAGS = -fast -fixed -I/opt/MCT/include"
and it went a little further but still had problem as I mentioned above.

(Just now, I tried FFLAGS += -fast -fixed -I/opt/MCT/include, but it stoped because of the same problem with "FFLAGS = -fast -fixed -I/opt/MCT/include")

(BTW, as for the FFLAGS I used without coupling SWAN, it is:
FFLAGS += -03)

[jcwarner]

I would like to know what FFLAGS were used when you compiled MCT. So go to the MCT directory, and look for a file called Makefile.conf and post all the contents in there please.

[chenyf]

The Makefile.conf:

# COMPILER, LIBRARY, AND MACHINE MAKE VARIABLES

# FORTRAN COMPILER VARIABLES #

# FORTRAN COMPILER COMMAND
FC = pathf90

# FORTRAN AND FORTRAN90 COMPILER FLAGS
FCFLAGS = -O2 -m64

# FORTRAN90 ONLY COMPILER FLAGS
F90FLAGS =

# FORTRAN COMPILE FLAG FOR AUTOPROMOTION
# OF NATIVE REAL TO 8 BIT REAL
REAL8 = -r8

# FORTRAN COMPILE FLAG FOR CHANGING BYTE ORDERING
ENDIAN =

# INCLUDE FLAG FOR LOCATING MODULES (-I, -M, or -p)
INCFLAG = -I

# INCLUDE PATHS (PREPEND INCLUDE FLAGS -I, -M or -p)
INCPATH = -I/opt/mpich.path/include

# MPI LIBRARIES (USUALLY -lmpi)
MPILIBS = -L/opt/mpich.path/lib -lmpich -lmpichf90

# PREPROCESSOR VARIABLES #

# COMPILER AND OS DEFINE FLAGS
DEFS = -DSYSLINUX -DCPRPATHSC

# SET A SEPARATE PREPROCESSOR COMMAND IF FORTRAN COMPILER
# DOES NOT HANDLE PREPROCESSING OF SOURCE FILES

# FORTRAN PREPROCESSOR COMMAND
FPP = cpp

# FOTRAN PREPROCESSOR FLAGS
FPPFLAGS = -P -C -N -traditional

# C COMPILER VARIABLES #

# C COMPILER
CC = pathcc

# C COMPILER FLAGS - APPEND CFLAGS
ALLCFLAGS = -DFORTRAN_UNDERSCORE_ -O

# LIBRARY SPECIFIC VARIABLES #

# USED BY MCT BABEL BINDINGS
COMPILER_ROOT =
BABELROOT =
PYTHON =
PYTHONOPTS =
# SIZE OF FORTRAN REAL AND DOUBLE
FORT_SIZE =

# SOURCE FILE TARGETS #

# ENABLE RULE FOR PROCESSING C SOURCE FILES
# BY SETTING TO .c.o
CRULE = .c.o

# ENABLE RULE FOR PROCESSING F90 SOURCE FILES
# BY SETTING TO .F90.o AND DISABLING F90RULECPP
F90RULE = .F90.o

# ENABLE RULE FOR PROCESSING F90 SOURCE FILES
# WITH SEPARATE PREPROCESSING INVOCATION
# BY SETTING TO .F90.o AND DISABLING F90RULE
F90RULECPP = .F90RULECPP

# INSTALLATION VARIABLES #

# INSTALL COMMANDS
INSTALL = /opt/MCT/src/install-sh -c
MKINSTALLDIRS = /opt/MCT/src/mkinstalldirs

# INSTALLATION DIRECTORIES
abs_top_builddir= /opt/MCT/src
MCTPATH = /opt/MCT/src/mct
MPEUPATH = /opt/MCT/src/mpeu
EXAMPLEPATH = /opt/MCT/src/examples
MPISERPATH =
libdir = /opt/MCT/lib
includedir = /opt/MCT/include

# OTHER COMMANDS #
AR = ar cq
RM = rm -f




# USED BY MPI-SERIAL LIBRARY

[jcwarner]

here are some differences that I am not sure about:

Compilation of MCT used:
FC = pathf90
FCFLAGS = -O2 -m64

Compilation of ROMS is using
FC = mpif90
FFLAGS = -c -fast -fixed

Questions :
1) Does mpif90 use pathf90 ??
2) What kind of system is this ? 32, 64 bit? Do you need the -m64 flag to compile roms?

[chenyf]

Hi,jcwarner,

I think i get what you said by the compilation of MCT used. And I re-installed and built MCT on my machine by mpif90 and i think it works.

But I came across another problem which shows something like:

"mpif90 -fast -fixed -I/home/chenyf/mct/MCT/include master.o ocean_control.o ocean_coupler.o propagator.o refdif_couple.o -o oceanM libSWAN.a libADM.a libRPM.a libTLM.a libNLM.a libICE.a libUTIL.a libMODS.a -L/opt/netcdf/current/lib -lnetcdf -L/home/chenyf/mct/MCT/lib -lmct -lmpeu
IPO: performing multi-file optimizations
IPO: generating object file /tmp/ipo_ifortB1QsU0.o
IPO Error: unresolved : mod_ncparam_mp_trstindx_
Referenced in /tmp/ipo_ifortB1QsU0.o
................
/opt/intel/fc/9.0/lib/libifcore.a(for_open_proc.o)(.text+0xc14): In function `for__compute_filename.':
: warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/opt/mpich.intel/lib/libmpich.a(p4_secure.o)(.text+0x4d): In function `start_slave':
: warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/opt/mpich.intel/lib/libmpich.a(p4_sock_util.o)(.text+0x16ba): In function `gethostbyname_p4.':
: warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/tmp/ipo_ifortB1QsU0.o(.text+0x7c): In function `MAIN__':
..................

make: *** [oceanM] Error 1



DO you have anyidea about how to figure out this problem?
Thank you so much for your valuable time.

[jcwarner]

Glad to hear back from you. I was wondering what had happened.

Looks like the compilation made it past the MCT library issue (although it is not fully linked yet!). So I assume that the problem was using different compilers for MCT as compared to ROMS? (i know other users will have this same issue, so if we can figure that one out now it will help in the future).

For the current issue - i am not sure.
There is a variable tRSTindx in mod_ncparam. Can you check that file and search for that variable?

[chenyf]

Hey,jcwarner

As for the compiler, i use FC=mpif90 for ROMS/SWAN and FC=mpif90 as well as CC=mpicc for MCT.

But i'm not quite sure about what you said about "tRSTindx". I can find that in mod_ncparam.F, and in the "Output NetCDF variables IDs" part, it says:

"integer :: tRSTindx(Ngrids) ! restart time record index"

and during the "Initialize several variables" part, it says:

" tRSTindx(ng)=0".

Do you mean those information by saying checking it?

[chenyf]

..

[jivica]

Hej, just wondering if there is any gain in compiling MCT using mpicc?!
(What do you use for netcdf lib compilation?)
We are using SWAN+ROMS but using for fortran mpif90 and for preprocessor cpp. One more thing, if using ifort and -O3 we got blowups quite often, and by reducing so aggressive optimization with -O2 everything is running smoothly.

Cheers, Ivica

[chenyf]

I made some changes about the FFLAGS in the compiler makefile

"ifdef SWAN_COUPLE
FFLAGS = -fixed -I/home/chenyf/mct/MCT/include"

which is originally
"FFLAGS = -fast -Mfixed -I/home/chenyf/mct/MCT/include"

and then it can be compiled successfully.


!!!!!

BUT still with some problem in running it. I'll double check it and write the error information later.

[chenyf]

Now i have trobule with submiting the job.

First I would like to ask whether the following input file is necessary or not:
"ROMS/External/inlet_test2_grid.nc"?
(When I tried to run it, it said:
READ_PHYPAR - could not find input file:ROMS/External/inlet_test2_grid.nc.
BUT I can not find such file under /ROMS/External/)

Second, I'm a little confusing about the processor distribution.
When we set "NtileJ=4" in "ocean_inlet_test.in", does that mean we use 4 as J-direction partition for both ROMS and SWAN or only for ROMS? If only for ROMS, where can i set the I/J-direction partition for SWAN?

(All I can find is the total processor set for ROMS/SWAN in "coupling_inlet_test.in" :

nodesATM = 0 ! atmospheric model
NnodesWAV = 2 ! wave model
NnodesOCN = 2 ! ocean model

BUT how can we set I/J-direction partition seperately for SWAN?)

[jcwarner]

Sounds like there needs to be some documentation on this. I will give you some guidance below. The version you have has been modified, we are working on this now, and so the roms 3.0 may be slightly different. I will work on posting a very detailed decription on the ROMS Wiki. But for now here are some quick answers.

For a coupled application, the job is submitted with the coupling.in file as the main controlling input file, such as :

mpiexec -np 6 -hostfile nodes.list ./oceanM ROMS/External/coupling.in

or something like that.

Edit coupling.in and set the number of processors to be allocated for each model:
NnodesATM = 0 ! atmospheric model
NnodesWAV = 2 ! wave model
NnodesOCN = 4 ! ocean model
We do not have the atmospheric model coupled yet, so don't worry about that. Set the number of nodes for the ocean (here i set 4) and the number of nodes for wave (here i set 2). Notice that 4+2 =6, and i asked for 6 nodes on my mpi call above (as an example).

Then set the synchronization interval:
TI_ATM_WAV = 60.0d0 ! atmosphere-wave coupling interval
TI_ATM_OCN = 10.0d0 ! atmosphere-ocean coupling interval
TI_WAV_OCN = 60.0d0 ! wave-ocean coupling interval

Again, do not worry about the ATM linking yet. So we only need to worry about TI_WAV_OCN = time interval of coupling between wave and ocean. In this case it is set at 60 seconds (the user can change this of course). The user needs to be smart here. So if we want the 2 models to communicate every 60 seconds, make sure that the dt of each model divides evenly into 60. So have roms run on a 10, 20 30 or 60 sec step (for example). Same for the wave model.

Then enter the input file names for the wave and ocean models:
ATM_name = atmos.in ! atmospheric model
WAV_name = wave.in ! wave model
OCN_name = ocean.in ! ocean model
Dont worry about atm name. OCN_name could be ROMS/External/ocean.in or whatever. For the wave name, your version requires the file name to be INPUT. Just that, INPUT. And it needs to be located in the SWAN directory. So put
WAV_name = SWAN/INPUT

Now you need to worry about the roms input file and the SWAN input file.
For roms, it is the same as you would do normally. Set the tile partitions for roms. I had chosed 4 tiles above, so set ntileI=1, ntileJ=4 or whatever.
For SWAN, it does its own tiling. You can not tell SWAN how to partition the grid. It will figure it out itself. The fact that you set NnodesWAV = 2 (or whatever) will allocate that number of processors to swan.

see if that is enough to get you going.

SWAN itself is a very comlicated model. It needs boundary conditions, wind stress, etc. You should familiarize yourself with swan , and run some applications with just swan alone before trying a coupled application.

[chenyf]

I've tried to follow your instruction, but it still failed and the error information shows that we need some input file which can not be found:

"READ_PHYPAR - could not find input file: ROMS/External/inlet_test2_grid.nc"

##############

I'm runing the HEAD TEST case and it seems that it's running well at least till now. So for the failure of INLET case, maybe it's just because of the lack of "inlet_test2_grid.nc" file?