ROMS Upwelling Idealisation / Test Case Failure

[dpath2o]

Hello,

After much time compiling OpenMPI, HDF, NETCDF, with a dark alley in there with Intel compliers only to find that my root does not have enough space ... Anyhow after a good week of attempts I was finally "successful" in compiling ROMS via the attached "build_roms.sh" and predominantly using the package manager of CentOS 8 to get all of the prerequisites installed. I think with the only exception of szip library that was required for the source code of HDF5 and appeared not to install in the yum package version of HDF5 and HDF5-devel, so I just installed it and attempt to give that library to yum HDF5. It appeared to work and so after giving some export commands via BASH to give 'build_roms.sh" some knowledge of the HDF5 and NETCDF libraries without complaining.

However!, after running

Code: Select all

mpirun -np 16 ./romsG roms_upwelling.in > my_upwelling.log

the amount of vomit on my screen was intolerable, ultimately, resulting in this:

Direct leak of 40 byte(s) in 1 object(s) allocated from:
#0 0x7f9acb016db0 in calloc (/usr/lib64/libasan.so.5+0xefdb0)
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[61333,1],15]
Exit code: 1
--------------------------------------------------------------------------

Prior to that barf there were hundreds of lines of this regurgitation:

#0 0x7f91ef724171 in ???
#1 0x7f91ef724d19 in ???
#2 0x7f91ef725e8e in ???
#3 0x40a953 in myroms
at /home/ec2-user/ROMS/projects/upwelling/Build_romsG/master.f90:125
#4 0x40aa00 in main
at /home/ec2-user/ROMS/projects/upwelling/Build_romsG/master.f90:51

Looking at the "master.f90" line 51 it has a rather innocuous looking

Code: Select all

use mod_param

on that line. In fact it's the first line of that F90.

If it's not obvious already, I'm just starting out here on this journey of using ROMS coupled to CICE. I'm feeling a bit dismayed at the amount difficulty I've encountered already, but I suppose I'm learning something. Though over the past week I feel more like a blind person just randomly feeling around in different directions to get the model working then a methodical journey to a known destination.

[dpath2o]

Just to update this post, if other's are interested. I recompiled

Code: Select all

./build_roms.sh

with mpich2, instead of openmpi. The shell environment was nearly identical as openmpi, with the following:

Code: Select all

export NETCDF_INCDIR=/usr/lib64/gfortran/modules/mpich/
export HDF_LIBDIR=$NETCDF_INCDIR
export PATH=/usr/lib64/mpich/bin:$PATH

Then:

Code: Select all

mpirun -np 16 ./romsG roms_upwelling.in > my_upwelling.log

Results in:

ERROR STOP

Error termination. Backtrace:
ERROR STOP

Error termination. Backtrace:
ERROR STOP

...

#0 0x7f95ebc94171 in ???
#1 0x7f95ebc94d19 in ???
#2 0x7f95ebc95e8e in ???
#3 0x40a7e3 in myroms
at /home/ec2-user/ROMS/projects/upwelling/Build_romsG/master.f90:125
#3 0x40a7e3 in myroms
at /home/ec2-user/ROMS/projects/upwelling/Build_romsG/master.f90:125
#4 0x40a890 in main
at /home/ec2-user/ROMS/projects/upwelling/Build_romsG/master.f90:51

Possibly I need to compile with flags or possibly this needs to be done when installing openmpi and mpich2? I've not read about how to do this using the package manager yum, and thought it could only be done when building from source?

[sissalVE]

I recognize some of these errors:
#0 0x7f95ebc94171 in ???
#1 0x7f95ebc94d19 in ???
#2 0x7f95ebc95e8e in ???

I think they are from either netcdf (or other) being compiled with a different compiler, than the ROMS code itself, or if HDF5, szlib or netcdf does not come out with a perfect make check when you build from source. My experience is that you need a stronger compiler than gfortran, I currently use nvfortran.

[sissalVE]

I just checked out the attachments centos_setup.tar, as well, and from the my_upwelling.log file there seems to be an error in the roms_upwelling.in file, where the path for ROMS/External/varinfo.dat is wrong.

[dpath2o]

Hey mate, thank you for your feedback. I really do appreciate your words and consideration. Your second reply struck me as something to immediately remediate, but, alas, nothing meaningful came of that rectification. I spent the past day attempting to pursue your first comment and nix 'gfortran' and ensure that HDF5 and NETCDF, and hence ROMS, were compiled with the same comelier -- intel mpiifort . Unfortunately, HDF5 is not having a bar of my coaxing it to play in these reindeer games and I've gone to my ICT desk at the Uni. They've replied with a Spack bread crumb ( https://spack.readthedocs.io/en/latest/ ... .html#roms ) that I'm now following. Thanks again sissaIVE!

[arango]

Okay, it seems that you are struggling to install the required libraries. The upwelling test case only requires the NetCDF library. You can either use the NetCDF-3 or NetCDF-4 library. The NetCDF-4 library has a lot of options to install. The HDF5 library is only needed for parallel I/O. One of the first things that you need to decide is to pick up the compiler. Then, the MPI library (mpiifort, MPICH, MPICH2, MVAPICH2, or OpenMPI). Installing MPI is complicated and you may need technical support. If using OpenMPI, it needs to be built with the same version of the compiler. Finally, compile ROMS required third-party libraries. We added a page to WikiROMS to guide you on how to compile such libraries, which will help you to play these reindeer games, as you call it

To compile ROMS use build_roms.sh or cbuild_roms.sh (CMake). There are plenty of information in WikiROMS. Also, check out the test repository, which has all the tests for beginners and advanced users.

Good luck

[dpath2o]

Thank you Arango. My apologies for the delayed response, but I eventually got there/here or at least I can now say, I have compiled and run the roms_upwelling.in idealised / test case without errors.

For those who may stumble upon this post and benefit I include the build description here.

Platform: ARDC NECTAR virtual computing environment
Instance: 32 VCPU with 128GB RAM
OS: CentOS 8

1.) Install Intel OneAPI: Base-kit, HPC-kit and IoT-kit
2.) Install GNU Utilities and PERL from package manager
2.) Source-Compile and install libaec (configure options: [PREFIX], ‘with-pic’)
3.) Source-Compile and install zlib (configure options: [PREFIX] )
4.) Source-Compile and install HDF5 (configure options: [PREFIX], ‘enable-shared’ , ‘enable-parallel’ , ‘enable fortran’ , ‘with-pic’ , ‘host=x86_64-pc-none’)
5.) Source-Compile and install NETCDF-C (configure options: [PREFIX] , ‘enable-shared’ , ‘enable-parallel’)
6.) Source-Compile and install NETCDF-FORTRAN (configure options: [PREFIX] , ‘enable-shared’ , ‘enable-parallel’)
7.) Download, build and run ROMS and roms_upwelling.in test case with edits to build_roms.sh, my_build_paths.sh and roms_upwelling.in

Important shell environment definitions and notes:

Code: Select all

. /opt/intel/oneapi/servars.sh
export CC=mpiicc
export FC=mpiifort
export FF=$FC
export CFLAGS=‘ O3 -ip -no-prec-div -fp-model fast=2 -qopenmp -g -mp1 -fno-strict-aliasing -fpic’
export FFLAGS=$CFLAGS
export FCFLAGS=$CFLAGS
# this definition grows with the addition of each source-compiled software, but this is what it looked like at the ROM build stage
export CPPFLAGS=‘-DNDEBUG -DpgiFortran -I/opt/intel/hdf5/include -I/opt/intel/zlib/include -I/opt/intel/netcdf-c/include -I/opt/intel/netcd-fortran/include’
# this definition grows with the addition of each source-compiled software, but this is what it looked like at the ROM build stage
export LDFLAGS=‘-L/opt/intel/hdf5/lib -L/opt/intel/zlib/lib -L/opt/intel/netcdf-c/lib -L/opt/intel/netcdf-fortran/lib’
# PATH already should contain the standard paths and the Intel OneAPI paths, this extended definition grows with the addition of each source-compiled software, but this is what it looked like at the ROM build stage
export PATH=/opt/intel/hdf5/bin:/opt/intel/netcdf-c/bin:/opt/intel/netcdf-fortran/bin:$PATH

WIth that setup (installed code and environment variables), then

Code: Select all

./build_roms.sh -j 16

gave no errors and not very many warnings. Then

Code: Select all

mpirun -np 16 ./romsM rom_upwelling.in > my_upwelling.log

bombed because the number of tiles was not consistent with the number of processors. So running

Code: Select all

mpirun -np 1 ./romsM rom_upwelling.in > my_upwelling.log

worked fine and appeared to use all the processors when I viewed it with the shell environment tool “htop”.

The results match those provided in MyROMS tutorial.

Now on to building my own ROMS. Thanks for your help so far, I’m sure I’ll be back to ask more questions later.

Cheers,
Dan

[arango]

Yes, your learning process just started. To run over a particular number of processes you need to modify parameters NtileI and NtileJ in roms_upwelling.in. The number of process to use is determined by the product of those two parameters.