ROMS compilation in multi processed environment

[isoares]

Dear all,

I am running Linux Ubuntu 20.04.3 LTS in a multi-processed PC with 2 sockets, 2 Thread(s) per core and 22 cores per socket (see specs below). In theory I should be able to use 88 cores to run ROMS but it is not happening. I tried different configurations of tiles. I tried ntiles_i = 2 and ntiles_j = 44, or ntiles_i = 4 and ntiles_j = 22, or ntiles_i = 8 and ntiles_j = 10. I am using flags USE_MPI = on and USE_MPIF90 = on.

I tried which_MPI = mpich and it didn't work no matter which tiles I try.
Then I tried which_MPI = openmpi and It works if I use 44 cores. Now I am using ntiles_i=2 and ntiles_j = 22

I have version 3.9 of ROMS and I use gfortran, and I specify the entire path to openmpi-4.1.1

I wonder if I am missing a configuration FLAG or anything else, maybe in the config file Linux-gfortran.mk


Any comment will be appreciated.

Best

Ivan

the FLAGS:

export USE_MPI=on # distributed-memory parallelism
#export USE_MPIF90=on # compile with mpif90 script
#export which_MPI=mpich # compile with MPICH library
#export which_MPI=mpich2 # compile with MPICH2 library
export which_MPI=openmpi # compile with OpenMPI library

#export USE_OpenMP=on # shared-memory parallelism

#export FORT=ifort
export FORT=gfortran
#export FORT=pgi

#export USE_DEBUG=on # use Fortran debugging flags
export USE_LARGE=on # activate 64-bit compilation
export USE_NETCDF4=on # compile with NetCDF-4 library
#export USE_PARALLEL_IO=on # Parallel I/O with Netcdf-4/HDF5

export USE_MY_LIBS=on # use my library paths below

The system specs:

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 88
On-line CPU(s) list: 0-87
Thread(s) per core: 2
Core(s) per socket: 22
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 79
Model name: Intel(R) Xeon(R) CPU E5-2696 v4 @ 2.20GHz
Stepping: 1
CPU MHz: 1400.000
CPU max MHz: 3700,0000
CPU min MHz: 1200,0000
BogoMIPS: 4399.83
Virtualization: VT-x
L1d cache: 1,4 MiB
L1i cache: 1,4 MiB
L2 cache: 11 MiB
L3 cache: 110 MiB
NUMA node0 CPU(s): 0-21,44-65
NUMA node1 CPU(s): 22-43,66-87
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable

[robertson]

Your machine has 44 real cores with hyper-threading resulting in 88 CPUs being listed. Open MPI (and all other MPI implementations I have used) looks at your real cores only and that is the limit for mpirun. You can override this by issuing the --oversubscribe flag in your mpirun command but even Open MPI doesn't recommend this and, in our experience, the simulation will run slower this way. Depending on the size of your grid and configured options even 44 cores might be overkill and not optimal for your simulation.

[isoares]

Dear Robertson,

Thank You for the reply. It is the number of Threads that determine the virtual components, right ?

well, thanks a lot.

Best

Ivan