faster MPI on single computer / -bind-to core /Ubuntu 18.04

[jpringle]

Dear all --

I hope some of you find this useful. I am running ROMS on a dual processor 32 core AMD EPYC machine. I noticed on the latest version of Ubuntu my smaller machines showed good improvement with ROMS runs with openMP parallelization.

So I upgraded the big machine to Ubuntu 18.04, and my ROMS runs became about 15% slower. This new version has a new kernel (4.15) which is supposed to allocate processes to cores in a way that works better with the EPYC memory architecture (it is NUMA with 16 memory channels). It is not doing so by default.

I returned to the old performance (actually, a few percent faster) by invoking mpirun with the "-bind-to core" option -- I run the model with "mpirun -bind-to core -np 32 ./oceanM jnk.in"

Hope this helps someone,
Jamie

[jpringle]

With the same version of Ubuntu, but updated kernels (nothing special, just the default updates), this no longer appears to be true. A reminder to test your tiling and MPI options every so often.

Jamie