ROMS on the IBM Regatta

[Paul_Budgell]

Hi folks,

As I had mentioned in an e-mail last January, our supercomputing here at IMR switched from the SGI Origin 2000 to an IBM Regatta (p690) at Parallab at the University of Bergen. We had tremendous problems running ROMS in parallel on the new system. By playing with some of the compiler switches that Kate Hedstrom suggested, we were able (sometimes) to run the ROMS UPWELLING case on 4 processors, but not consistently on more processors and not without using 'unsafe' optimization.

To try to get a grip on this problem, I created a simple test case (advection-diffusion of a rotating cone) coded in the same style as ROMS that replicated the problem. I sent the code to Jacko Koster at Parallab, University of Bergen, who reproduced the same problem with 50 lines of code. Jacko then found a solution: instead of using the IBM Fortran 77 SMP compiler (xlf_r), use the IBM Fortran 90 SMP compiler (xlf90_r) with the -qfixed compiler flag. The xlf_r compiler seems to have problems with the common blocks in ROMS, whereas the xlf90_r handles them with ease. Jacko sent his test problem to IBM in the US who replicated the bug on their system. The problem lies with the Fortran 77 compiler and IBM say they will release a patch to fix the problem in due course. So, if you are using ROMS on an IBM system under OpenMP, you might want to try these compiler settings:

xlf90_r -c -qfixed -qsmp=omp -O4 -qmaxmem=-1

We are finding the Regatta to be a good platform for running ROMS. On the UPWELLING case, we get a speedup of 11.3 on 16 processors. On larger problems, the performance should increase.

Regards,
Paul Budgell

[rsignell]

Too bad I heard about this later rather than sooner. In my opinion you should always use the F90 compiler on the IBM (xlf90,xlf90_r, etc). Might have saved them some time and trouble. Scaling at 11 out of 16 pe's is about as much as can be expected on this system.

Rich