Speed-up advice?

[kate]

We are hoping to have a long simulation of a Pan-Arctic grid with a BGC model. My colleagues have therefore obtained a computing grant at NCAR.
I have done a few timings of a day-long simulation on Cheyenne. I'm finding it no faster than a not-so-new Penguin Supercomputing cluster at home:

On 216 cores: 1.68 hours (6x36 tiling)

On 432 cores: 1.0 hours (8x48 tiling)

Compared to Chinook:

On 192 cores: 1.71 hours (6x32 tiling)

This is a large domain using the ROMS model with a complex BGC component, such that the tracer timestepping is over 65% of the run time. I'm using TS_MPDATA because otherwise I get instabilities in the BGC fields at the open boundaries.

I believe we are limited by the memory bandwidth of the system. This is a very large simulation, taking 340 GB for the 216 cores and 530 GB for the 432 cores (according to the internal reporting). Do you have any advice for ways to possibly speed this thing up on Cheyenne?

[jcwarner]

we have been using TS_HSIMT in COAWST. it is a pos def scheme, faster than mpdata. would be nice to know how the tracer advection compares between the 2 schemes. Have you tried different tiling? how many cores / node does it have?

[kate]

Thanks, John, I'd forgotten about that TS_HSIMT option. I'll give it a whirl.

I haven't tried other tilings. I heard from their consultants saying the two systems are remarkably similar so they would expect the performance I got. Cheyenne has 36 cores/node.