ROMS on XSEDE systems

[wilkin]

Does anyone have any recent experience running ROMS on one of the XSEDE (https://www.xsede.org/home Extreme Science and Engineering Discovery Environment - the successor to TERAGRID) high performance computing systems?

I'd appreciate hearing any advice on what hosts have proven useful for running long, modest sized ROMS applications, and whether users encountered any particular difficulties building and running the model, and experience with the bandwidth to these hosts if there a large amount of data to return.

I'm looking at requesting an allocation on host "stampede" at UT Austin.

[kchen]

John,

No problem building ROMS on Stampede, and it's fairly fast (IfinityBand FDR, 16-processor/node Xeon E5 (sandy bridge),pretty good I/O hardware ).

Quota of /work directory is 1T. No limit writing on the purged-storage /scratch (7.5P).

However, the max runtime is only 48-hr. The other major downside is that under heavy-load, the wait could be long. It easily takes hours before the job gets running.

Ke

[vincentcombesOSU]

Dear John and Ke,

it seems that you were able to run ROMS on stampede from utexas (although your posts are from 2014)
I do have a question for you about this.


While I was able to get ROMS run there (now stampede2) using openmp, I am having trouble making it run using mpi.
The compiling does not complain (using ifort / mpfi90 / mvapich2)
When I run however I get the following errors (depending on the mpfi90 flags):

forrtl: severe (194): Run-Time Check Failure. The variable 'distribute_mod_mp_mp_reduce_1d_$SERROR' is being used in 'distribute.f90(4812,19)' without being defined
when I use the following fflags:
mpif90 -c -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit

or

Fatal error in PMPI_Bcast: Message truncated, error stack:
PMPI_Bcast(1645)............: MPI_Bcast(buf=0x2b6bf2e372e0, count=20, MPI_DOUBLE_PRECISION, root=0, MPI_COMM_WORLD) failed
when I use the following fflags:
mpif90 -c -fp-model precise -heap-arrays -ip -O3


I am guessing that I cannot run it because I don't have the right flags for mpi.


If you were able to run it there using mpi, could you please share your Linux-ifort.mk ?
Thank you

Or any other tips would be great

Vincent

[vincentcombesOSU]

Just found what I was missing.

I did not have defined REDUCE_ALLREDUCE

everything is fine now