I've run into a problem working with ROMS that has me a bit confused and I'm hopeful that someone out there can help.
I have set up a model run of the California Current and while testing the initialization of the model, it seems to runs fine on a single processor.
However, when I went to launch the run on 24 processors (tiling; NtileI == 4 NtileJ == 6) to run it for a longer period of time, the simulation "blows-up" within a few time steps.
To try and determine what was happening, I ran the model with various tiling. Some successful (NtileI x NtileJ; 1x1, 2x2, 3x3, 2x4, 2x5, 2x6) and some not (4x4, 5x5, 3x4, 4x3, 2x8). As a starting point, I am defining "successful" as running without blowing up for one day (480 time steps).
I am wondering if this has anything to do with the land mask on my grid. In certain places the mask extends up to 1/3rd of way along the I-dimension. So in certain tiling scenarios (e.g. 5x5) one or more tiles will be entirely masked. Could this be causing my runs to "blow-up" after just a few timesteps?
I created my grid in Seagrid and purposely tried to keep the grid simple (i.e. orthogonal) to begin with. I am aware that I could have reduced some of the masked area by creating a boundary that follows the general contour of the coastline, but was unsure of how non-orthogonality in the grid might influence model results.
I've copied and pasted the initial output from a representative model run below if that is helpful. Please let me know if there is any other information that might be useful.
Thanks for any input you have,
Jeff
******************************************************************************
Code: Select all
Model Input Parameters: ROMS/TOMS version 3.2
Thursday - February 26, 2009 - 9:20:04 AM
-----------------------------------------------------------------------------
Test of California Current System created by ROMSTOOLS
Operating system : Linux
CPU/hardware : i686
Compiler system : pgi
Compiler command : /opt/mpich/pgi/bin/mpif90
Compiler flags : -fastsse -Mipa=fast -tp p6 -Mfree
Input Script : External/nptopc.in
SVN Root URL : https://www.myroms.org/svn/src/trunk
SVN Revision : 313M
Local Root : /home/jeffdorman2/nptopc_roms
Header Dir : /home/jeffdorman2/nptopc_roms/Projects/nptopc
Header file : nptopc.h
Analytical Dir: /home/jeffdorman2/nptopc_roms/Projects/nptopc/Functionals
Resolution, Grid 01: 0148x0183x020, Parallel Nodes: 8, Tiling: 002x004
Physical Parameters, Grid: 01
=============================
87841 ntimes Number of timesteps for 3-D equations.
180.000 dt Timestep size (s) for 3-D equations.
30 ndtfast Number of timesteps for 2-D equations between
each 3D timestep.
1 ERstr Starting ensemble/perturbation run number.
1 ERend Ending ensemble/perturbation run number.
0 nrrec Number of restart records to read from disk.
T LcycleRST Switch to recycle time-records in restart file.
120 nRST Number of timesteps between the writing of data
into restart fields.
1 ninfo Number of timesteps between print of information
to standard output.
T ldefout Switch to create a new output NetCDF file(s).
120 nHIS Number of timesteps between the writing fields
into history file.
1 ntsAVG Starting timestep for the accumulation of output
time-averaged data.
120 nAVG Number of timesteps between the writing of
time-averaged data into averages file.
1 ntsDIA Starting timestep for the accumulation of output
time-averaged diagnostics data.
120 nDIA Number of timesteps between the writing of
time-averaged data into diagnostics file.
0.0000E+00 tnu2(01) Horizontal, harmonic mixing coefficient (m2/s)
for tracer 01: temp
0.0000E+00 tnu2(02) Horizontal, harmonic mixing coefficient (m2/s)
for tracer 02: salt
5.0000E+00 visc2 Horizontal, harmonic mixing coefficient (m2/s)
for momentum.
1.0000E-06 Akt_bak(01) Background vertical mixing coefficient (m2/s)
for tracer 01: temp
1.0000E-06 Akt_bak(02) Background vertical mixing coefficient (m2/s)
for tracer 02: salt
1.0000E-05 Akv_bak Background vertical mixing coefficient (m2/s)
for momentum.
3.0000E-04 rdrg Linear bottom drag coefficient (m/s).
3.0000E-03 rdrg2 Quadratic bottom drag coefficient.
2.0000E-02 Zob Bottom roughness (m).
1.0000E+01 blk_ZQ Height (m) of surface air humidity measurement.
1.0000E+01 blk_ZT Height (m) of surface air temperature measurement.
1.0000E+01 blk_ZW Height (m) of surface winds measurement.
1 lmd_Jwt Jerlov water type.
5.0000E+00 theta_s S-coordinate surface control parameter.
3.0000E-01 theta_b S-coordinate bottom control parameter.
20.000 Tcline S-coordinate surface/bottom layer width (m) used
in vertical coordinate stretching.
1025.000 rho0 Mean density (kg/m3) for Boussinesq approximation.
0.000 dstart Time-stamp assigned to model initialization (days).
19000101.00 time_ref Reference time for units attribute (yyyymmdd.dd)
0.0000E+00 Tnudg(01) Nudging/relaxation time scale (days)
for tracer 01: temp
0.0000E+00 Tnudg(02) Nudging/relaxation time scale (days)
for tracer 02: salt
0.0000E+00 Znudg Nudging/relaxation time scale (days)
for free-surface.
0.0000E+00 M2nudg Nudging/relaxation time scale (days)
for 2D momentum.
0.0000E+00 M3nudg Nudging/relaxation time scale (days)
for 3D momentum.
0.0000E+00 obcfac Factor between passive and active
open boundary conditions.
14.000 T0 Background potential temperature (C) constant.
35.000 S0 Background salinity (PSU) constant.
1027.000 R0 Background density (kg/m3) used in linear Equation
of State.
1.7000E-04 Tcoef Thermal expansion coefficient (1/Celsius).
0.0000E+00 Scoef Saline contraction coefficient (1/PSU).
1.000 gamma2 Slipperiness variable: free-slip (1.0) or
no-slip (-1.0).
T Hout(idFsur) Write out free-surface.
T Hout(idUbar) Write out 2D U-momentum component.
T Hout(idVbar) Write out 2D V-momentum component.
T Hout(idUvel) Write out 3D U-momentum component.
T Hout(idVvel) Write out 3D V-momentum component.
T Hout(idWvel) Write out W-momentum component.
T Hout(idOvel) Write out omega vertical velocity.
T Hout(idTvar) Write out tracer 01: temp
T Hout(idTvar) Write out tracer 02: salt
T Hout(idUsms) Write out surface U-momentum stress.
T Hout(idVsms) Write out surface V-momentum stress.
T Hout(idUbms) Write out bottom U-momentum stress.
T Hout(idVbms) Write out bottom V-momentum stress.
T Hout(idTsur) Write out surface net heat flux.
T Hout(idTsur) Write out surface net salt flux.
T Hout(idSrad) Write out shortwave radiation flux.
T Hout(idLrad) Write out longwave radiation flux.
T Hout(idLhea) Write out latent heat flux.
T Hout(idShea) Write out sensible heat flux.
T Hout(idevap) Write out evaporation rate.
T Hout(idrain) Write out rain rate.
Output/Input Files:
Output Restart File: Data/night_090225runs/ocean_rst_24tile.nc
Output History File: Data/night_090225runs/ocean_his_24tile.nc
Output Averages File: Data/night_090225runs/ocean_avg_24tile.nc
Output Diagnostics File: Data/night_090225runs/ocean_dia_24tile.nc
Input Grid File: External/roms_grd_NPtoPC.nc
Input Nonlinear Initial File: External/roms_ini_NPtoPC_ECCO_Y2004M7.nc
Input Forcing File 01: External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_dlwr.nc
Input Forcing File 02: External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_qair.nc
Input Forcing File 03: External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_rain.nc
Input Forcing File 04: External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_swra.nc
Input Forcing File 05: External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_tair.nc
Input Forcing File 06: External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_uwin.nc
Input Forcing File 07: External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_vwin.nc
Input Boundary File: External/roms_bry_NPtoPC_ECCO_jul2004_dec2005_npzdeqzeros.nc
Biology Parameters File: External/npzd_Powell.in
Tile partition information for Grid 01: 0148x0183x0020 tiling: 002x004
tile Istr Iend Jstr Jend Npts
0 1 74 1 46 68080
1 75 148 1 46 68080
2 1 74 47 92 68080
3 75 148 47 92 68080
4 1 74 93 138 68080
5 75 148 93 138 68080
6 1 74 139 183 66600
7 75 148 139 183 66600
Tile minimum and maximum fractional grid coordinates:
(interior points only)
tile Xmin Xmax Ymin Ymax grid
0 0.50 74.50 0.50 46.50 RHO-points
1 74.50 148.50 0.50 46.50 RHO-points
2 0.50 74.50 46.50 92.50 RHO-points
3 74.50 148.50 46.50 92.50 RHO-points
4 0.50 74.50 92.50 138.50 RHO-points
5 74.50 148.50 92.50 138.50 RHO-points
6 0.50 74.50 138.50 183.50 RHO-points
7 74.50 148.50 138.50 183.50 RHO-points
0 1.00 74.50 0.50 46.50 U-points
1 74.50 148.00 0.50 46.50 U-points
2 1.00 74.50 46.50 92.50 U-points
3 74.50 148.00 46.50 92.50 U-points
4 1.00 74.50 92.50 138.50 U-points
5 74.50 148.00 92.50 138.50 U-points
6 1.00 74.50 138.50 183.50 U-points
7 74.50 148.00 138.50 183.50 U-points
0 0.50 74.50 1.00 46.50 V-points
1 74.50 148.50 1.00 46.50 V-points
2 0.50 74.50 46.50 92.50 V-points
3 74.50 148.50 46.50 92.50 V-points
4 0.50 74.50 92.50 138.50 V-points
5 74.50 148.50 92.50 138.50 V-points
6 0.50 74.50 138.50 183.00 V-points
7 74.50 148.50 138.50 183.00 V-points
Maximum halo size in XI and ETA directions:
HaloSizeI(1) = 172
HaloSizeJ(1) = 116
TileSide(1) = 80
TileSize(1) = 4160
NPZD Model Parameters, Grid: 01
===============================
1 BioIter Number of iterations for nonlinear convergence.
1.7000E+01 BioIni(iNO3_) Nitrate initial concentration (mmol/m3).
1.0000E+00 BioIni(iPhyt) Phytoplankton initial concentration (mmol/m3).
1.0000E+00 BioIni(iZoop) Zooplankton initial concentration (mmol/m3).
1.0000E+00 BioIni(iSDet) Small detritus initial concentration (mmol/m3).
4.3000E-01 PARfrac Fraction of shortwave radiation that is
photosynthetically active (nondimensional).
6.7000E-02 AttSW Light attenuation of seawater (m-1).
9.5000E-03 AttPhy Light attenuation by phytoplankton (m2/mmole_N).
2.5000E-02 PhyIS Phytoplankton growth, initial slope of P-I curve
(m2/W).
1.5000E+00 Vm_NO3 Nitrate upatake rate (day-1).
1.0000E-01 PhyMRD Phytoplankton mortality rate to Detritus (day-1)
0.0000E+00 PhyMRN Phytoplankton mortality rate to Nitrogen (day-1)
1.0000E+00 K_NO3 Inverse half-saturation for phytoplankton NO3
uptake (1/(mmol m-3)).
6.0000E-02 Ivlev Ivlev constant for zooplankton grazing
(nondimensional).
5.2000E-01 ZooGR Zooplankton maximum growth rate (day-1).
0.0000E+00 ZooEED Zooplankton excretion efficiency to Detritus
pool (nondimensional).
3.0000E-01 ZooEEN Zooplankton excretion efficiency to Nitrogen
pool (nondimensional).
1.4500E-01 ZooMRD Zooplankton mortality rate to Detritus (day-1).
0.0000E+00 ZooMRN Zooplankton mortality rate to Nitrogen (day-1).
1.0300E+00 DetRR Detritus remineralization rate (day-1).
0.0000E+00 wPhy Phytoplankton sinking rate (m/day).
8.0000E+00 wDet Detrital sinking rate (m/day).
0.0000E+00 tnu2(03) Horizontal, harmonic mixing coefficient (m2/s)
for tracer 03: NO3
0.0000E+00 tnu2(04) Horizontal, harmonic mixing coefficient (m2/s)
for tracer 04: phytoplankton
0.0000E+00 tnu2(05) Horizontal, harmonic mixing coefficient (m2/s)
for tracer 05: zooplankton
0.0000E+00 tnu2(06) Horizontal, harmonic mixing coefficient (m2/s)
for tracer 06: detritus
5.0000E-06 Akt_bak(03) Background vertical mixing coefficient (m2/s)
for tracer 03: NO3
5.0000E-06 Akt_bak(04) Background vertical mixing coefficient (m2/s)
for tracer 04: phytoplankton
5.0000E-06 Akt_bak(05) Background vertical mixing coefficient (m2/s)
for tracer 05: zooplankton
5.0000E-06 Akt_bak(06) Background vertical mixing coefficient (m2/s)
for tracer 06: detritus
0.0000E+00 Tnudg(03) Nudging/relaxation time scale (days)
for tracer 03: NO3
0.0000E+00 Tnudg(04) Nudging/relaxation time scale (days)
for tracer 04: phytoplankton
0.0000E+00 Tnudg(05) Nudging/relaxation time scale (days)
for tracer 05: zooplankton
0.0000E+00 Tnudg(06) Nudging/relaxation time scale (days)
for tracer 06: detritus
T Hout(idTvar) Write out tracer 03: NO3
T Hout(idTvar) Write out tracer 04: phytoplankton
T Hout(idTvar) Write out tracer 05: zooplankton
T Hout(idTvar) Write out tracer 06: detritus
Activated C-preprocessing Options:
NPTOPC Test of California Current System created by ROMSTOOLS
ATM_PRESS Impose atmospheric pressure onto sea surface.
ANA_BIOLOGY Analytical biology initial conditions.
ANA_BPFLUX Analytical bottom passive tracers fluxes.
ANA_BSFLUX Analytical kinematic bottom salinity flux.
ANA_BTFLUX Analytical kinematic bottom temperature flux.
ANA_PAIR Analytical surface air pressure.
ANA_SPFLUX Analytical surface passive tracer fluxes.
ASSUMED_SHAPE Using assumed-shape arrays.
AVERAGES Writing out time-averaged fields.
BULK_FLUXES Surface bulk fluxes parametererization.
DIAGNOSTICS_TS Computing and writing tracer diagnostic terms.
DIAGNOSTICS_UV Computing and writing momentum diagnostic terms.
DJ_GRADPS Parabolic Splines density Jacobian (Shchepetkin, 2002).
DOUBLE_PRECISION Double precision arithmetic.
EASTERN_WALL Wall boundary at Eastern edge.
EMINUSP Compute Salt Flux using E-P.
LMD_CONVEC LMD convective mixing due to shear instability.
LMD_MIXING Large/McWilliams/Doney interior mixing.
LMD_NONLOCAL LMD convective nonlocal transport.
LMD_RIMIX LMD diffusivity due to shear instability.
LMD_SKPP KPP surface boundary layer mixing.
LONGWAVE_OUT Compute outgoing longwave radiation internally.
MASKING Land/Sea masking.
MIX_GEO_TS Mixing of tracers along geopotential surfaces.
MIX_GEO_UV Mixing of momentum along geopotential surfaces.
MPI MPI distributed-memory configuration.
NONLINEAR Nonlinear Model.
!NONLIN_EOS Linear Equation of State for seawater.
NORTH_FSCHAPMAN Northern edge, free-surface, Chapman condition.
NORTH_M2FLATHER Northern edge, 2D momentum, Flather condition.
NORTH_M3CLAMPED Northern edge, 3D momentum, Clamped condition.
NORTH_TCLAMPED Northern edge, tracers, Clamped condition.
NPZD_POWELL NPZD Biological Model, Powell et al. fomulation.
POWER_LAW Power-law shape time-averaging barotropic filter.
PROFILE Time profiling activated .
!RST_SINGLE Double precision fields in restart NetCDF file.
SALINITY Using salinity.
SOLAR_SOURCE Solar Radiation Source Term.
SOLVE3D Solving 3D Primitive Equations.
SOUTH_FSCHAPMAN Southern edge, free-surface, Chapman condition.
SOUTH_M2FLATHER Southern edge, 2D momentum, Flather condition.
SOUTH_M3CLAMPED Southern edge, 3D momentum, Clamped condition.
SOUTH_TCLAMPED Southern edge, tracers, Clamped condition.
SPLINES Conservative parabolic spline reconstruction.
TS_U3HADVECTION Third-order upstream horizontal advection of tracers.
TS_C4VADVECTION Fourth-order centered vertical advection of tracers.
TS_DIF2 Harmonic mixing of tracers.
UV_ADV Advection of momentum.
UV_COR Coriolis term.
UV_U3HADVECTION Third-order upstream horizontal advection of 3D momentum.
UV_C4VADVECTION Fourth-order centered vertical advection of momentum.
UV_LDRAG Linear bottom stress.
UV_VIS2 Harmonic mixing of momentum.
VAR_RHO_2D Variable density barotropic mode.
WEST_FSCHAPMAN Western edge, free-surface, Chapman condition.
WEST_M2FLATHER Western edge, 2D momentum, Flather condition.
WEST_M3CLAMPED Western edge, 3D momentum, Clamped condition.
WEST_TCLAMPED Western edge, tracers, Clamped condition.
INITIAL: Configuring and initializing forward nonlinear model ...
Vertical S-coordinate System:
level S-coord Cs-curve at_hmin over_slope at_hmax
20 0.0000000 0.0000000 0.000 0.000 0.000
19 -0.0500000 -0.0036887 -1.000 -9.633 -18.266
18 -0.1000000 -0.0083498 -2.000 -21.541 -41.083
17 -0.1500000 -0.0146352 -3.000 -37.252 -71.504
16 -0.2000000 -0.0234720 -4.000 -58.933 -113.866
15 -0.2500000 -0.0361429 -5.000 -89.587 -174.175
14 -0.3000000 -0.0542976 -6.000 -133.076 -260.152
13 -0.3500000 -0.0797586 -7.000 -193.664 -380.328
12 -0.4000000 -0.1139562 -8.000 -274.698 -541.397
11 -0.4500000 -0.1570182 -9.000 -376.479 -743.958
10 -0.5000000 -0.2070749 -10.000 -494.630 -979.260
9 -0.5500000 -0.2607175 -11.000 -621.173 -1231.346
8 -0.6000000 -0.3147622 -12.000 -748.657 -1485.314
7 -0.6500000 -0.3680293 -13.000 -874.321 -1735.642
6 -0.7000000 -0.4218448 -14.000 -1001.269 -1988.538
5 -0.7500000 -0.4794204 -15.000 -1137.016 -2259.033
4 -0.8000000 -0.5450552 -16.000 -1291.625 -2567.251
3 -0.8500000 -0.6237264 -17.000 -1476.744 -2936.489
2 -0.9000000 -0.7211044 -18.000 -1705.644 -3393.288
1 -0.9500000 -0.8438388 -19.000 -1993.887 -3968.773
0 -1.0000000 -1.0000000 -20.000 -2360.360 -4700.720
Time Splitting Weights: ndtfast = 30 nfast = 42
Primary Secondary Accumulated to Current Step
1-0.0008094437383769 0.0333333333333333-0.0008094437383769 0.0333333333333333
2-0.0014053566728197 0.0333603147912792-0.0022148004111966 0.0666936481246126
3-0.0017877524645903 0.0334071600137066-0.0040025528757869 0.1001008081383191
4-0.0019566842408176 0.0334667517625262-0.0059592371166045 0.1335675599008453
5-0.0019122901320372 0.0335319745705535-0.0078715272486418 0.1670995344713988
6-0.0016548570247459 0.0335957175749547-0.0095263842733877 0.2006952520463535
7-0.0011849025289723 0.0336508794757796-0.0107112868023600 0.2343461315221331
8-0.0005032751608631 0.0336903762267453-0.0112145619632231 0.2680365077488784
9 0.0003887272597151 0.0337071520654408-0.0108258347035081 0.3017436598143192
10 0.0014892209965583 0.0336941944901169-0.0093366137069498 0.3354378543044362
11 0.0027955815694920 0.0336445537902317-0.0065410321374577 0.3690824080946679
12 0.0043042707117221 0.0335513677379153-0.0022367614257356 0.4026337758325831
13 0.0060106451121704 0.0334078920475245 0.0037738836864348 0.4360416678801077
14 0.0079087469427945 0.0332075372104522 0.0116826306292293 0.4692492050905599
15 0.0099910761708919 0.0329439123123590 0.0216737068001213 0.5021931174029188
16 0.0122483446563884 0.0326108764399960 0.0339220514565096 0.5348039938429148
17 0.0146692120341107 0.0322025982847830 0.0485912634906204 0.5670065921276978
18 0.0172400033810439 0.0317136245503127 0.0658312668716642 0.5987202166780104
19 0.0199444086685725 0.0311389577709445 0.0857756755402367 0.6298591744489550
20 0.0227631639997064 0.0304741441486588 0.1085388395399431 0.6603333185976138
21 0.0256737146312910 0.0297153720153352 0.1342125541712341 0.6900486906129490
22 0.0286498597812016 0.0288595815276255 0.1628624139524357 0.7189082721405747
23 0.0316613792205219 0.0279045862015855 0.1945237931729576 0.7468128583421601
24 0.0346736416507075 0.0268492068942347 0.2291974348236651 0.7736620652363948
25 0.0376471948657328 0.0256934188392112 0.2668446296893979 0.7993554840756060
26 0.0405373376992232 0.0244385123436867 0.3073819673886211 0.8237939964192927
27 0.0432936737565710 0.0230872677537126 0.3506756411451921 0.8468812641730054
28 0.0458596469320356 0.0216441452951603 0.3965352880772277 0.8685254094681657
29 0.0481720587108284 0.0201154903974257 0.4447073467880561 0.8886408998655913
30 0.0501605672561819 0.0185097551070648 0.4948679140442381 0.9071506549726561
31 0.0517471682814030 0.0168377361985254 0.5466150823256410 0.9239883911711814
32 0.0528456577069106 0.0151128305891453 0.5994607400325517 0.9391012217603267
33 0.0533610761022577 0.0133513086655816 0.6528218161348094 0.9524525304259084
34 0.0531891349131380 0.0115726061288397 0.7060109510479473 0.9640251365547481
35 0.0522156244733761 0.0097996349650684 0.7582265755213234 0.9738247715198165
36 0.0503158038019031 0.0080591141492892 0.8085423793232264 0.9818838856691057
37 0.0473537721847154 0.0063819206892258 0.8558961515079417 0.9882658063583316
38 0.0431818225418188 0.0048034616164019 0.8990779740497604 0.9930692679747334
39 0.0376397765791564 0.0033640675316746 0.9367177506289169 0.9964333355064081
40 0.0305543017255207 0.0021094083123694 0.9672720523544376 0.9985427438187775
41 0.0217382098544505 0.0010909315881854 0.9890102622088881 0.9996336754069629
42 0.0109897377911119 0.0003663245930371 0.9999999999999998 1.0000000000000000
ndtfast, nfast = 30 42 nfast/ndtfast = 1.40000
Centers of gravity and integrals (values must be 1, 1, approx 1/2, 1, 1):
1.000000000000 1.047601458608 0.523800729304 1.000000000000 1.000000000000
Power filter parameters, Fgamma, gamma = 0.28400 0.18933
Minimum X-grid spacing, DXmin = 2.37987620E+00 km
Maximum X-grid spacing, DXmax = 3.21666520E+00 km
Minimum Y-grid spacing, DYmin = 4.18668041E+00 km
Maximum Y-grid spacing, DYmax = 6.65924384E+00 km
Minimum Z-grid spacing, DZmin = 1.00000000E+00 m
Maximum Z-grid spacing, DZmax = 7.31946956E+02 m
Minimum barotropic Courant Number = 2.97227410E-02
Maximum barotropic Courant Number = 4.97801292E-01
Maximum Coriolis Courant Number = 1.84785633E-02
I am sure that such application will run faster in less number of processes. If you have that many CPU available to use, I recommend to run different set-ups simultaneously. That will be a better use of the computational resources. You need to increase the number of nodes as you problem becomes larger and larger. Our benchmark 3 is 2048x256x30 and we use around 24 to 32 nodes. 