Abnormal Termination of ROMS run

Message

prakrati · #1 Unread post by **prakrati** » Tue Nov 02, 2010 3:19 am

Hey Folks

I am running ROMS with the input file ocean_benchmark4.in with NtileI = 2 and NtileJ = 4 and number of processes = 8 on 1 node.It is giving the following error in the output file :
ERROR: Abnormal termination: NetCDF OUTPUT.
REASON: No such file or directory

Please help asap

Thanks
Prakrati

arango · #2 Unread post by **arango** » Tue Nov 02, 2010 3:28 am

You really do not provide a great deal the information and want our help right away... The benchmark does not use I/O files. You can activate writing history NetCDF files, if you like. You need to redirect standard output into a log file. You will find out that there is a lot of information dumped that helps to diagnose the problem. You are just printing out the last line of standard output. There is not way for us to tell what it is going on.

Please check your standard output carefully.

prakrati · #3 Unread post by **prakrati** » Tue Nov 02, 2010 4:06 am

Code: Select all

Benchmark Test, Idealized Southern Ocean, Large Grid

 Operating system : Linux
 CPU/hardware     : x86_64
 Compiler system  : ifort
 Compiler command : /app/intel/impi/3.1/bin64/mpiifort
 Compiler flags   : -heap-arrays -g -ftrapuv -traceback -free

 Input Script  : /homenc/ipaes84/FinalROMS/ROMS/ROMS/roms-3.1/ROMS/ocean_benchmark4.in

 SVN Root URL  : https://www.myroms.org/svn/src/trunk
 SVN Revision  : 

 Local Root    : /homenc/ipaes84/FinalROMS/ROMS/ROMS/roms-3.1
 Header Dir    : /homenc/ipaes84/FinalROMS/ROMS/ROMS/roms-3.1/ROMS
 Header file   : benchmark.h
 Analytical Dir: /homenc/ipaes84/FinalROMS/ROMS/ROMS/roms-3.1/ROMS

 Resolution, Grid 01: 4192x0272x030,  Parallel Nodes:   8,  Tiling: 002x004


 Physical Parameters, Grid: 01
 =============================

      17532  ntimes          Number of timesteps for 3-D equations.
    150.000  dt              Timestep size (s) for 3-D equations.
         20  ndtfast         Number of timesteps for 2-D equations between
                               each 3D timestep.
          1  ERstr           Starting ensemble/perturbation run number.
          1  ERend           Ending ensemble/perturbation run number.
          0  nrrec           Number of restart records to read from disk.
          T  LcycleRST       Switch to recycle time-records in restart file.
      17532  nRST            Number of timesteps between the writing of data
                               into restart fields.
          1  ninfo           Number of timesteps between print of information
                               to standard output.
          T  ldefout         Switch to create a new output NetCDF file(s).
      17532  nHIS            Number of timesteps between the writing fields
                               into history file.
      17532  ndefHIS         Number of timesteps between creation of new
                               history files.
 5.0000E+02  tnu2(01)        Horizontal, harmonic mixing coefficient (m2/s)
                               for tracer 01: temp
 5.0000E+02  tnu2(02)        Horizontal, harmonic mixing coefficient (m2/s)
                               for tracer 02: salt
 5.0000E+03  visc2           Horizontal, harmonic mixing coefficient (m2/s)
                               for momentum.
 1.0000E-05  Akt_bak(01)     Background vertical mixing coefficient (m2/s)
                               for tracer 01: temp
 1.0000E-05  Akt_bak(02)     Background vertical mixing coefficient (m2/s)
                               for tracer 02: salt
 1.0000E-04  Akv_bak         Background vertical mixing coefficient (m2/s)
                               for momentum.
 3.0000E-04  rdrg            Linear bottom drag coefficient (m/s).
 3.0000E-03  rdrg2           Quadratic bottom drag coefficient.
 2.0000E-02  Zob             Bottom roughness (m).
 1.0000E+01  blk_ZQ          Height (m) of surface air humidity measurement.
 1.0000E+01  blk_ZT          Height (m) of surface air temperature measurement.
 1.0000E+01  blk_ZW          Height (m) of surface winds measurement.
          1  lmd_Jwt         Jerlov water type.
          1  Vtransform      S-coordinate transformation equation.
          1  Vstretching     S-coordinate stretching function.
 5.0000E+00  theta_s         S-coordinate surface control parameter.
 4.0000E-01  theta_b         S-coordinate bottom  control parameter.
    200.000  Tcline          S-coordinate surface/bottom layer width (m) used
                               in vertical coordinate stretching.
   1025.000  rho0            Mean density (kg/m3) for Boussinesq approximation.
      0.000  dstart          Time-stamp assigned to model initialization (days).
       0.00  time_ref        Reference time for units attribute (yyyymmdd.dd)
 0.0000E+00  Tnudg(01)       Nudging/relaxation time scale (days)
                               for tracer 01: temp
 0.0000E+00  Tnudg(02)       Nudging/relaxation time scale (days)
                               for tracer 02: salt
 0.0000E+00  Znudg           Nudging/relaxation time scale (days)
                               for free-surface.
 0.0000E+00  M2nudg          Nudging/relaxation time scale (days)
                               for 2D momentum.
 0.0000E+00  M3nudg          Nudging/relaxation time scale (days)
                               for 3D momentum.
 0.0000E+00  obcfac          Factor between passive and active
                               open boundary conditions.
     10.000  T0              Background potential temperature (C) constant.
     35.000  S0              Background salinity (PSU) constant.
      1.000  gamma2          Slipperiness variable: free-slip (1.0) or 
                                                    no-slip (-1.0).
          T  Hout(idFsur)    Write out free-surface.
          T  Hout(idUbar)    Write out 2D U-momentum component.
          T  Hout(idVbar)    Write out 2D V-momentum component.
          T  Hout(idUvel)    Write out 3D U-momentum component.
          T  Hout(idVvel)    Write out 3D V-momentum component.
          T  Hout(idWvel)    Write out W-momentum component.
          T  Hout(idOvel)    Write out omega vertical velocity.
          T  Hout(idTvar)    Write out tracer 01: temp
          T  Hout(idTvar)    Write out tracer 02: salt
          T  Hout(idTsur)    Write out surface net heat flux.
          T  Hout(idTsur)    Write out surface net salt flux.
          T  Hout(idSrad)    Write out shortwave radiation flux.
          T  Hout(idLrad)    Write out longwave radiation flux.
          T  Hout(idLhea)    Write out latent heat flux.
          T  Hout(idShea)    Write out sensible heat flux.
          T  Hout(idDano)    Write out density anomaly.
          T  Hout(idHsbl)    Write out depth of surface boundary layer.

 Output/Input Files:

             Output Restart File:  ./output/ocean_rst.nc
        Prefix for History Files:  ./output/ocean_his

 Tile partition information for Grid 01:  4192x0272x0030  tiling: 002x004

     tile     Istr     Iend     Jstr     Jend     Npts

        0        1     2096        1       68  4275840
        1     2097     4192        1       68  4275840
        2        1     2096       69      136  4275840
        3     2097     4192       69      136  4275840
        4        1     2096      137      204  4275840
        5     2097     4192      137      204  4275840
        6        1     2096      205      272  4275840
        7     2097     4192      205      272  4275840

 Tile minimum and maximum fractional grid coordinates:
   (interior points only)

     tile     Xmin     Xmax     Ymin     Ymax     grid

        0     0.50  2096.50     0.50    68.50  RHO-points
        1  2096.50  4192.50     0.50    68.50  RHO-points
        2     0.50  2096.50    68.50   136.50  RHO-points
        3  2096.50  4192.50    68.50   136.50  RHO-points
        4     0.50  2096.50   136.50   204.50  RHO-points
        5  2096.50  4192.50   136.50   204.50  RHO-points
        6     0.50  2096.50   204.50   272.50  RHO-points
        7  2096.50  4192.50   204.50   272.50  RHO-points

        0     1.00  2096.50     0.50    68.50    U-points
        1  2096.50  4192.00     0.50    68.50    U-points
        2     1.00  2096.50    68.50   136.50    U-points
        3  2096.50  4192.00    68.50   136.50    U-points
        4     1.00  2096.50   136.50   204.50    U-points
        5  2096.50  4192.00   136.50   204.50    U-points
        6     1.00  2096.50   204.50   272.50    U-points
        7  2096.50  4192.00   204.50   272.50    U-points

        0     0.50  2096.50     1.00    68.50    V-points
        1  2096.50  4192.50     1.00    68.50    V-points
        2     0.50  2096.50    68.50   136.50    V-points
        3  2096.50  4192.50    68.50   136.50    V-points
        4     0.50  2096.50   136.50   204.50    V-points
        5  2096.50  4192.50   136.50   204.50    V-points
        6     0.50  2096.50   204.50   272.00    V-points
        7  2096.50  4192.50   204.50   272.00    V-points

 Maximum halo size in XI and ETA directions:

               HaloSizeI(1) =    6327
               HaloSizeJ(1) =     240
                TileSide(1) =    2103
                TileSize(1) =  155622


 Activated C-preprocessing Options:

  BENCHMARK          Benchmark Test, Idealized Southern Ocean, Large Grid
  ALBEDO             Shortwave radiation from albedo equation.
  ANA_BSFLUX         Analytical kinematic bottom salinity flux.
  ANA_BTFLUX         Analytical kinematic bottom temperature flux.
  ANA_CLOUD          Analytical cloud fraction.
  ANA_GRID           Analytical grid set-up.
  ANA_HUMIDITY       Analytical surface air humidity.
  ANA_INITIAL        Analytical initial conditions.
  ANA_PAIR           Analytical surface air pressure.
  ANA_RAIN           Analytical rain fall rate.
  ANA_SRFLUX         Analytical kinematic shortwave radiation flux.
  ANA_SSFLUX         Analytical kinematic surface salinity flux.
  ANA_WINDS          Analytical surface wind components.
  ASSUMED_SHAPE      Using assumed-shape arrays.
  BULK_FLUXES        Surface bulk fluxes parametererization.
  CURVGRID           Orthogonal curvilinear grid.
  DJ_GRADPS          Parabolic Splines density Jacobian (Shchepetkin, 2002).
  DOUBLE_PRECISION   Double precision arithmetic.
  EW_PERIODIC        East-West periodic boundaries.
  LMD_CONVEC         LMD convective mixing due to shear instability.
  LMD_MIXING         Large/McWilliams/Doney interior mixing.
  LMD_NONLOCAL       LMD convective nonlocal transport.
  LMD_RIMIX          LMD diffusivity due to shear instability.
  LMD_SKPP           KPP surface boundary layer mixing.
  LONGWAVE           Compute net longwave radiation internally.
  MIX_GEO_TS         Mixing of tracers along geopotential surfaces.
  MIX_S_UV           Mixing of momentum along constant S-surfaces.
  MPI                MPI distributed-memory configuration.
  NONLINEAR          Nonlinear Model.
  NONLIN_EOS         Nonlinear Equation of State for seawater.
  NORTHERN_WALL      Wall boundary at Northern edge.
  POWER_LAW          Power-law shape time-averaging barotropic filter.
  PROFILE            Time profiling activated .
  !RST_SINGLE        Double precision fields in restart NetCDF file.
  SALINITY           Using salinity.
  SOLAR_SOURCE       Solar Radiation Source Term.
  SOLVE3D            Solving 3D Primitive Equations.
  SOUTHERN_WALL      Wall boundary at Southern edge.
  SPLINES            Conservative parabolic spline reconstruction.
  SPHERICAL          Spherical grid configuration.
  TS_U3HADVECTION    Third-order upstream horizontal advection of tracers.
  TS_C4VADVECTION    Fourth-order centered vertical advection of tracers.
  TS_DIF2            Harmonic mixing of tracers.
  UV_ADV             Advection of momentum.
  UV_COR             Coriolis term.
  UV_U3HADVECTION    Third-order upstream horizontal advection of 3D momentum.
  UV_C4VADVECTION    Fourth-order centered vertical advection of momentum.
  UV_QDRAG           Quadratic bottom stress.
  UV_VIS2            Harmonic mixing of momentum.
  VAR_RHO_2D         Variable density barotropic mode.

 INITIAL: Configuring and initializing forward nonlinear model ...


 Vertical S-coordinate System: 

 level   S-coord     Cs-curve          at_hmin  over_slope     at_hmax

    30   0.0000000   0.0000000           0.000       0.000       0.000
    29  -0.0333333  -0.0024174          -7.392     -11.622     -15.853
    28  -0.0666667  -0.0052838         -14.918     -24.165     -33.412
    27  -0.1000000  -0.0087922         -22.638     -38.024     -53.410
    26  -0.1333333  -0.0131904         -30.624     -53.707     -76.790
    25  -0.1666667  -0.0187972         -38.973     -71.868    -104.763
    24  -0.2000000  -0.0260168         -47.805     -93.334    -138.864
    23  -0.2333333  -0.0353508         -57.272    -119.136    -181.000
    22  -0.2666667  -0.0473981         -67.553    -150.499    -233.446
    21  -0.3000000  -0.0628318         -78.850    -188.805    -298.761
    20  -0.3333333  -0.0823381         -91.368    -235.460    -379.551
    19  -0.3666667  -0.1065031        -105.284    -291.665    -478.045
    18  -0.4000000  -0.1356491        -120.695    -358.081    -595.466
    17  -0.4333333  -0.1696534        -137.563    -434.456    -731.349
    16  -0.4666667  -0.2078237        -155.680    -519.372    -883.063
    15  -0.5000000  -0.2489214        -174.676    -610.289   -1045.901
    14  -0.5333333  -0.2913811        -194.081    -703.998   -1213.915
    13  -0.5666667  -0.3336756        -213.436    -797.368   -1381.301
    12  -0.6000000  -0.3746809        -232.404    -888.096   -1543.788
    11  -0.6333333  -0.4138998        -250.837    -975.161   -1699.486
    10  -0.6666667  -0.4514899        -268.780   -1058.888   -1848.995
     9  -0.7000000  -0.4881475        -286.444   -1140.702   -1994.960
     8  -0.7333333  -0.5249360        -304.147   -1222.785   -2141.423
     7  -0.7666667  -0.5631362        -322.274   -1307.762   -2293.251
     6  -0.8000000  -0.6041494        -341.245   -1398.506   -2455.768
     5  -0.8333333  -0.6494565        -361.504   -1498.053   -2634.601
     4  -0.8666667  -0.7006196        -383.519   -1609.604   -2835.688
     3  -0.9000000  -0.7593114        -407.793   -1736.588   -3065.383
     2  -0.9333333  -0.8273620        -434.875   -1882.759   -3330.642
     1  -0.9666667  -0.9068164        -465.378   -2052.307   -3639.236
     0  -1.0000000  -1.0000000        -500.000   -2250.000   -4000.000

 Time Splitting Weights: ndtfast =  20    nfast =  29

    Primary            Secondary            Accumulated to Current Step

  1-0.0009651193358779 0.0500000000000000-0.0009651193358779 0.0500000000000000
  2-0.0013488780126037 0.0500482559667939-0.0023139973484816 0.1000482559667939
  3-0.0011514592651645 0.0501156998674241-0.0034654566136460 0.1501639558342179
  4-0.0003735756740661 0.0501732728306823-0.0038390322877122 0.2003372286649002
  5 0.0009829200513762 0.0501919516143856-0.0028561122363360 0.2505291802792858
  6 0.0029141799764308 0.0501428056118168 0.0000580677400948 0.3006719858911026
  7 0.0054132615310267 0.0499970966129952 0.0054713292711215 0.3506690825040978
  8 0.0084687837865132 0.0497264335364439 0.0139401130576347 0.4003955160405417
  9 0.0120633394191050 0.0493029943471183 0.0260034524767397 0.4496985103876600
 10 0.0161716623600090 0.0486998273761630 0.0421751148367486 0.4983983377638230
 11 0.0207585511322367 0.0478912442581626 0.0629336659689853 0.5462895820219856
 12 0.0257765478740990 0.0468533167015507 0.0887102138430843 0.5931428987235363
 13 0.0311633730493853 0.0455644893078458 0.1198735868924696 0.6387073880313821
 14 0.0368391158442262 0.0440063206553765 0.1567127027366958 0.6827137086867585
 15 0.0427031802506397 0.0421643648631652 0.1994158829873354 0.7248780735499237
 16 0.0486309868367616 0.0400292058506332 0.2480468698240970 0.7649072794005569
 17 0.0544704302037591 0.0375976565087951 0.3025173000278562 0.8025049359093520
 18 0.0600380921294285 0.0348741349986072 0.3625553921572847 0.8373790709079592
 19 0.0651152103984763 0.0318722303921358 0.4276706025557610 0.8692513013000949
 20 0.0694434033194839 0.0286164698722119 0.4971140058752449 0.8978677711723068
 21 0.0727201499285569 0.0251442997062377 0.5698341558038018 0.9230120708785445
 22 0.0745940258796570 0.0215082922098099 0.6444281816834588 0.9445203630883544
 23 0.0746596950216180 0.0177785909158270 0.7190878767050768 0.9622989540041814
 24 0.0724526566618460 0.0140456061647461 0.7915405333669228 0.9763445601689276
 25 0.0674437485167025 0.0104229733316538 0.8589842818836253 0.9867675335005814
 26 0.0590334053485720 0.0070507859058187 0.9180176872321973 0.9938183194064002
 27 0.0465456732896125 0.0040991156383901 0.9645633605218099 0.9979174350447904
 28 0.0292219798521905 0.0017718319739095 0.9937853403740005 0.9996892670186999
 29 0.0062146596259994 0.0003107329813000 0.9999999999999999 0.9999999999999999

 ndtfast, nfast =   20  29   nfast/ndtfast = 1.45000

 Centers of gravity and integrals (values must be 1, 1, approx 1/2, 1, 1):

    1.000000000000 1.060707743385 0.530353871693 1.000000000000 1.000000000000

 Power filter parameters, Fgamma, gamma =  0.28400   0.14200

 Minimum X-grid spacing, DXmin =  3.26041548E+00 km
 Maximum X-grid spacing, DXmax =  6.14309257E+00 km
 Minimum Y-grid spacing, DYmin =  8.17650180E+00 km
 Maximum Y-grid spacing, DYmax =  8.17650180E+00 km
 Minimum Z-grid spacing, DZmin =  7.39187981E+00 m
 Maximum Z-grid spacing, DZmax =  3.60764219E+02 m

 Minimum barotropic Courant Number =  1.62217863E-01
 Maximum barotropic Courant Number =  4.49102838E-01
 Maximum Coriolis   Courant Number =  2.05618380E-02


 Maximum grid stiffness ratios:  rx0 =   3.966042E-01 (Beckmann and Haidvogel)
                                 rx1 =   1.150589E+01 (Haney)


 Initial basin volumes: TotVolume =  1.65993987026582E+17 m3
                        MinVolume =  1.97754634567622E+08 m3
                        MaxVolume =  1.80931355725002E+10 m3
                          Max/Min =  9.14928523018421E+01

NL ROMS/TOMS: started time-stepping: (Grid: 01 TimeSteps: 00000001 - 00017532)

   STEP   Day HH:MM:SS  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME

      0     0 00:00:00  0.000000E+00  1.963001E+04  1.963001E+04  1.659940E+17
      DEF_HIS   - creating history file: ./output/ocean_his_0001.nc

 NETCDF_CREATE - unable to create output NetCDF file:
                 ./output/ocean_his_0001.nc
                 call from:  def_his.F

 DEF_HIS - unable to create history NetCDF file: ./output/ocean_his_0001.nc

 Elapsed CPU time (seconds):

 Node   #  1 CPU:      23.822
 Node   #  2 CPU:      24.275
 Node   #  6 CPU:      24.522
 Node   #  5 CPU:      24.664
 Node   #  3 CPU:      24.089
 Node   #  7 CPU:      24.488
 Node   #  0 CPU:      24.254
 Node   #  4 CPU:      24.625
 Total:               194.741

 Nonlinear model elapsed time profile:

  Initialization ...................................        46.832  (24.0482 %)
  Processing of input data .........................         0.426  ( 0.2187 %)
  Computation of vertical boundary conditions ......         0.241  ( 0.1237 %)
  Computation of global information integrals ......         4.905  ( 2.5189 %)
  2D/3D coupling, vertical metrics .................         7.077  ( 3.6340 %)
  Omega vertical velocity ..........................         5.294  ( 2.7186 %)
  Equation of state for seawater ...................        34.125  (17.5231 %)
  Atmosphere-Ocean bulk flux parameterization ......         3.658  ( 1.8786 %)
  KPP vertical mixing parameterization .............        50.665  (26.0167 %)
                                              Total:       153.224   78.6806

 Nonlinear model message Passage profile:

  Message Passage: 2D halo exchanges ...............         3.334  ( 1.7118 %)
  Message Passage: 3D halo exchanges ...............         5.504  ( 2.8264 %)
  Message Passage: 4D halo exchanges ...............         1.132  ( 0.5812 %)
  Message Passage: data broadcast ..................         0.892  ( 0.4580 %)
  Message Passage: data reduction ..................         0.195  ( 0.1001 %)
                                              Total:        11.056    5.6774

 All percentages are with respect to total time =          194.741

 ROMS/TOMS - Output NetCDF summary for Grid 01:

 Analytical header files used:

     ROMS/Functionals/ana_btflux.h
     ROMS/Functionals/ana_cloud.h
     ROMS/Functionals/ana_grid.h
     ROMS/Functionals/ana_humid.h
     ROMS/Functionals/ana_initial.h
     ROMS/Functionals/ana_pair.h
     ROMS/Functionals/ana_rain.h
     ROMS/Functionals/ana_srflux.h
     ROMS/Functionals/ana_stflux.h
     ROMS/Functionals/ana_tair.h
     ROMS/Functionals/ana_winds.h

 ROMS/TOMS - Output error ............ exit_flag:   3


 ERROR: Abnormal termination: NetCDF OUTPUT.
 REASON: No such file or directory

PS:

Read file <err_file> for stderr output of this job.

arango · #4 Unread post by **arango** » Tue Nov 02, 2010 4:20 am

prakrati wrote: NETCDF_CREATE - unable to create output NetCDF file: ./output/ocean_his_0001.nc
call from: def_his.F

DEF_HIS - unable to create history NetCDF file: ./output/ocean_his_0001.nc
...
ERROR: Abnormal termination: NetCDF OUTPUT.
REASON: No such file or directory

The error cannot be clearer

It is telling that the model is unable to create the file ./output/ocean_his_0001.nc. Either the directory ./output doesn't exist or you don't have privileges to write into that directory. If this is not the case, something is wrong with your NetCDF library. The NetCDF library has to be build with the same compiler used in ROMS.

prakrati · #5 Unread post by **prakrati** » Tue Nov 02, 2010 7:59 am

Please can you help and tell me where this output file should be? I created the output file in the same directory which contains my executable but it still gave the same error .

prakrati · #6 Unread post by **prakrati** » Tue Nov 02, 2010 11:15 am

Also the NetCDF library has been compiled with the same compilers as the ROMS
Please help

kate · #7 Unread post by **kate** » Tue Nov 02, 2010 4:18 pm

Did you build netcdf and run it through its own tests with "make check"?

I always get in trouble when I fill my disk quota. Do you have a system with quotas? Are you running interactively or through some batch system with possibly different permissions?

arango · #8 Unread post by **arango** » Tue Nov 02, 2010 6:13 pm

You can set NHIS and NRST in ocean_benchmark4.in to a value larger than 17532, so no output files are created. This will allow the model to continue running. Your grid is very large 4192x0272x030 and you may be having memory issues. Perhaps, you also need to increase the number of nodes to fit your model in cache. You are using a 2x4 partition. Your tile in the x-dimension is still large. Try any of the other input script like ocean_benchmark1.in to see if run with smaller grid.

Ocean Modeling Discussion

Abnormal Termination of ROMS run

Abnormal Termination of ROMS run

Re: Abnormal Termination of ROMS run

Re: Abnormal Termination of ROMS run

Re: Abnormal Termination of ROMS run

Re: Abnormal Termination of ROMS run

Re: Abnormal Termination of ROMS run

Re: Abnormal Termination of ROMS run

Re: Abnormal Termination of ROMS run