ROMS4 built with PIO, freeze on creating history file

[cjroach]

G'day,
I've just installed ROMS4.0 with the new parallel IO setup on the NZ NESI HPC system. Compiling ROMS didn't produce any errors and I added PIO options to roms.in, mostly following the IO page on the wiki with the exception of switching PIO_METHOD to either 1, 3 or 4 (tried this multiple times with different configurations).

Code: Select all

! Input and Output files processing library to use:
!
!   [1] Standard NetCDF-3 or NetCDF-4 library
!   [2] Serial or Parallel I/O with Parallel-IO (PIO) library (MPI only)

     INP_LIB =  2
     OUT_LIB =  2

! PIO library methods for reading/writing NetCDF files:
!
!   [0] parallel read and write of PnetCDF (CDF-5, not recommended)
!   [1] parallel read and write of NetCDF3 (64-biat offset)
!   [2] serial   read and write of NetCDF3 (64-bit offset)
!   [3] parallel read and serial write of NetCDF4/HDF5
!   [4] parallel read and write of NETCDF4/HDF5

  PIO_METHOD =  4

! PIO library MPI processes set-up:

 PIO_IOTASKS =  1                 ! number of I/O tasks to define
  PIO_STRIDE =  1                 ! stride in the MPI-ran between I/O tasks
    PIO_BASE =  0                 ! offset for the first I/O task
  PIO_AGGREG =  1                 ! number of MPI-aggregators to use

! PIO library rearranger methods for moving data between computational and I/O
! processes:
!
!   [1] Box rearrangement
!   [2] Subset rearrangement

   PIO_REARR =  1

! PIO library rearranger flag for MPI communications between computational
! and I/O processes:
!
!   [0] Point-to-Point (low-level communications)
!   [1] Collective (high-level grouped communications)

PIO_REARRCOM =  0

! PIO library rearranger flow control direction flag for MPI communications
! between computational and I/O processes:
!
!   [0] Enable computational to I/O processes, and vice versa
!   [2] Enable computational to I/O processes only
!   [3] Enable I/O to computational processes only
!   [4] Disable flow control

PIO_REARRDIR = 0

! PIO rearranger options for computational to I/O processes (C2I):

  PIO_C2I_HS = T                  ! Enable C2I handshake (T/F)
PIO_C2I_Send = F                  ! Enable C2I Isends (T/F)
PIO_C2I_Preq = 64                 ! Maximum pending C2I requests

! PIO rearranger options for I/O to computational processes (I2C):

  PIO_I2C_HS = F                  ! Enable I2C handshake (T/F)
PIO_I2C_Send = T                  ! Enable I2C Isends (T/F)
PIO_I2C_Preq = 65                 ! Maximum pending I2C requests

Anyway, for all PIO_METHOD options tested, ROMS runs through all the model start up stuff fine, but on the first timestep while creating the history file things just freeze up. No errors are produced and the job scheduler (in this case SLURM) shows the task as still running. The history file is created but apparently not fully written. Given for our experiment a completed history file for a 30 day run with output every 1hr is about 250GB, we expect output for the first timestep to be about 300MB, the history file as written before everything freezes up is about 65MB. The history file does appear to be a valid (ncdump -h [filename] produces sensible output)

I have no idea if there's something wrong at my end

For reference active modules are:

Code: Select all

  1) slurm                           7) APR-util/.1.6.1             (H)  13) iccifort/2020.0.166                  19) cURL/7.64.0-GCCcore-9.2.0         25) iimpi/2020a
  2) NeSI                      (S)   8) Subversion/1.9.7                 14) libpmi                               20) PCRE/8.43-GCCcore-9.2.0           26) PnetCDF/1.9.0-intel-2020a
  3) craype-broadwell                9) shared                           15) impi/2019.6.166-iccifort-2020.0.166  21) netCDF/4.7.3-iimpi-2020a          27) ParallelIO/2.5.4-intel-2020a
  4) craype-network-infiniband      10) GCCcore/9.2.0                    16) imkl/2020.0.166-iimpi-2020a          22) netCDF-Fortran/4.5.2-iimpi-2020a
  5) SQLite/3.25.3                  11) zlib/1.2.11-GCCcore-9.2.0        17) intel/2020a                          23) HDF5/1.10.5-iimpi-2020a
  6) APR/.1.6.5                (H)  12) binutils/2.32-GCCcore-9.2.0      18) Szip/2.1.1-GCCcore-9.2.0             24) arpack-ng/3.6.3-intel-2020a

I'm yet to try compiling ROMS4.0 with PIO disabled but I do know that ROMS3.9 works fine on this HPC system.

[arango]

Well, if you are going to use PIO, you need to follow the instructions provided in WikiROMS or in the trac ticket. Also, there is information available in the wiki for roms.in. Please read the documentation carefully. PIO is too complicated to ignore the instructions.

The freeze is what we observed when the ASYNCHRONOUS_PIO CPP option is activated. That's the reason that we don't recommend using this option at this time. However, you can still use the default synchronous I/O. On close inspection of your log file, I noticed that the C-preprocessing option PIO_LIB is not activated So, I really don't know what ROMS is doing. Probably, it is hanging out in a collective MPI communication for eternity. The instructions are very clear everywhere that you need to activate both MPI and PIO_LIB options in ROMS for this capability to work.

[cjroach]

Thanks for the pointers.

The missing PIO_LIB CPP option? I'll need to check the build scripts but I think I'd assume like the USE_NETCDF4 option it could be activated by adding export USE_PIO=on to my build script. Might need to try recompiling with PIO_LIB activated through the export my_CPP_options=... method instead.

I'm certain I haven't (intentionally) activated the ASYNCHRONOUS_PIO CPP option.

[arango]

You also need the USE_PIO option in the build script. Both libraries are needed at running time since you may activate both in roms.in.

[cjroach]

Let's go through this again:
I have USE_PIO=on enabled in my build script. This results in the issues described in my opening post, including the lack of PIO_LIB in the CPP flags listed in the log file.

I have also tried adding # define PIO_LIB to roms3d.h, just in case there's something wrong in the various build scripts, make files etc. that means USE_PIO is not actually activating the PIO_LIB CPP flag (tried poking round the build scripts, make file and the various contents of the Compilers folder... couldn't sufficiently decipher things to be certain USE_PIO was actually working). This results in a crash during compilation with an error message:

Code: Select all

get_idata.f90(73): error #6404: This name does not have a type, and must have an explicit type.   [ING]
     &                    TIDE(ing)%pioFile,                            &
-------------------------------^
compilation aborted for get_idata.f90 (code 1)
make: *** [/nesi/project/mocean03294/NZB_3/build/get_idata.o] Error 1

For reference I'm using SVN Revision 1069, downloaded on the 1st of June. Just in case there's been bug fixes not listed on the trac ticket page...

[robertson]

USE_PIO (and USE_SCORPIO) are used only by the build script to add PIO (or SCORPIO) libraries and includes to the ROMS compilation configuration. They do not automatically define the PIO_LIB CPP option. The same is true of USE_NETCDF4 which does not instruct ROMS to output NetCDF4/HDF5 files. You must also define the HDF5 CPP option for ROMS to output NetCDF4/HDF5 files.

To my knowledge, the only USE_* assignments that automatically set a CPP option are the mutually exclusive USE_MPI and USE_OpenMP.

[kate]

get_idata.f90(73): error #6404: This name does not have a type, and must have an explicit type. [ING]
& TIDE(ing)%pioFile, &
-------------------------------^
compilation aborted for get_idata.f90 (code 1)
make: *** [/nesi/project/mocean03294/NZB_3/build/get_idata.o] Error 1

Looks to me like you should change "ing" to "ng".

[arango]

Yes, it is a typo, I will load the update later today.

[cjroach]

Thanks everyone. I'll wait until tomorrow, pull down the latest update, add the relevant CPP flags and then try again.

[arango]

I already updated the repositories.

[cjroach]

Those fixes (new version downloaded from SVN, adding PIO_LIB and HDF5 to the CPP definitions in roms3d.h) get me beyond the previous error but now I've run into a string of errors to do with undefined references to various PIO functions.

Code: Select all

ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(lbc.o): in function `lbc_mod_mp_lbc_getatt_pio_':
lbc.f90:(.text+0x2738): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(lbc.o): in function `lbc_mod_mp_lbc_putatt_pio_':
lbc.f90:(.text+0x5c81): undefined reference to `pionfatt_mod_mp_put_att_vid_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(set_pio.o): in function `set_pio_mod_mp_finalize_pio_':
set_pio.f90:(.text+0xac): undefined reference to `piolib_mod_mp_freedecomp_ios_'
ld: set_pio.f90:(.text+0xfd): undefined reference to `piolib_mod_mp_freedecomp_ios_'
ld: set_pio.f90:(.text+0x14e): undefined reference to `piolib_mod_mp_freedecomp_ios_'
ld: set_pio.f90:(.text+0x19f): undefined reference to `piolib_mod_mp_freedecomp_ios_'
ld: set_pio.f90:(.text+0x1f0): undefined reference to `piolib_mod_mp_freedecomp_ios_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(set_pio.o):set_pio.f90:(.text+0x241): more undefined references to `piolib_mod_mp_freedecomp_ios_' follow
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(set_pio.o): in function `set_pio_mod_mp_finalize_pio_':
set_pio.f90:(.text+0xf1e): undefined reference to `piolib_mod_mp_finalize_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(set_pio.o): in function `set_pio_mod_mp_field_iodecomp_':
set_pio.f90:(.text+0x5132): undefined reference to `piolib_mod_mp_pio_initdecomp_dof_i8_'
ld: set_pio.f90:(.text+0x51cb): undefined reference to `piolib_mod_mp_pio_initdecomp_dof_i8_'
ld: set_pio.f90:(.text+0x5256): undefined reference to `piolib_mod_mp_pio_initdecomp_dof_i8_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(set_pio.o): in function `set_pio_mod_mp_initialize_pio_':
set_pio.f90:(.text+0x55fb): undefined reference to `piolib_mod_mp_setdebuglevel_'
ld: set_pio.f90:(.text+0x5693): undefined reference to `piolib_mod_mp_init_intracom_'
ld: set_pio.f90:(.text+0x5731): undefined reference to `piolib_mod_mp_pio_set_rearr_opts_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(def_var.o): in function `def_var_mod_mp_def_var_pio_':
def_var.f90:(.text+0x59d5): undefined reference to `pio_nf_mp_def_var_md_desc_'
ld: def_var.f90:(.text+0x5be6): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x5d94): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x5fd2): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x61ba): undefined reference to `pionfatt_mod_mp_put_att_desc_int_'
ld: def_var.f90:(.text+0x6232): undefined reference to `pionfatt_mod_mp_put_att_desc_real_'
ld: def_var.f90:(.text+0x6293): undefined reference to `pionfatt_mod_mp_put_att_desc_double_'
ld: def_var.f90:(.text+0x63aa): undefined reference to `pionfatt_mod_mp_put_att_desc_int_'
ld: def_var.f90:(.text+0x6426): undefined reference to `pionfatt_mod_mp_put_att_desc_real_'
ld: def_var.f90:(.text+0x6487): undefined reference to `pionfatt_mod_mp_put_att_desc_double_'
ld: def_var.f90:(.text+0x65b8): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x67c9): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x69d7): undefined reference to `pionfatt_mod_mp_put_att_1d_desc_int_'
ld: def_var.f90:(.text+0x6a3a): undefined reference to `pionfatt_mod_mp_put_att_1d_desc_real_'
ld: def_var.f90:(.text+0x6a94): undefined reference to `pionfatt_mod_mp_put_att_1d_desc_double_'
ld: def_var.f90:(.text+0x6ca1): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x6e53): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x7005): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x71b3): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x7365): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(def_var.o):def_var.f90:(.text+0x7519): more undefined references to `pionfatt_mod_mp_put_att_desc_text_' follow
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(def_var.o): in function `def_var_mod_mp_def_var_pio_':
def_var.f90:(.text+0x78b8): undefined reference to `pionfatt_mod_mp_put_att_desc_int_'
ld: def_var.f90:(.text+0x7930): undefined reference to `pionfatt_mod_mp_put_att_desc_real_'
ld: def_var.f90:(.text+0x7991): undefined reference to `pionfatt_mod_mp_put_att_desc_double_'
ld: def_var.f90:(.text+0x7aa4): undefined reference to `pionfatt_mod_mp_put_att_desc_int_'
ld: def_var.f90:(.text+0x7b1c): undefined reference to `pionfatt_mod_mp_put_att_desc_real_'
ld: def_var.f90:(.text+0x7b72): undefined reference to `pionfatt_mod_mp_put_att_desc_double_'
ld: def_var.f90:(.text+0x7c89): undefined reference to `pionfatt_mod_mp_put_att_desc_int_'
ld: def_var.f90:(.text+0x7d05): undefined reference to `pionfatt_mod_mp_put_att_desc_real_'
ld: def_var.f90:(.text+0x7d6a): undefined reference to `pionfatt_mod_mp_put_att_desc_double_'
ld: def_var.f90:(.text+0x7e4a): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x7ff8): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x82ed): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x8459): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x8686): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(def_var.o):def_var.f90:(.text+0x8852): more undefined references to `pionfatt_mod_mp_put_att_desc_text_' follow
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(def_var.o): in function `def_var_mod_mp_def_var_pio_':
def_var.f90:(.text+0x9524): undefined reference to `pionfatt_mod_mp_put_att_desc_double_'
ld: def_var.f90:(.text+0x954b): undefined reference to `pionfatt_mod_mp_put_att_desc_real_'
ld: def_var.f90:(.text+0x9a2e): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x9b94): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0x9d21): undefined reference to `pionfatt_mod_mp_put_att_1d_desc_int_'
ld: def_var.f90:(.text+0x9e78): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0xa006): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0xa194): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0xa322): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: def_var.f90:(.text+0xa486): undefined reference to `pionfatt_mod_mp_put_att_desc_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(def_var.o):def_var.f90:(.text+0xa5da): more undefined references to `pionfatt_mod_mp_put_att_desc_text_' follow
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(def_var.o): in function `def_var_mod_mp_def_var_pio_':
def_var.f90:(.text+0xb0be): undefined reference to `pio_nf_mp_def_var_0d_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(def_dim.o): in function `def_dim_mod_mp_def_dim_pio_':
def_dim.f90:(.text+0x2ff): undefined reference to `pio_nf_mp_def_dim_int_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(nf_fwrite3d.o): in function `nf_fwrite3d_mod_mp_pio_fwrite3d_':
nf_fwrite3d.f90:(.text+0x2859): undefined reference to `piolib_mod_mp_setframe_'
ld: nf_fwrite3d.f90:(.text+0x2957): undefined reference to `piodarray_mp_write_darray_3d_real_'
ld: nf_fwrite3d.f90:(.text+0x3e39): undefined reference to `piolib_mod_mp_setframe_'
ld: nf_fwrite3d.f90:(.text+0x3f29): undefined reference to `piodarray_mp_write_darray_3d_double_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(nf_fwrite2d.o): in function `nf_fwrite2d_mod_mp_pio_fwrite2d_':
nf_fwrite2d.f90:(.text+0x1c98): undefined reference to `piolib_mod_mp_setframe_'
ld: nf_fwrite2d.f90:(.text+0x1d58): undefined reference to `piodarray_mp_write_darray_2d_real_'
ld: nf_fwrite2d.f90:(.text+0x2cc1): undefined reference to `piolib_mod_mp_setframe_'
ld: nf_fwrite2d.f90:(.text+0x2d7b): undefined reference to `piodarray_mp_write_darray_2d_double_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(nf_fread2d.o): in function `nf_fread2d_mod_mp_pio_fread2d_':
nf_fread2d.f90:(.text+0x4bcc): undefined reference to `piolib_mod_mp_setframe_'
ld: nf_fread2d.f90:(.text+0x4ca6): undefined reference to `piodarray_mp_read_darray_2d_real_'
ld: nf_fread2d.f90:(.text+0x576b): undefined reference to `piodarray_mp_read_darray_2d_double_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(nf_fread3d.o): in function `nf_fread3d_mod_mp_pio_fread3d_':
nf_fread3d.f90:(.text+0x1c1a): undefined reference to `piolib_mod_mp_setframe_'
ld: nf_fread3d.f90:(.text+0x1d3f): undefined reference to `piodarray_mp_read_darray_3d_real_'
ld: nf_fread3d.f90:(.text+0x2eea): undefined reference to `piolib_mod_mp_setframe_'
ld: nf_fread3d.f90:(.text+0x2ffa): undefined reference to `piodarray_mp_read_darray_3d_double_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(def_info.o): in function `def_info_mod_mp_def_info_pio_':
def_info.f90:(.text+0xd4c7): undefined reference to `pionfatt_mod_mp_put_att_vid_text_'
ld: def_info.f90:(.text+0xdf40): undefined reference to `pionfatt_mod_mp_put_att_vid_text_'
ld: def_info.f90:(.text+0xe2d8): undefined reference to `pionfatt_mod_mp_put_att_vid_text_'
ld: def_info.f90:(.text+0xe66a): undefined reference to `pionfatt_mod_mp_put_att_vid_text_'
ld: def_info.f90:(.text+0xe815): undefined reference to `pionfatt_mod_mp_put_att_vid_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libUTIL.a(def_info.o):def_info.f90:(.text+0xe9b0): more undefined references to `pionfatt_mod_mp_put_att_vid_text_' follow
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_copy_att_':
mod_pio_netcdf.f90:(.text+0x4e): undefined reference to `pio_nf_mp_inquire_desc_'
ld: mod_pio_netcdf.f90:(.text+0xe9): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x12f): undefined reference to `pio_nf_mp_inquire_variable_desc_'
ld: mod_pio_netcdf.f90:(.text+0x1dd): undefined reference to `pio_nf_mp_inq_attname_desc_'
ld: mod_pio_netcdf.f90:(.text+0x223): undefined reference to `pio_nf_mp_inq_att_desc_'
ld: mod_pio_netcdf.f90:(.text+0x2cb): undefined reference to `pionfatt_mod_mp_get_att_desc_1d_int_'
ld: mod_pio_netcdf.f90:(.text+0x36c): undefined reference to `pionfatt_mod_mp_get_att_desc_1d_double_'
ld: mod_pio_netcdf.f90:(.text+0x3ba): undefined reference to `pionfatt_mod_mp_get_att_desc_text_'
ld: mod_pio_netcdf.f90:(.text+0x41e): undefined reference to `pionfatt_mod_mp_put_att_vid_text_'
ld: mod_pio_netcdf.f90:(.text+0x59b): undefined reference to `pio_nf_mp_inq_attname_vid_'
ld: mod_pio_netcdf.f90:(.text+0x74e): undefined reference to `pio_nf_mp_inq_att_vid_'
ld: mod_pio_netcdf.f90:(.text+0x7a3): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x922): undefined reference to `pionfatt_mod_mp_put_att_vid_text_'
ld: mod_pio_netcdf.f90:(.text+0xc1c): undefined reference to `pionfatt_mod_mp_put_att_1d_vid_int_'
ld: mod_pio_netcdf.f90:(.text+0xdc1): undefined reference to `pionfatt_mod_mp_put_att_1d_vid_double_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_find_var_':
mod_pio_netcdf.f90:(.text+0x1309): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_dim_':
mod_pio_netcdf.f90:(.text+0x24d3): undefined reference to `pio_nf_mp_inquire_desc_'
ld: mod_pio_netcdf.f90:(.text+0x2543): undefined reference to `pio_nf_mp_inquire_dimension_desc_'
ld: mod_pio_netcdf.f90:(.text+0x2a3e): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x2a74): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_open_':
mod_pio_netcdf.f90:(.text+0x2e30): undefined reference to `piolib_mod_mp_pio_openfile_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_fvar_0d_':
mod_pio_netcdf.f90:(.text+0x416b): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x4403): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x4439): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x45ad): undefined reference to `pionfget_mod_mp_get_vara_vdesc_1d_double_'
ld: mod_pio_netcdf.f90:(.text+0x472f): undefined reference to `pionfget_mod_mp_get_var_vdesc_0d_double_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_ivar_0d_':
mod_pio_netcdf.f90:(.text+0x486b): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x4ae5): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x4b1b): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x4c84): undefined reference to `pionfget_mod_mp_get_vara_vdesc_1d_int_'
ld: mod_pio_netcdf.f90:(.text+0x4dfa): undefined reference to `pionfget_mod_mp_get_var_vdesc_0d_int_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_inq_var_':
mod_pio_netcdf.f90:(.text+0x50ce): undefined reference to `pio_nf_mp_inquire_desc_'
ld: mod_pio_netcdf.f90:(.text+0x513d): undefined reference to `pio_nf_mp_inq_attname_vid_'
ld: mod_pio_netcdf.f90:(.text+0x519b): undefined reference to `pio_nf_mp_inq_att_vid_'
ld: mod_pio_netcdf.f90:(.text+0x51ef): undefined reference to `pio_nf_mp_inq_attname_vid_'
ld: mod_pio_netcdf.f90:(.text+0x53da): undefined reference to `pio_nf_mp_inquire_variable_vid_'
ld: mod_pio_netcdf.f90:(.text+0x5442): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x54b1): undefined reference to `pio_nf_mp_inq_attname_desc_'
ld: mod_pio_netcdf.f90:(.text+0x557b): undefined reference to `pio_nf_mp_inq_attname_desc_'
ld: mod_pio_netcdf.f90:(.text+0x5ee3): undefined reference to `pio_nf_mp_inquire_dimension_desc_'
ld: mod_pio_netcdf.f90:(.text+0x5f83): undefined reference to `pio_nf_mp_inq_attname_desc_'
ld: mod_pio_netcdf.f90:(.text+0x5fe3): undefined reference to `pio_nf_mp_inq_att_desc_'
ld: mod_pio_netcdf.f90:(.text+0x604d): undefined reference to `pio_nf_mp_inq_attname_desc_'
ld: mod_pio_netcdf.f90:(.text+0x6277): undefined reference to `pionfatt_mod_mp_get_att_desc_text_'
ld: mod_pio_netcdf.f90:(.text+0x6667): undefined reference to `pionfatt_mod_mp_get_att_desc_double_'
ld: mod_pio_netcdf.f90:(.text+0x689f): undefined reference to `pionfatt_mod_mp_get_att_desc_int_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_close_':
mod_pio_netcdf.f90:(.text+0x7260): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x7299): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_inq_varid_':
mod_pio_netcdf.f90:(.text+0x730e): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_satt_g_':
mod_pio_netcdf.f90:(.text+0x755d): undefined reference to `pio_nf_mp_inquire_desc_'
ld: mod_pio_netcdf.f90:(.text+0x75a5): undefined reference to `pio_nf_mp_inq_attname_vid_'
ld: mod_pio_netcdf.f90:(.text+0x76e5): undefined reference to `pio_nf_mp_inq_attname_vid_'
ld: mod_pio_netcdf.f90:(.text+0x78d7): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x7cbc): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x7cf2): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_fvar_1d_':
mod_pio_netcdf.f90:(.text+0x8252): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x913b): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x9171): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x92f8): undefined reference to `pionfget_mod_mp_get_vara_vdesc_1d_double_'
ld: mod_pio_netcdf.f90:(.text+0x947a): undefined reference to `pionfget_mod_mp_get_var_vdesc_1d_double_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_fatt_':
mod_pio_netcdf.f90:(.text+0x9796): undefined reference to `pio_nf_mp_inquire_desc_'
ld: mod_pio_netcdf.f90:(.text+0x97bb): undefined reference to `pio_nf_mp_inquire_variable_desc_'
ld: mod_pio_netcdf.f90:(.text+0x9803): undefined reference to `pio_nf_mp_inq_attname_desc_'
ld: mod_pio_netcdf.f90:(.text+0x994d): undefined reference to `pio_nf_mp_inq_attname_desc_'
ld: mod_pio_netcdf.f90:(.text+0x9b31): undefined reference to `pionfatt_mod_mp_get_att_desc_double_'
ld: mod_pio_netcdf.f90:(.text+0x9f1b): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x9f51): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_fvar_2d_':
mod_pio_netcdf.f90:(.text+0xa1c7): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0xb65b): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0xb691): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0xb85a): undefined reference to `pionfget_mod_mp_get_vara_vdesc_2d_double_'
ld: mod_pio_netcdf.f90:(.text+0xb9dc): undefined reference to `pionfget_mod_mp_get_var_vdesc_2d_double_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_fvar_3d_':
mod_pio_netcdf.f90:(.text+0xbb75): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0xd525): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0xd55b): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0xd6ca): undefined reference to `pionfget_mod_mp_get_vara_vdesc_3d_double_'
ld: mod_pio_netcdf.f90:(.text+0xd850): undefined reference to `pionfget_mod_mp_get_var_vdesc_3d_double_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_fvar_4d_':
mod_pio_netcdf.f90:(.text+0xda19): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0xf7da): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0xf810): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0xf96a): undefined reference to `pionfget_mod_mp_get_vara_vdesc_4d_double_'
ld: mod_pio_netcdf.f90:(.text+0xfad9): undefined reference to `pionfget_mod_mp_get_var_vdesc_4d_double_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_lvar_0d_':
mod_pio_netcdf.f90:(.text+0xfc4b): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0xfc9a): undefined reference to `pio_nf_mp_inquire_variable_desc_'
ld: mod_pio_netcdf.f90:(.text+0x1000d): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x10043): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x101c5): undefined reference to `pionfget_mod_mp_get_vara_vdesc_1d_int_'
ld: mod_pio_netcdf.f90:(.text+0x10257): undefined reference to `pionfget_mod_mp_get_vara_vdesc_1d_text_'
ld: mod_pio_netcdf.f90:(.text+0x102d3): undefined reference to `pionfget_mod_mp_get_var_vdesc_0d_int_'
ld: mod_pio_netcdf.f90:(.text+0x10427): undefined reference to `pionfget_mod_mp_get_var_vdesc_1d_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_lvar_1d_':
mod_pio_netcdf.f90:(.text+0x105b6): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x10605): undefined reference to `pio_nf_mp_inquire_variable_desc_'
ld: mod_pio_netcdf.f90:(.text+0x10988): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x109be): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x10b5a): undefined reference to `pionfget_mod_mp_get_vara_vdesc_1d_int_'
ld: mod_pio_netcdf.f90:(.text+0x10cf5): undefined reference to `pionfget_mod_mp_get_vara_vdesc_1d_text_'
ld: mod_pio_netcdf.f90:(.text+0x10e00): undefined reference to `pionfget_mod_mp_get_var_vdesc_1d_int_'
ld: mod_pio_netcdf.f90:(.text+0x10f67): undefined reference to `pionfget_mod_mp_get_var_vdesc_1d_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_ivar_1d_':
mod_pio_netcdf.f90:(.text+0x1109b): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x11312): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x11348): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x11478): undefined reference to `pionfget_mod_mp_get_vara_vdesc_1d_int_'
ld: mod_pio_netcdf.f90:(.text+0x115dc): undefined reference to `pionfget_mod_mp_get_var_vdesc_1d_int_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_ivar_2d_':
mod_pio_netcdf.f90:(.text+0x1170b): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x11982): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x119b8): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x11ae8): undefined reference to `pionfget_mod_mp_get_vara_vdesc_2d_int_'
ld: mod_pio_netcdf.f90:(.text+0x11c4c): undefined reference to `pionfget_mod_mp_get_var_vdesc_2d_int_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_svar_0d_':
mod_pio_netcdf.f90:(.text+0x11da7): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x12030): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x12066): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x121ee): undefined reference to `pionfget_mod_mp_get_vara_vdesc_1d_text_'
ld: mod_pio_netcdf.f90:(.text+0x1238f): undefined reference to `pionfget_mod_mp_get_var_vdesc_0d_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_svar_1d_':
mod_pio_netcdf.f90:(.text+0x124cb): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x12742): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x12778): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x128a8): undefined reference to `pionfget_mod_mp_get_vara_vdesc_1d_text_'
ld: mod_pio_netcdf.f90:(.text+0x12a0c): undefined reference to `pionfget_mod_mp_get_var_vdesc_1d_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_svar_2d_':
mod_pio_netcdf.f90:(.text+0x12b3b): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x12db2): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x12de8): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x12f18): undefined reference to `pionfget_mod_mp_get_vara_vdesc_2d_text_'
ld: mod_pio_netcdf.f90:(.text+0x1307c): undefined reference to `pionfget_mod_mp_get_var_vdesc_2d_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_svar_3d_':
mod_pio_netcdf.f90:(.text+0x131ab): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x13422): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x13458): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x13588): undefined reference to `pionfget_mod_mp_get_vara_vdesc_3d_text_'
ld: mod_pio_netcdf.f90:(.text+0x136ec): undefined reference to `pionfget_mod_mp_get_var_vdesc_3d_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_time_0d_':
mod_pio_netcdf.f90:(.text+0x13824): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x13e50): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x13e86): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x141e3): undefined reference to `pionfget_mod_mp_get_vara_vdesc_1d_double_'
ld: mod_pio_netcdf.f90:(.text+0x14345): undefined reference to `pionfget_mod_mp_get_var_vdesc_0d_double_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_satt_v_':
mod_pio_netcdf.f90:(.text+0x144e4): undefined reference to `pio_nf_mp_inquire_desc_'
ld: mod_pio_netcdf.f90:(.text+0x14509): undefined reference to `pio_nf_mp_inquire_variable_desc_'
ld: mod_pio_netcdf.f90:(.text+0x14551): undefined reference to `pio_nf_mp_inq_attname_desc_'
ld: mod_pio_netcdf.f90:(.text+0x14691): undefined reference to `pio_nf_mp_inq_attname_desc_'
ld: mod_pio_netcdf.f90:(.text+0x14872): undefined reference to `pionfatt_mod_mp_get_att_desc_text_'
ld: mod_pio_netcdf.f90:(.text+0x14c5c): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x14c92): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_get_time_1d_':
mod_pio_netcdf.f90:(.text+0x151e3): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: mod_pio_netcdf.f90:(.text+0x161f7): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x1622d): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x16f0a): undefined reference to `pionfget_mod_mp_get_vara_vdesc_1d_double_'
ld: mod_pio_netcdf.f90:(.text+0x1706b): undefined reference to `pionfget_mod_mp_get_var_vdesc_1d_double_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_fvar_0d_':
mod_pio_netcdf.f90:(.text+0x173bd): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x173f3): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x1742d): undefined reference to `pionfput_mod_mp_put_var_vdesc_0d_double_'
ld: mod_pio_netcdf.f90:(.text+0x17484): undefined reference to `pionfput_mod_mp_put_vara_vdesc_1d_double_'
ld: mod_pio_netcdf.f90:(.text+0x17625): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_fvar_1d_':
mod_pio_netcdf.f90:(.text+0x17afa): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x17b30): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x17b4d): undefined reference to `pionfput_mod_mp_put_vara_vdesc_1d_double_'
ld: mod_pio_netcdf.f90:(.text+0x17ce4): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_fvar_2d_':
mod_pio_netcdf.f90:(.text+0x181ba): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x181f0): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x1820d): undefined reference to `pionfput_mod_mp_put_vara_vdesc_2d_double_'
ld: mod_pio_netcdf.f90:(.text+0x183a4): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_fvar_3d_':
mod_pio_netcdf.f90:(.text+0x1887a): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x188b0): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x188cd): undefined reference to `pionfput_mod_mp_put_vara_vdesc_3d_double_'
ld: mod_pio_netcdf.f90:(.text+0x18a64): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_fvar_4d_':
mod_pio_netcdf.f90:(.text+0x18f3a): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x18f70): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x18f8d): undefined reference to `pionfput_mod_mp_put_vara_vdesc_4d_double_'
ld: mod_pio_netcdf.f90:(.text+0x19124): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_ivar_0d_':
mod_pio_netcdf.f90:(.text+0x195fd): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x19633): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x1966d): undefined reference to `pionfput_mod_mp_put_var_vdesc_0d_int_'
ld: mod_pio_netcdf.f90:(.text+0x196c3): undefined reference to `pionfput_mod_mp_put_vara_vdesc_1d_int_'
ld: mod_pio_netcdf.f90:(.text+0x19864): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_ivar_1d_':
mod_pio_netcdf.f90:(.text+0x19d3a): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x19d70): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x19d8d): undefined reference to `pionfput_mod_mp_put_vara_vdesc_1d_int_'
ld: mod_pio_netcdf.f90:(.text+0x19f24): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_ivar_2d_':
mod_pio_netcdf.f90:(.text+0x1a3fa): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x1a430): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x1a44d): undefined reference to `pionfput_mod_mp_put_vara_vdesc_2d_int_'
ld: mod_pio_netcdf.f90:(.text+0x1a5e4): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_lvar_0d_':
mod_pio_netcdf.f90:(.text+0x1aae2): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x1ab18): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x1ab56): undefined reference to `pionfput_mod_mp_put_var_vdesc_0d_int_'
ld: mod_pio_netcdf.f90:(.text+0x1aba9): undefined reference to `pionfput_mod_mp_put_vara_vdesc_1d_int_'
ld: mod_pio_netcdf.f90:(.text+0x1ad4a): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_lvar_1d_':
mod_pio_netcdf.f90:(.text+0x1b343): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x1b379): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x1b3d9): undefined reference to `pionfput_mod_mp_put_vara_vdesc_1d_int_'
ld: mod_pio_netcdf.f90:(.text+0x1b580): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_lvar_2d_':
mod_pio_netcdf.f90:(.text+0x1bc2b): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x1bc61): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x1bcea): undefined reference to `pionfput_mod_mp_put_vara_vdesc_2d_int_'
ld: mod_pio_netcdf.f90:(.text+0x1be98): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_svar_0d_':
mod_pio_netcdf.f90:(.text+0x1c1e9): undefined reference to `pionfput_mod_mp_put_vara_vdesc_1d_text_'
ld: mod_pio_netcdf.f90:(.text+0x1c480): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x1c4b6): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x1c5f1): undefined reference to `pionfput_mod_mp_put_var_vdesc_0d_text_'
ld: mod_pio_netcdf.f90:(.text+0x1c647): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_svar_1d_':
mod_pio_netcdf.f90:(.text+0x1cb1a): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x1cb50): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x1cb6d): undefined reference to `pionfput_mod_mp_put_vara_vdesc_1d_text_'
ld: mod_pio_netcdf.f90:(.text+0x1cd04): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_svar_2d_':
mod_pio_netcdf.f90:(.text+0x1d1da): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x1d210): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x1d22d): undefined reference to `pionfput_mod_mp_put_vara_vdesc_2d_text_'
ld: mod_pio_netcdf.f90:(.text+0x1d3c4): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_put_svar_3d_':
mod_pio_netcdf.f90:(.text+0x1d89a): undefined reference to `piolib_mod_mp_closefile_'
ld: mod_pio_netcdf.f90:(.text+0x1d8d0): undefined reference to `pionfatt_mod_mp_get_att_text_'
ld: mod_pio_netcdf.f90:(.text+0x1d8ed): undefined reference to `pionfput_mod_mp_put_vara_vdesc_3d_text_'
ld: mod_pio_netcdf.f90:(.text+0x1da84): undefined reference to `pio_nf_mp_inq_varid_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_create_':
mod_pio_netcdf.f90:(.text+0x1ddf8): undefined reference to `piolib_mod_mp_createfile_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_enddef_':
mod_pio_netcdf.f90:(.text+0x1e18c): undefined reference to `pio_nf_mp_enddef_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_redef_':
mod_pio_netcdf.f90:(.text+0x1e2ac): undefined reference to `pio_nf_mp_redef_desc_'
ld: /nesi/project/mocean03294/NZB_3/build/libMODS.a(mod_pio_netcdf.o): in function `mod_pio_netcdf_mp_pio_netcdf_sync_':
mod_pio_netcdf.f90:(.text+0x1e3b4): undefined reference to `piolib_mod_mp_syncfile_'
make: *** [/nesi/project/mocean03294/NZB_3/romsM] Error 1

I have defined the HDF5, PIO and PNetcdf lib and include file paths (see below), so I don't know why it's unable to find those PIO functions.

Code: Select all

#!/usr/bin/env bash

# ROMS Building configuration file
if [[ -z "${ROMS_SRC}" ]]; then
  export ROMS_SRC=/nesi/project/mocean03294/roms_4_src_latest
fi
#export ROMS_CONFIGDIR=${PWD}
#export MY_CPP_FLAGS="${MY_CPP_FLAGS} -DFLOATS"
export MY_CPP_FLAGS=
export ROMS_APPLICATION=ROMS3D
#export GIT_BRANCH=roms3.4
export USE_OpenMP=
export USE_MPI=on
export USE_MPIF90=on
export USE_LARGE=on
export USE_DEBUG=
export USE_PIO=on
export USE_NETCDF4=on
#export USE_PARALLEL_IO=on
export ARPACK_LIBDIR=${EBROOTARPACKMINNG}
#export PARPACK_LIBDIR=/share/apps/romstools/lib
export SCRATCH_DIR=${ROMS_CONFIGDIR}/build
export BINDIR=${ROMS_CONFIGDIR}
export MY_HEADER_DIR=${ROMS_CONFIGDIR}
export MY_ANALYTICAL_DIR=${ROMS_CONFIGDIR}/include
export USE_ARPACK=on
#export UPDATE_VARINFO=
export NC_CONFIG=/opt/nesi/mahuika/netCDF/4.7.3-iimpi-2020a/bin/nc-config
#export NC_CONFIG=/opt/nesi/mahuika/netCDF/4.4.1-intel-2017a/bin/nc-config
#export NETCDF_INCDIR=${EBROOTNETCDFMINFORTRAN}/include
#export NETCDF_LIBDIR=${EBROOTNETCDFMINFORTRAN}/lib64
export NETCDF_INCDIR=$EBROOTNETCDFMINFORTRAN/include
export NETCDF_LIBDIR=$EBROOTNETCDFMINFORTRAN/lib
export PIO_INCDIR=/opt/nesi/CS400_centos7_bdw/ParallelIO/2.5.4-intel-2020a/include
export PIO_LIBDIR=/opt/nesi/CS400_centos7_bdw/ParallelIO/2.5.4-intel-2020a/lib
export PNETCDF_LIBDIR=/opt/nesi/CS400_centos7_bdw/PnetCDF/1.9.0-intel-2020a/include
export PNETCDF_INCDIR=/opt/nesi/CS400_centos7_bdw/PnetCDF/1.9.0-intel-2020a/lib
#export HDF5_LIBDIR=/cm/shared/apps/hdf5/1.10.0/lib
export HDF5_LIBDIR=/cm/shared/apps/hdf5/current/lib
export FC=ifort

[robertson]

We need to see the link line that produces that error so we can see if you have the all the includes and libraries that you need listed.

[arango]

Check the debugging section of the build_roms wiki page to learn how to inquire things about compiling variables and macros. There is lots of useful information in the wiki about the makefile, build scripts, libraries, and how to compile and link ROMS,

[cjroach]

Output of make script, excluding errors (see previous) and warnings.

A quick search suggests that the build script is linking to the PIO_INCDIR but not to the PIO_LIBDIR.

[arango]

As mention previously, the information in the Wiki tells you how to inquire the build script about the libraries that your are using to compile ROMS without actually compiling:

Code: Select all

> build_roms.sh -p LIBS

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
Using customized library paths from:  /Users/arango/Compilers/ROMS/my_build_paths.sh
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

LIBS = /Users/arango/ocean/repository/Projects/WC13/RBL4DVAR_split/Unsplit/Build_roms/libNLM.a -L/opt/intelsoft/openmpi/pio/lib -lpiof -lpioc -L/opt/intelsoft/openmpi/pnetcdf/lib -lpnetcdf -L/opt/intelsoft/serial/netcdf-4.7.4/lib -lnetcdff -L/opt/intelsoft/serial/hdf5-1.10.6/lib -L/opt/intelsoft/serial/netcdf-4.7.4/lib -lnetcdf -lm -lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl -L/opt/intelsoft/openmpi/PARPACK -lparpack -L/opt/intelsoft/serial/ARPACK -larpack

That's all! Then, you can check if the PIO library is there.

[robertson]

I see some rather amazing technical issues with that build. First it appears that you somehow have a mix of pgf90 (PGI), ifort (Intel), and gfortran (GNU). The compiling flags (-fastsse -Mipa=fast -Kieee) in your build are for the PGI compilers, suggesting that you have either not set FORT or set it to pgi. Then you have the path for Intel MPI but since you haven't set which_MPI to intel ROMS is compiling with mpif90 which is actually linked to gfortran unless your sysadmin has intentionally linked mpif90 to Intel. With Intel MPI, the compiler is actually called mpiifort (notice the double "i") and not mpif90. Are you using the build script or trying to configure this all on your own?

Then there is the fact that PIO (and pnetcdf) are not in your list of libraries at link time which is probably an artifact of the combination of compilers.

[cjroach]

As mention previously, the information in the Wiki tells you how to inquire the build script about the libraries that your are using to compile ROMS without actually compiling:
***
That's all! Then, you can check if the PIO library is there.

Sorry, but I'm working with a configuration set up by my supervisor, and he's additional build scripts wrapped around the ROMS default build script, thus it's a bit of a pain in the arse to . I've recompiled following the suggestion below and runlog files now indicate that it's loading PIO and similar properly.

I see some rather amazing technical issues with that build. First it appears that you somehow have a mix of pgf90 (PGI), ifort (Intel), and gfortran (GNU). The compiling flags (-fastsse -Mipa=fast -Kieee) in your build are for the PGI compilers, suggesting that you have either not set FORT or set it to pgi. Then you have the path for Intel MPI but since you haven't set which_MPI to intel ROMS is compiling with mpif90 which is actually linked to gfortran unless your sysadmin has intentionally linked mpif90 to Intel. With Intel MPI, the compiler is actually called mpiifort (notice the double "i") and not mpif90. Are you using the build script or trying to configure this all on your own?

Then there is the fact that PIO (and pnetcdf) are not in your list of libraries at link time which is probably an artifact of the combination of compilers.

Thanks, I've double checked: mpif90 is linked to miifort on the HPC I'm using; FORT was undefined, but FC was defined with intel... So, ROMS setup was defaulting to setting things up for PGI then actually doing the compilation with IFORT. Somehow wasn't an issue when setting up 3.9 (beyond strings of warnings). Fixed by simply adding FORT=intel.

***
This seems to have fixed the compile issue, now I'll get on and see if I can get everything running (initial tests with PIO_METHOD=3 or 4 result in out of memory crashes... Will see if a simple change to the SLURM script will fix that...).

Edit: Requesting more memory and splitting IO load over more workers kinda worked... Now getting density blow up on the first timestep. Are there any known issues about using older forms of netcdf file as input with NETCDF4/HDF5 IO enabled?

[robertson]

FORT was undefined, but FC was defined with intel... So, ROMS setup was defaulting to setting things up for PGI then actually doing the compilation with IFORT. Somehow wasn't an issue when setting up 3.9 (beyond strings of warnings). Fixed by simply adding FORT=intel.

ROMS uses a combination of operating system name (obtained with the uname -s command) and the FORT environment variable to determine which file to load from the Compilers directory. I am not sure what you mean by "wrapped around the ROMS default build script". Unless the ROMS build script you are wrapped around has had most of the lines commented out, many of the things you are setting are being overridden by the ROMS build script.

I took another look at your roms_config.sh and I'm guessing it's pretty old because NC_CONFIG was changed to NF_CONFIG over 4 years ago. It still works because ROMS defaults to the nf-config in your path when NF_CONFIG is not set.

Edit: Requesting more memory and splitting IO load over more workers kinda worked... Now getting density blow up on the first timestep. Are there any known issues about using older forms of netcdf file as input with NETCDF4/HDF5 IO enabled?

I don't recall any issues reading older format NetCDF files during testing. However, if you want to confirm that the problem is with PIO reading an older format NetCDF file you can change INP_LIB to 1 while leaving OUT_LIB set to 2 in your roms.in. That will tell ROMS to use the NetCDF library for reading files and PIO for writing. The other option is to convert the input NetCDF files to the latest format.