Bottleneck in ROMS application

[koushik]

I am using Cray PAT prfiling tool to find the bottlenecks in ROMS application.

I have used NetCDF4-HDF5 parallel with 1440 PEs for this experiment and writing 5 Quick files.

! Time-Stepping parameters ----
NTIMES == 1080
DT == 240.0d0
NDTFAST == 54

I have added these 2 flags in the header file to activate NetCDF parallel ----
#define HDF5
#define PARALLEL_IO

These are the settings in build file ---
setenv USE_MPI on # distributed-memory parallelism
setenv USE_MPIF90 on # compile with mpif90 script
#setenv which_MPI mpich # compile with MPICH library
setenv which_MPI mpich2 # compile with MPICH2 library
# setenv which_MPI openmpi # compile with OpenMPI library
#setenv USE_OpenMP on # shared-memory parallelism
setenv FORT ftnintel
#setenv FORT gfortran
#setenv FORT pgi
#setenv USE_DEBUG on # use Fortran debugging flags
setenv USE_LARGE on # activate 64-bit compilation
setenv USE_NETCDF4 on # compile with NetCDF-4 library
setenv USE_PARALLEL_IO on # Parallel I/O with NetCDF-4/HDF5
setenv USE_MY_LIBS on # use my library paths below

Doubts:
1. What is meant by Whole Program on top of main ?
2. Why H5FD_mpio_write and nc4_put_vara is called even before main function and they are consuming almost 66% and 24% of the total execution time(It is almost 90% of the execution time)?

Thanks,
Koushik