Possible Issue with Non-Periodic ana_pair Forcing in Parallel Runs

[hyc006]

Hi ROMS team,

I’d like to report a potential issue I’m encountering when applying non-periodic atmospheric pressure forcing using ana_pair in an idealized model setup.
I'm driving an idealized ROMS simulation using a pressure gradient imposed via the Pair field in ana_pair.h.
I intentionally want all other state variables to remain periodic, but for Pair only, I want the field to be non-periodic, to simulate a steady pressure gradient forcing.
The pressure varies with latitude and is meant to be non-periodic, so I explicitly disabled all periodic boundary exchange for this variable.

What Works :
On my old server (OpenMPI + gfortran), everything works as expected—both in serial and parallel.
On my new server, the model runs correctly in serial (Intel ifort + MPICH).

The Problem :
When I run the same model in parallel on the new server, the air pressure field (Pair) appears to behave as if it is still periodic, even though:
1. I commented out the default periodic exchange in ana_pair.h:

Code: Select all

! IF (EWperiodic(ng).or.NSperiodic(ng)) THEN
!   CALL exchange_r2d_tile(...)
! END IF

2 . I created a custom subroutine exchange_pair_tile in exchange_2d.F where I manually forced periodic flags to .FALSE.:

Code: Select all

IF (EWperiodic(ng)) THEN
  EW_exchange = .FALSE.
ELSE
  EW_exchange = .FALSE.
END IF

3 . I called exchange_pair_tile instead of exchange_r2d_tile in ana_pair.h.
I added the following call at the end of ana_pair.h:

Code: Select all

#ifdef DISTRIBUTE
  CALL mp_exchange2d(ng, tile, model, 1, LBi, UBi, LBj, UBj, NghostPoints, .FALSE., .FALSE., Pair)
#endif

Despite all of this, the pressure field still appears to wrap around at the domain edges in parallel runs—indicating some form of unintended periodic behavior.

What I’ve Tried :
Switching from ifort to gfortran on the new server — did not fix the issue.
Comparing behavior in serial vs. parallel — serial runs behave as expected.

Could this be an issue in how ROMS handles exchange routines internally in distributed parallel runs? Or is there another communication step I might be missing, even after disabling the periodicity at multiple levels?

I’ve attached both my modified ana_pair.h and exchange_2d.F files for reference. I would really appreciate any insight into whether this is an implementation issue on my side or potentially a bug related to parallel exchange behavior in custom variables like Pair.

ana_pair.h

(5.15 KiB) Downloaded 6 times

exchange_2d.F

(29.85 KiB) Downloaded 5 times

Thanks very much for your time and help!

Best,
Hsin-Yi Chen

[arango]

Wow, you don't understand how ROMS boundary conditions and parallelism work. Then, you start modifying the code wrongly without any knowledge. Your issues will be solved quickly if you pay attention to ROMS's standard input file section, where the lateral boundary conditions are specified. I assume you started modifying your application roms_*.in from an existing ROMS application that uses periodic boundary conditions. This input script is well-documented, and you need to consider all the parameters you need to configure your application!

Code: Select all

! Set lateral boundary conditions keyword. Notice that a value is expected
! for each boundary segment per nested grid for each state variable.
!
! Each tracer variable requires [1:4,1:NAT+NPT,Ngrids] values. Otherwise,
! [1:4,1:Ngrids] values are expected for other variables. The boundary
! order is: 1=west, 2=south, 3=east, and 4=north. That is, anticlockwise
! starting at the western boundary.
!
! The keyword is case insensitive and usually has three characters. However,
! it is possible to have compound keywords, if applicable. For example, the
! keyword "RadNud" implies radiation boundary condition with nudging. This
! combination is usually used in active/passive radiation conditions.
!
!   Keyword    Lateral Boundary Condition Type
!
!   Cha        Chapman_implicit (free-surface)
!   Che        Chapman_explicit (free-surface)
!   Cla        Clamped
!   Clo        Closed
!   Fla        Flather (2D momentum)                  _____N_____     j=Mm
!   Gra        Gradient                              |     4     |
!   Nes        Nested (refinement)                   |           |
!   Nud        Nudging                             1 W           E 3
!   Per        Periodic                              |           |
!   Rad        Radiation                             |_____S_____|
!   Red        Reduced Physics (2D momentum)               2          j=1
!   Shc        Shchepetkin (2D momentum)            i=1         i=Lm
!
!                   W       S       E       N
!                   e       o       a       o
!                   s       u       s       r
!                   t       t       t       t
!                           h               h
!
!                   1       2       3       4

You are getting periodic behavior because either the Western and Eastern boundaries, the Southern and Northern boundaries, or both have been set to periodic with the keyword Per. If one needs periodic boundary conditions, we must set it for all state variables! Check ROMS/External/roms_upwelling.in.

Please check your configuration and the standard ROMS printing output file. You should do this before modifying the code or posting panic messages in this Forum.

[hyc006]

Thank you for your reply and the detailed explanation about boundary conditions. I completely understand where you're coming from, and I appreciate your concern.

Just to clarify: I do understand how ROMS handles periodic boundaries, and in fact, I do want periodic boundary conditions for all the other state variables—this is intentional and consistent with the idealized setup I’m using (which is derived from a periodic base case, similar to roms_upwelling.in).

However, in my current experiment, I’m trying to apply an external latitudinal pressure gradient using the Pair field, and for this specific variable only, I do not want it to be periodic. That’s why I modified ana_pair.h and created a custom exchange_pair_tile subroutine in exchange_2d.F, where I explicitly set the periodic flags to .FALSE. just for Pair.

What’s puzzling is that this approach has worked before—on my old server running ROMS with OpenMPI and gfortran, I was able to run the model in parallel and get the correct non-periodic behavior for Pair, while keeping periodicity for other variables. The issue only shows up on my new server (MPICH + ifort), where it seems like the periodic exchange still happens for Pair even after all the changes I made.

So my question now is whether this could be related to a difference between MPICH and OpenMPI, or whether something about the ROMS internal communication for auxiliary fields like Pair behaves differently under certain configurations. I’m open to suggestions on how best to isolate or resolve this.

Thanks again for your time,
Hsin-Yi Chen

[hyc006]

Hi ROMS team,

I wanted to provide an update on my issue regarding the Pair (air pressure) field not being treated as periodic in the latest ROMS version (4.2), even when other variables are correctly handled.

To investigate further, I tried downloading and running ROMS version 4.1 with the same configuration (Intel ifort + MPICH, running in parallel). I made the following modifications:

ana_pair.h

(4.89 KiB) Downloaded 1 time

In ana_pair.h:
I removed the call to exchange_r2d_tile:

Code: Select all

!      IF (EWperiodic(ng).or.NSperiodic(ng)) THEN
!        CALL exchange_r2d_tile (ng, tile,                               &
!     &                          LBi, UBi, LBj, UBj,                     &
!     &                          Pair)
!      END IF

and I disabled the periodic flags for mp_exchange2d by doing

Code: Select all

#ifdef DISTRIBUTE
      CALL mp_exchange2d (ng, tile, model, 1,                           &
     &                    LBi, UBi, LBj, UBj,                           &
     &                    NghostPoints,                                 &
     &                    .FALSE.,.FALSE.,                              &
     &                    Pair)
#endif

With this change, everything behaves as expected in ROMS 4.1: all variables are treated as periodic except for Pair, which remains non-periodic, as desired. (I've attached the figure showing the correct behavior.)

However, when I copied the exact same changes into ROMS version 4.2, the model does not treat Pair as non-periodic—it behaves as if the field is still being exchanged periodically (figure also attached).

This suggests that something fundamental may have changed in ROMS 4.2 regarding how boundary exchanges are handled, especially with periodicity in distributed memory.

I’m wondering if there have been any changes in how mp_exchange2d or periodic boundary logic is applied in version 4.2 that could be overriding or bypassing my manual .FALSE. settings. Any insights would be greatly appreciated!

Thanks again for the support,
Hsin-Yi

[arango]

I think you need to abandon your idealized application. ROMS's analytical interface is for toy problems that usually include simple periodic boundary conditions in the x- or y-directions to avoid the issues with ill-conditioned boundary conditions in an idealized test. Otherwise, you have to specify the choices for other lateral boundary conditions, which you can do if you create such boundary data in a toy problem. The double periodic case converts ROMS into a one-dimensional column model to test vertical process ideas, as in the BIO_TOY test case.

Parallel exchanges between tile partitions work the same way as periodic boundary conditions exchanges. Periodic boundary conditions require special parallel treatment, and you cannot modify at will generic routines because it will destroy ROMS' parallelism. This design has been in ROMS since its beginnings more than two decades ago, and it has nothing to do with going from 4.1 to 4.2.

You should look at the estuary test case (roms_estuary.in), which is closed at the southern and northern boundaries and open at the east-west boundaries. Notice that periodic boundary conditions are not used.

This is the last thing I will say about this matter. I am too busy to pay attention to the nonsensical issues you are reporting.

[wilkin]

Check your f90 files in Build to see if you actually got what you intended.

[hyc006]

Hi Wilkin,

Thank you so much for your suggestion and your kind help!

Following your advice, I checked the Build/analytical.f90 file that was compiled during the ROMS 4.2 build process. I’ve confirmed that it does, in fact, contain exactly what I intended.

pair_part_in_analyticalf90.rtf

(10.21 KiB) Downloaded 2 times

Specifically, the relevant part of the ana_pair.h code that I modified shows up in analytical.f90 as follows:

Code: Select all

      CALL mp_exchange2d (ng, tile, model, 1,                           &
     &                    LBi, UBi, LBj, UBj,                           &
     &                    NghostPoints,                                 &
     &                    .FALSE.,.FALSE.,                              &  ! originally  EWperiodic(ng), NSperiodic(ng),
     &                    Pair)

and also comment out

Code: Select all

!      IF (EWperiodic(ng).or.NSperiodic(ng)) THEN
!        CALL exchange_r2d_tile (ng, tile,                               &
!     &                          LBi, UBi, LBj, UBj,                     &
!     &                          Pair)
!      END IF

This matches the intended setup — turning off periodicity explicitly for the Pair field. However, when I run the model, the behavior still looks like the air pressure field is being treated as periodic, which is quite confusing.

So, even though the compiled analytical.f90 looks correct, the runtime behavior doesn’t reflect that non-periodic configuration. I double-checked everything I could think of, but ROMS 4.2 still seems to override or ignore this .FALSE., .FALSE. flag during execution, while the same setup works fine in ROMS 4.1.

It’s been pretty frustrating, especially since the same logic behaves differently across versions. I really appreciate your support in helping to troubleshoot this — it means a lot.

Thanks again,
Hsin-Yi