ROMS not bit-reproducible for certain tilings

[k.alexander]

Since updating from 3.6 to the trunk (svn version 820), my configuration of ROMS is no longer bit-reproducible for certain tilings. For example 1x1, 2x2, 16x8 all give identical output (for multiple simulations with the same tiling as well as comparing different tilings) but 32x1, 32x16 diverge. See the full discussion here: viewtopic.php?f=14&t=4470 I am not the only one with this problem (see post by Serega.Osipov on the discussion page) so I don't think it's anything specific to my setup. I don't see any updates since svn 820 that look relevant to this problem, although I haven't yet updated to confirm.

[k.alexander]

I have confirmed that the same problem exists for the newest version of the code (as of today) with none of my own modifications (CICE coupling, ice shelves, etc.) See viewtopic.php?f=14&t=4470&p=17405#p17405.

[k.alexander]

I have fixed the problem by switching from openmpi 1.8.4 to openmpi 1.10.2. For ROMS 3.6, openmpi 1.8.4 was bit-reproducible. So something about the new kernel is now sensitive to the version of openmpi. Perhaps there should be a warning to users somewhere in the makefile comments to check their version of openmpi.