I spent again few days in agony trying to find a parallel/optimization bug in ROMS 4DVAR data assimilation algorithms (IS4DVAR, W4DPSAS, and W4DVAR). Suddently, the PGI compiler started to behave chaotically and the model blowed-up for a particular application. The pgf90 compiler behaved differently with various MPI libraries (mpich, mpich2, openmpi) and versions (7.0, 7.1, 7.2).
I finally convinced myself that this was a compiler bug which I have not noticed before in ROMS. I had succesful runs with the ifort, gfortran, and g95 compilers. Then, we decided to update to PGI version 8.0 and the problem desappeared. The same happen recently with the ifort compiler. We have to update to version 10.1.
Therefore, users need to be aware that compilers continue evolving and there are still bugs in them. If this kind of problem happens, it is better to update to a new version and hope the old problems are resolved. Also it is possible that the new compiler version may fail compiling ROMS. Althought this has never happened to me except g95 under CygWin.
One trend in compiler technology is that the newer versions are slower. This is the case with the Intel compiler. I recall that when we started developing ROMS, over 10 years ago, the ifc (now ifort) compiler was much faster. The optimization in the ifort compiler is much slower nowadays.