Test suite

[m.hadfield]

I think ROMS should have a test suite and I am writing one. At the moment it consists of a script that looks like this:

#! /bin/sh

make clean && make ROMS_APPLICATION=BASIN
make clean && make ROMS_APPLICATION=BENCHMARK
make clean && make ROMS_APPLICATION=BIO_TOY

I'm sure you get the idea. It's amazing how many problems can be identified with something this simple. Just look at my recent bug reports on the Trac system.

Any suggestions about how this could be expanded & made effective but relatively painless are welcome.

[rsignell]

Mark,

We agree that a test suite is a great idea for ROMS! Sachin Bhate, who is working on ROMS software tools under the NOPP Community Sediment Transport Project, has been working on just such a regression testing script (using Perl) to run a list of test cases in scalar, openmp and mpi modes, and do some simple checks on the netcdf results. Hopefully Sachin will post a follow-up here -- I know he had basically finished an alpha version of the script for ROMS 2.2 version when ROMS 3.0 was released, so now he's finishing up that. Perhaps you guys can work together on this!

-Rich

[skbhate]

Mark,

Elaborating on the point Rich just made, Yes I am working on a Regression test package for ROMS CSTM code. I did an earlier alpha release of the regression test package for ROMS 2.2. But, it was almost too close to the ROMS3.0 release, so decided to put it a hold on final release of the test package and upgrade it to work with ROMS3.0. Currently, I am working on upgrading it to work with ROMS3.0. It should be out for testing sometime this week (hopefully). The testing code is in Perl. You can look at some old documentation on it at this site http://skumar.tiddlyspot.com/ to have an idea about the test package.

You are more than welcome to be an alpha tester for the upcoming release of testing package and provide me with some valuble feedback.

-Sachin.

[arango]

Regression tests are used for production type applications whose configuration is usually the same but the source code evolves in time. However, what I think Mark is talking about is about a simple script to check if the model compiles at all. We need to clear compilation errors and other logic errors that arise as the code evolves. This usually involves turning on and off CPP options and checking for parallel (distributed- or shared-memory) bugs.

A regression test will fail for this type of code pruning. However, regression tests are good to check the vadility of new algorithms when compared with old algorithms. It will tell us that the solution is different, but it will not give us clues on how to resolve such differences.

ROMS has several sanity checks for the nonlinear, tangent linear, and adjoint numerical kernels that usually need to be done manually. There is no way to write a script for it, since we require a very sofisticated script training in algebra, adjoints, numerical kernel, and physics.

[m.hadfield]

My ideas on what a test suite should cover are very preliminary, but I guess I have been thinking of a regression test. I started with a simple script to make each of the idealised test applications in turn, because that is very easy to script with the current directory structure and uncovers enough bugs to keep me busy for a while. However it's clear one also needs to try *running* the model.

I would be happy to be a test tester for Sachin.

[kate]

I have long thought such a test option would be extremely useful. I agree that compiling for all the different cases is a good start, but running serial, OpenMP, and MPI would be a wonderful addition if it were possible. Hernan can tell you that I don't understand the OpenMP parts to be trusted not to break it. Then there's my stuff with all the goodies like tides, ice, point sources, etc, which don't always get tested for each little change.

[wilkin]

A testing framework that users could easily adapt to their own application for finer levels of granularity in testing would indeed be valuable. This would allow the communitiy to test the interaction of certain elements of the code which, when one considers all the possible combinations of CPP options, is practically-speaking infinite.

For the CBLAST application for example, I could test operation with/without various open boundary options, with/without surface forcing imposed via bulk fluxes, direct fluxes, or analytical options, with all vertical turbulence closures. These combinations exercise almost all possible netcdf I/O cases. For any given user's application, this sort of test would reassure a user like me that at least for the parts of the code that I care about everything is functioning correctly.

So I guess it would take some sort of perl script with a perl database (?) where a user could check of a table or matrix of CPP options to be tested.