Problem with scaling of non-gridded data

[m.hadfield]

There is a problem with the scaling of non-gridded data from netCDF files using the scale_factor and add_offset attributes. The scale factor is being applied twice.

For example I have a netCDF file with ubar data on the western boundary as follows:

Code: Select all

netcdf roms_bry {
dimensions:
        xi_u = 79 ;
        eta_u = 89 ;
        time_fix = 1 ;
variables:
        float time_fix(time_fix) ;
        short ubar_west(time_fix, eta_u) ;
                ubar_west:time = "time_fix" ;
                ubar_west:add_offset = -0.000152594875864068 ;
                ubar_west:scale_factor = 0.000305189751728137 ;

// global attributes:
                :type = "Boundary forcing file" ;
data:

 ubar_west =
  20, 11, -14, -55, -20, 43, -3, -122, -64, 254, 442, 301, 176, 182, 260, 
  299, 257, 200, 139, 82, 22, -10, 6, 34, 50, 41, 40, 42, 37, 24, 5, -13, 
  -24, -24, -18, -9, -1, 4, 10, 15, 16, 11, 1, -6, -6, 1, 5, 6, 5, 3, 3, 
  2, 4, 6, 7, 6, 2, -1, -1, 1, 3, 5, 2, -3, -6, 1, 6, 8, 7, 5, 3, 1, 3, 
  8, 13, 18, 19, 18, 14, 12, 12, 8, 6, 12, 20, 17, -3, -27, -38 ;
}
 

The maximum and minimum integer values are -122 and 442, respectively. With the scaling attributes in effect, these numbers scale to physical values of -0.0374 and 0.1347.

This file is being read by GET_NGFLD, which reports

Code: Select all

    GET_NGFLD   - 2D u-momentum western boundary condition,  t =     0 00:00:00
                   (Rec=0001, Index=2, File: roms_bry.nc)
                   (Tmin=          0.0000 Tmax=          0.0000)
                   (Min = -1.14097461E-05 Max =  4.11216564E-05)

The minimum and maximum reported by GET_NGFLD are much too small, by a factor equal to the scale_factor attribute, 0.0003051.

With a bit of debugging in get_ngfld.F, I see the data in question are being read from the netCDF file at lines 400-404 by function NETCDF_GET_FVAR, then the minimum and maximum values are calculated at lines 410-421 and written to stdout at lines 432-434. The scale_factor attribute was previously read from the file (line 147) and stored in Finfo(10,ifield,ng). This scale factor is applied in the calculation of the minimum and maximum at lines 410-421, but the data returned by NETCDF_GET_FVAR have already been scaled inside that function and are between -0.0374 and 0.1347, so the extra scaling causes the numbers to be much too small.

I'm not 100% sure but I think the scale factor in Finfo(10,ifield,ng) is also passed out of GET_NGFLD and used to scale the values used in the hydrodynamic simulation, leading to unrealistic results.

I think the same problem applies to point data processed by GET_2DFLD, see lines 448-456 in get_2dfld.F.

[jivica]

Are you sure that you are using the last version of ROMS?
This was corrected recently, we had the problem with atmo forcing scaled to short int (to save diskspace)!
Think there are still some issues if trying to read boundary files using scales (short int), but those files are small anyway and we are not using that feature.

Cheers,
Ivica

[m.hadfield]

I was using code from my own branch on the developers's SVN site, which I recently brought up to date with Hernan's branch through to revision 1155 (implementing Trac ticket 407). To the best of my knowledge the latter is the same as the master version

I just checked the master copy of get_ngfld.F. It has the double-scaling problem that I reported.

I agree that boundary files are generally small, so there is not a lot to gain by packing them. Still the present behaviour is clearly wrong and should be fixed. I'm happy to have a go myself if no-one else is willing/able.

[arango]

Yes, this is new bug. The generic routines in mod_netcdf.F take care of the scale_factor and add_offset attributes. The Fscale factor is a different issue in ROMS and doesn't have nothing to do with the compression of data but field units manipulation from varinfo.dat. Usually, Fscale=1.0 for mostly all input fields.

I fixed this problem and updated the repository. Thank you for reporting this problem.

[m.hadfield]

I believe the same erroneous scaling occurs for point data in get_2dfld.F (line 453), get_2dlfr.F (line 463), get_3dfld.F (line 434) and get_3dfldr.F (line 443).

[arango]

Yeah, the same correction is also needed for get_2dfld.F, get_2dfldr.F, get_3dfld.F, and get_3dfldr.F when processing point-data. However, it very unlikely to have an input NetCDF file with point-data that needs to be compressed as byte and scaled with the scale_factor and add_offset attributes.

Thank you, I re-opened the ticket 408 and updated the repository.