I've been tearing my hair out over time conventions in NetCDF, having managed to generate a bunch of ROMS forcing files that were off by two days. This has led me to a troubling conclusion that there is an inconsistency between julian date calculators in matlab and the COARDS standard.
The Matlab julian function I have (attributed to Rich) and the built-in datenum functions both give:
julian(1990,1,1) - julian(1,1,1) = 726467
datenum(1990,1,1) - datenum(1,1,1) = 726467
However, http://www.numerical-recipes.com/julian.html says that Jan 1, 1 A.D. is followed 726469 days later by Jan 1, 1990. So, there is a 2-day difference.
I'm looking at data from the ETA-12km forecast on the GraDS/DODS server at
http://nomads.ncdc.noaa.gov:9090/dods/N ... ETA/200405
These data claim to implement the COARDS and Grads/DODS standards and give times in units: "days since 1-1-1 00:00:0.0" . When I apply the Matlab julian or datenum functions to this base date plus the time data in the files, I need to subtract 2 days from the result to get the correct dates. I know what the correct dates should be because the data files include attributes for minimum and maximum time values in the string format, e.g. minimum: "00Z01MAY2004".
Do you have any clue how to verify the results of julian and datenum vs. the COARDS standards?
Thanks, John
Model Time Clocks
John's question on time convention brings up a point that I think is important for the ROMS community, and I'd be interested in hearing what others have to say.
Here's my take on handling time:
Because the world did not agree on a calendar until after 1753, it is very dangerous to use any date (YYYY-MM-DD) before this to specify your time origin.
Thus days since 1-1-1 00:00:0.0 is a very bad idea, because there was no agreed upon Jan 1 back then! Usually this means that whoever wrote the data just wrote their julian day time variable into the file without modification, assuming that other people's gregorian(0) function would give the same as theirs!
For all the NetCDF files I generate, I use:
time-units since 1968-05-23 00:00:00 GMT
which is also how ROMS stores time (if you use TIME_REF = -2).
This has several nice features:
1) Even if you are not familiar with ISO 8601 time specification, http://www.cl.cam.ac.uk/~mgk25/iso-time.html , it's hard to get confused between month and day, unless you think there are 23 months in the year.
2) The Julian Day number corresponding to this date is a nice round number = 2440000. (You can type in May 23, 1968 at http://www.numerical-recipes.com/julian.html to confirm this).
3) It is close enough to the present time that the time variable values are not too huge, and have good precision.
However, a stronger case might be made for using:
time-units since 1858-11-16 00:00:00 GMT
since it shares the same advantages of 1968-05-23, but also has a recognized name (Modified Julian Date) according to the official masters of time
http://tycho.usno.navy.mil/mjd.html
and would allow data collected since the beginning of oceanographic data collection to have positive time values.
The main point is that it either one of these would be completely unambigious, whereas using dates prior to 1752 leads to confusion, as John unfortunately found out.
Thoughts?
-Rich
Here's my take on handling time:
Because the world did not agree on a calendar until after 1753, it is very dangerous to use any date (YYYY-MM-DD) before this to specify your time origin.
Thus days since 1-1-1 00:00:0.0 is a very bad idea, because there was no agreed upon Jan 1 back then! Usually this means that whoever wrote the data just wrote their julian day time variable into the file without modification, assuming that other people's gregorian(0) function would give the same as theirs!
For all the NetCDF files I generate, I use:
time-units since 1968-05-23 00:00:00 GMT
which is also how ROMS stores time (if you use TIME_REF = -2).
This has several nice features:
1) Even if you are not familiar with ISO 8601 time specification, http://www.cl.cam.ac.uk/~mgk25/iso-time.html , it's hard to get confused between month and day, unless you think there are 23 months in the year.
2) The Julian Day number corresponding to this date is a nice round number = 2440000. (You can type in May 23, 1968 at http://www.numerical-recipes.com/julian.html to confirm this).
3) It is close enough to the present time that the time variable values are not too huge, and have good precision.
However, a stronger case might be made for using:
time-units since 1858-11-16 00:00:00 GMT
since it shares the same advantages of 1968-05-23, but also has a recognized name (Modified Julian Date) according to the official masters of time
http://tycho.usno.navy.mil/mjd.html
and would allow data collected since the beginning of oceanographic data collection to have positive time values.
The main point is that it either one of these would be completely unambigious, whereas using dates prior to 1752 leads to confusion, as John unfortunately found out.
Thoughts?
-Rich
I consulted another Time Lord: Jim Mansbridge also emphasizes that using times before 1582 is asking for trouble...
John,
There is a well-known problem with dates in general. Julias Caeser didn't get leap years right and so there was a slow drift between the Julian calendar and astronomical time. This was fixed up by Pope Gregory with the Gregorian calendar in 1582. The new formula involved a discontinuity; at the changeover something like 10 days was lost. matlab and Rich's function use Pope Gregory's formula and so they should not strictly be used before 1582. However, using this formula for all dates means that at least time is continuous. Numerical recipes is correct for historical time (you can see the IF statement to change calendars) but time is discontinuous. To make matters even more confusing, udunits (which is used by the netcdf libraries) used to do things the matlab way but changed some years ago to switch calendars in 1582. The end result is that any given netcdf file with a reference date before 1582 could have time calculated either way.
If you really care about historical dates it is even more confusing since the changeover of calendars happened at different times in different countries - catholics in 1582, protestants a few hundred years later mostly and orthodox christians often not until the twentieth century. Sweden was the funniest, changing backwards and forwards a few times and sometimes using a home-grown version of the calendar. There were riots in London over the matter.
Jim
________________________________________________________________________
Jim Mansbridge
mail: CSIRO Marine Laboratories, GPO Box 1538, Hobart, Australia 7001
email: Jim.Mansbridge@csiro.au
phone: +61 3 6232 5416 fax: +61 3 6232 5123
John,
There is a well-known problem with dates in general. Julias Caeser didn't get leap years right and so there was a slow drift between the Julian calendar and astronomical time. This was fixed up by Pope Gregory with the Gregorian calendar in 1582. The new formula involved a discontinuity; at the changeover something like 10 days was lost. matlab and Rich's function use Pope Gregory's formula and so they should not strictly be used before 1582. However, using this formula for all dates means that at least time is continuous. Numerical recipes is correct for historical time (you can see the IF statement to change calendars) but time is discontinuous. To make matters even more confusing, udunits (which is used by the netcdf libraries) used to do things the matlab way but changed some years ago to switch calendars in 1582. The end result is that any given netcdf file with a reference date before 1582 could have time calculated either way.
If you really care about historical dates it is even more confusing since the changeover of calendars happened at different times in different countries - catholics in 1582, protestants a few hundred years later mostly and orthodox christians often not until the twentieth century. Sweden was the funniest, changing backwards and forwards a few times and sometimes using a home-grown version of the calendar. There were riots in London over the matter.
Jim
________________________________________________________________________
Jim Mansbridge
mail: CSIRO Marine Laboratories, GPO Box 1538, Hobart, Australia 7001
email: Jim.Mansbridge@csiro.au
phone: +61 3 6232 5416 fax: +61 3 6232 5123
I delved a bit deeper into descriptions of the time metadata formats used in the Grads/DODS server (the UDUNITS definitions). This link is useful:
http://www.cgd.ucar.edu/cms/eaton/cf-me ... .html#time
It says:
On a positive note, I was saved in this case by the fact that the files I am accessing have check values in the attributes. The ETA model meteorological forecast data on the GDS server http://nomads.ncdc.noaa.gov have attributes that give the minimum and maximum times in each file.
So in addition to the safe time keeping practice of sticking to "days since sometime-in-recent-memory" I'd also advise adding attributes to your ROMS forcing files that clearly spell out the valid range of the time data.
http://www.cgd.ucar.edu/cms/eaton/cf-me ... .html#time
It says:
This further emphasizes Rich's contention, and my discovery, that using "days since 1-1-1" or any date before 1582 is asking for trouble. In most of the data I've dealt with recently base dates in the 20th century are used, and then there is no ambiguity.The mixed Gregorian/Julian calendar used by Udunits is explained in the following excerpt from the udunits(3) man page:
The udunits(3) package uses a mixed Gregorian/Julian calendar system. Dates prior to 1582-10-15 are assumed to use the Julian calendar, which was introduced by Julius Caesar in 46 BCE and is based on a year that is exactly 365.25 days long. Dates on and after 1582-10-15 are assumed to use the Gregorian calendar, which was introduced on that date and is based on a year that is exactly 365.2425 days long. (A year is actually approximately 365.242198781 days long.) Seemingly strange behavior of the udunits(3) package can result if a user-given time interval includes the changeover date. For example, utCalendar() and utInvCalendar() can be used to show that 1582-10-15 *preceded* 1582-10-14 by 9 days.
Due to problems caused by the discontinuity in the default mixed Gregorian/Julian calendar, we strongly recommend that this calendar should only be used when the time coordinate does not cross the discontinuity. For time coordinates that do cross the discontinuity the proleptic_gregorian calendar should be used instead.
On a positive note, I was saved in this case by the fact that the files I am accessing have check values in the attributes. The ETA model meteorological forecast data on the GDS server http://nomads.ncdc.noaa.gov have attributes that give the minimum and maximum times in each file.
So in addition to the safe time keeping practice of sticking to "days since sometime-in-recent-memory" I'd also advise adding attributes to your ROMS forcing files that clearly spell out the valid range of the time data.
-
arango
- Site Admin
- Posts: 1367
- Joined: Wed Feb 26, 2003 4:41 pm
- Location: DMCS, Rutgers University
- Contact:
I think that ROMS/TOMS already has a consistent and unambiguos way to determine time. See the documentation in the input script ocean_*.in for the input variable time_ref. The problem is that some atmosphere and/or ocean products do not follow a consistent standard. Also, sometimes there is a misinterpretation for January 1. It is interpreted as day zero by some and day one by others.
The parameter time_ref is the reference time (yyyymmdd.f) used to compute relative model time: elapsed time interval since reference-time. In the model NetCDF files, the units attribute takes the form time-unit since reference-time. This parameter also provides information about the calendar used:
(1) If time_ref = -2, model time and dstart are in modified Julian days units. The units attribute is:
time-units since 1968-05-23 00:00:00 GMT
(2) If time_ref = -1, model time and dstart are in a calendar with 360 days in every year (30 days each month). The units attribute is:
time-units since 0001-01-01 00:00:00
(3) If time_ref = 0, model time and dstart are in a common year calendar with 365.25 days. The units attribute is:
time-units since 0001-01-01 00:00:00
(4) If time_ref > 0, model time and dstart are the elapsed time units since specified reference time. For example, time_ref=20020115.5 gives the following attribute:
time-units since 2002-01-15 12:00:00
Options 2 and 3 can be used only in climatological solutions with climatological forcing where the time clock is not important. The problem mentioned above arises because some atmosphere products use similar time reference. As we can see from this discussion, time references before 1582 must be avoided.
Hernan
The parameter time_ref is the reference time (yyyymmdd.f) used to compute relative model time: elapsed time interval since reference-time. In the model NetCDF files, the units attribute takes the form time-unit since reference-time. This parameter also provides information about the calendar used:
(1) If time_ref = -2, model time and dstart are in modified Julian days units. The units attribute is:
time-units since 1968-05-23 00:00:00 GMT
(2) If time_ref = -1, model time and dstart are in a calendar with 360 days in every year (30 days each month). The units attribute is:
time-units since 0001-01-01 00:00:00
(3) If time_ref = 0, model time and dstart are in a common year calendar with 365.25 days. The units attribute is:
time-units since 0001-01-01 00:00:00
(4) If time_ref > 0, model time and dstart are the elapsed time units since specified reference time. For example, time_ref=20020115.5 gives the following attribute:
time-units since 2002-01-15 12:00:00
Options 2 and 3 can be used only in climatological solutions with climatological forcing where the time clock is not important. The problem mentioned above arises because some atmosphere products use similar time reference. As we can see from this discussion, time references before 1582 must be avoided.
Hernan
Last edited by arango on Fri Jan 25, 2008 3:32 pm, edited 1 time in total.
-
m.hadfield
- Posts: 521
- Joined: Tue Jul 01, 2003 4:12 am
- Location: NIWA
Hi guys
I've just been rereading this topic and would like to comment on a couple of things. First Rich's contibution...
Unfortunately "days since 1-1-1 00:00:0.0" does seem to be an established convention, eg. the CDC netCDF conventions say it is the "normally used CDC base date/time". So it's sometimes necessary to cope with it, though I think it's unwise to perpetuate it.
I use IDL for most of my data analysis, and it seems to agree with the CDC software about when 0001-01-01 00:00 was, so I'm happy.
The above URL says
The reason for the confusion, I think, is the half day mentioned above. Julian Dates reach integer values at 12:00 each day. Thus the time right now is 2004-10-13 22:42 UTC, which is Julian Date/time 2453292.44583. So the Julian Day number is 2453292, but the Julian Date/Time at 00:00 this morning was 2453291.5.
Having confidently written the above, I thought I should check it against Matlab. So I tried the julian function, part of the OMVIZ tools
What?!
But then I read the documentation for this function, which says:
Anyway, I submit that the base date for the MJD system is 1858-11-17 00:00:00 UTC. If we can agree on that, then it doesn't really matter what functions we use for intermediate calculations.
On another matter, what's the reasoning behind ROMS's default time units:
I see the CDC netCDF conventions bless this usage
I've just been rereading this topic and would like to comment on a couple of things. First Rich's contibution...
Amen to that, brother!rsignell wrote: Because the world did not agree on a calendar until after 1753, it is very dangerous to use any date (YYYY-MM-DD) before this to specify your time origin.
Thus days since 1-1-1 00:00:0.0 is a very bad idea, because there was no agreed upon Jan 1 back then!
Unfortunately "days since 1-1-1 00:00:0.0" does seem to be an established convention, eg. the CDC netCDF conventions say it is the "normally used CDC base date/time". So it's sometimes necessary to cope with it, though I think it's unwise to perpetuate it.
I use IDL for most of my data analysis, and it seems to agree with the CDC software about when 0001-01-01 00:00 was, so I'm happy.
I think you're out by one day.rsignell wrote: However, a stronger case might be made for using:
time-units since 1858-11-16 00:00:00 GMT
since it shares the same advantages of 1968-05-23, but also has a recognized name (Modified Julian Date) according to the official masters of time
http://tycho.usno.navy.mil/mjd.html
The above URL says
By my reckoning (actually by the reckoning of the JULDAY function built into IDL) Julian Date 2400000.5 (the base date for the MJD system) is 1858-11-17 00:00:00 UTC.The Modified Julian Day, on the other hand, was introduced by space scientists in the late 1950's. It is defined as
MJD = JD - 2400000.5
The half day is subtracted so that the day starts at midnight in conformance with civil time reckoning.
The reason for the confusion, I think, is the half day mentioned above. Julian Dates reach integer values at 12:00 each day. Thus the time right now is 2004-10-13 22:42 UTC, which is Julian Date/time 2453292.44583. So the Julian Day number is 2453292, but the Julian Date/Time at 00:00 this morning was 2453291.5.
Having confidently written the above, I thought I should check it against Matlab. So I tried the julian function, part of the OMVIZ tools
Code: Select all
>> julian([2004 10 13 22 42 0])
ans =
2.453292945833333e+006
But then I read the documentation for this function, which says:
Sorry, but that's just wrong. The numbers returned by this function are not Julian Dates, and it's very confusing to call them such.JULIAN
Converts Gregorian calendar dates to corresponding Julian day numbers. Although the formal definition holds that Julian days start and end at noon, here Julian days start and end at midnight.
In this convention, Julian day 2440000 began at 0000 hours, May 23, 1968.
Anyway, I submit that the base date for the MJD system is 1858-11-17 00:00:00 UTC. If we can agree on that, then it doesn't really matter what functions we use for intermediate calculations.
On another matter, what's the reasoning behind ROMS's default time units:
There was no year 0! (Well, OK, it's pretty silly to worry about whether there was or wasn't, for the reasons given by Rich. But the IDL routines think there wasn't.)arango wrote: If time_ref = 0, model time and dstart are in a common year calendar with 365.25 days. The units attribute is:
time-units since 0000-01-01 00:00:00
I see the CDC netCDF conventions bless this usage
OK, but in some model runs, the time is not related to the time of year. (Think of the ROMS UPWELLING case.) In the default case, wouldn't it be simpler not to mention any base date and just set the units to "seconds" or "days" or whatever.Time coordinate variables representing climatological time (an axis of 12 months, 4 seasons, etc. that is located in no particular year) should be encoded like other time axes but with the added restriction that they be encoded to begin in the year 0000.
-
arango
- Site Admin
- Posts: 1367
- Joined: Wed Feb 26, 2003 4:41 pm
- Location: DMCS, Rutgers University
- Contact:
0000-01-01 versus 0001-01-01 convention
The climatological time clock is kind arbitrary. In the past we have 0000-01-01 as the started base time. This is what is descibed in the COARDS netCDF 
convention. However, the encoding to start the climatological time at year 0000 is problematic in other software. There are some problems with this starting base and new attributes are suggested. I made the change to base year 0001 because it was necessary in the ESMF library during multiple model coupling. I did changed all Fortran code but not the documentation in the input scripts.
A new trac ticket was issue today about this. I am very aware of this inconsistence and it is already fixed in the new version of ROMS that will be released soon. This new version already has all the ESMF drivers. It also has several changes which are needed in the multidomain nested (refine, mosaic, composite) grids.
convention. However, the encoding to start the climatological time at year 0000 is problematic in other software. There are some problems with this starting base and new attributes are suggested. I made the change to base year 0001 because it was necessary in the ESMF library during multiple model coupling. I did changed all Fortran code but not the documentation in the input scripts.A new trac ticket was issue today about this. I am very aware of this inconsistence and it is already fixed in the new version of ROMS that will be released soon. This new version already has all the ESMF drivers. It also has several changes which are needed in the multidomain nested (refine, mosaic, composite) grids.
-
m.hadfield
- Posts: 521
- Joined: Tue Jul 01, 2003 4:12 am
- Location: NIWA
That's an old thread you've posted on, Hernan!
Why does ESMF require this change? Though I know next-to-nothing about ESMF, it appears to be very flexible in its supprt for different calendars, eg see:
http://www.esmf.ucar.edu/esmf_releases/ ... 0000000000
Might it be better to drop all this "days since (some meaningless or long-ago date)" stuff? If I'm doing a model simulation in which the starting time is not tied to any actual date, why can't I specify a units string of "days" or "days since beginning of year".
Here's a concrete proposal: make the units and cycle_length attributes of the ROMS time variable (optionally) settable by the user.
Why does ESMF require this change? Though I know next-to-nothing about ESMF, it appears to be very flexible in its supprt for different calendars, eg see:
http://www.esmf.ucar.edu/esmf_releases/ ... 0000000000
Might it be better to drop all this "days since (some meaningless or long-ago date)" stuff? If I'm doing a model simulation in which the starting time is not tied to any actual date, why can't I specify a units string of "days" or "days since beginning of year".
Here's a concrete proposal: make the units and cycle_length attributes of the ROMS time variable (optionally) settable by the user.
-
filipe
Re:
Hello all,
I know that this thread is really old...
Anyway, I think that some people might be interested in this shell-script:
http://ocefpaf.tiddlyspot.com/#gymd2rday
I got the original here:
http://home.online.no/~pjacklam/shell/programs/time/
It converts dates (Julian/Gregorian) using the Rata Die Julian. It agrees with ETA-12km forecast on the GraDS/DODS standards and I use it to avoid hardcoding the date shift in scripts. Here is part of Wilkin's script where I modified:
Hope that helps, Filipe.
I know that this thread is really old...
Anyway, I think that some people might be interested in this shell-script:
http://ocefpaf.tiddlyspot.com/#gymd2rday
I got the original here:
http://home.online.no/~pjacklam/shell/programs/time/
It converts dates (Julian/Gregorian) using the Rata Die Julian. It agrees with ETA-12km forecast on the GraDS/DODS standards and I use it to avoid hardcoding the date shift in scripts. Here is part of Wilkin's script where I modified:
Code: Select all
scratch=scr.$local_file
# forcing time shift starting at Jan day 01 from the input year
shift=`gymd2rday $yyyy 1 1`
ncap2 -s 'time=time-'$shift --overwrite $local_file $scratch
ncatted -a units,time,m,c,"days since $yyyy-01-01 00:00:0.0" $scratch
ncatted -a long_name,time,m,c,"time" $scratch
ncatted -a grads_step,time,m,c,"3hr" $scratch
ncatted -a minimum,time,m,c," " $scratch
ncatted -a maximum,time,m,c," " $scratch
mv $scratch $local_file-
turuncu
- Posts: 128
- Joined: Tue Feb 01, 2005 8:21 pm
- Location: Istanbul Technical University (ITU)
- Contact:
Re: Model Time Clocks
Hi All,
When i try to couple ROMS and WRF using ESMF, i modify the ROMS/Utility/get_date.F file to get correct date value. Because ROMS used each gridded component start and end date to calculate the coupler simulation interval, start and end date. In my application, the ocean.in definition was,
but the non-modified ROMS gave the 1999-06-31 as a start date of the ROMS model. So, i modify the file as,
i am not sure it is correct or not but it generates the correct date interval for my application and this way i could calculate the coupler time clock.
--ufuk
When i try to couple ROMS and WRF using ESMF, i modify the ROMS/Utility/get_date.F file to get correct date value. Because ROMS used each gridded component start and end date to calculate the coupler simulation interval, start and end date. In my application, the ocean.in definition was,
Code: Select all
DSTART = 0.0d0 ! days
TIDE_START = 0.0d0 ! days
TIME_REF = 19990701.0d0 ! yyyymmdd.dd
Code: Select all
integer, parameter :: gregorian = 2299161
- integer :: leap, ja, jalpha, jb, jc, jd, jday, je
+ integer :: i, leap, ja, jalpha, jb, jc, jd, jday, je
integer, dimension(13) :: iyd = &
& (/ 1,32,60,91,121,152,182,213,244,274,305,335,366 /)
@@ -301,9 +301,14 @@
month=MIN(12,1+INT((yday+1.0_r8)/30.6001_r8))
day=INT(yday)-iyd(month)+1
ELSE
- month=MIN(12,1+INT(yday/30.6001_r8))
- IF ((yday.gt.59).and.(yday.le.90)) month=3
- day=INT(yday)-iydl(month)+1
+ month = 0
+ DO i = 1, size(iyd, 1)-1
+ IF ((yday .GE. iyd(i)) .AND. (yday .LT. iyd(i+1)) &
+ & .AND. (month .EQ. 0)) THEN
+ month = i
+ END IF
+ END DO
+ day=INT(yday)-iyd(month)+1
END IF
hour=(rday-AINT(rday))*24.0_r8
--ufuk