Number of history files greater than 9999
Number of history files greater than 9999
Hello! I need some help; I hope someone can help me.
I have a simulation with output files greater than 9,999, but I don't know how to configure the ROMS to write the files without restart the simulation because the limit of history files is 9,999.
Thank you!
Gabriela
CICESE
PhD student
CICESE
PhD student
Re: Number of history files greater than 9999
It's quite easy. Just change output.F so that you have:
instead of
and similar for all the other format statements.
Code: Select all
10 FORMAT (a,'_',i5.5,'.nc')
Code: Select all
10 FORMAT (a,'_',i4.4,'.nc')
Re: Number of history files greater than 9999
Thank you for your help, Kate! I made the changes and will run the model.
Gabriela
CICESE
PhD student
CICESE
PhD student
- arango
- Site Admin
- Posts: 1367
- Joined: Wed Feb 26, 2003 4:41 pm
- Location: DMCS, Rutgers University
- Contact:
Re: Number of history files greater than 9999
Yes, this is an easy change to make. However, making such a change will affect our nomenclature conventions for ensemble runs, 4D-Var forward solutions, operational set-ups, etc. I am reluctant to make such a change.
You are the first user since ROMS was released more than 20 years ago that creates over 10000 history files with the same base filename. That's a lot of I/O for a single solution and not that easy to manage. When we run applications for multiple years, we create a series of files where the year enters the filename to facilitate the management, analysis, and storage. ROMS has a perfect restart option, so the solution can be run for chunks of time. It is very unlikely that we create a 10000 counter in a single year. In the operational environment, we can have files like xxxx_roms_his_2022_0001.nc, and so on for solutions corresponding to year 2022. If we are doing forecasting, we usually have xxxx_roms_his_20220115.nc4 for a forecast issued on Jan 15, 2022. Here xxxx is a prefix to identify a particular application or region. As you can see, the user has full control over how the files are named and how many time records are included in each file. The I/O can be a bottleneck, so the user also takes into consideration how frequently to write fields into the average, diagnostics, floats, stations, history, quicksave, and restart files. When doing long solutions, we use the average file to record monthly, seasonal, and annual trends. The time-averaged file has important characteristics that you cannot see in the instantaneous fields.
You are the first user since ROMS was released more than 20 years ago that creates over 10000 history files with the same base filename. That's a lot of I/O for a single solution and not that easy to manage. When we run applications for multiple years, we create a series of files where the year enters the filename to facilitate the management, analysis, and storage. ROMS has a perfect restart option, so the solution can be run for chunks of time. It is very unlikely that we create a 10000 counter in a single year. In the operational environment, we can have files like xxxx_roms_his_2022_0001.nc, and so on for solutions corresponding to year 2022. If we are doing forecasting, we usually have xxxx_roms_his_20220115.nc4 for a forecast issued on Jan 15, 2022. Here xxxx is a prefix to identify a particular application or region. As you can see, the user has full control over how the files are named and how many time records are included in each file. The I/O can be a bottleneck, so the user also takes into consideration how frequently to write fields into the average, diagnostics, floats, stations, history, quicksave, and restart files. When doing long solutions, we use the average file to record monthly, seasonal, and annual trends. The time-averaged file has important characteristics that you cannot see in the instantaneous fields.
Re: Number of history files greater than 9999
He is at least the second person to run into this issue. I ran into it with daily output over 40 years, each day in its own file. Maybe it wouldn't happen with today's code, I don't know.
- arango
- Site Admin
- Posts: 1367
- Joined: Wed Feb 26, 2003 4:41 pm
- Location: DMCS, Rutgers University
- Contact:
Re: Number of history files greater than 9999
Hi Kate, since ROMS now has a full-time management module (dateclock.F) that even includes a Fortran version of Matlab's datenum and datestr, it has been in my TODO list to add an option that includes some date format in the filename. Additionally, it also has day-of-the-year, ISO 8601 format, and others. I added all that capability when I coded the ESMF/NUOPC coupling interface. In my opinion, a date string is a way to go when having a large number of files in a solution. It is easy to identify a file for a particular date instead of trial and error by looking at the counter. The command ncdump -f -v ocean_time file.nc is also very useful.
Re: Number of history files greater than 9999
Yes, someone in Enrique's lab wrote a Python code to rename files based on the date in there. Very useful!
MOM6 files all have names starting with the date, such as 20100101.ice_daily.nc
MOM6 files all have names starting with the date, such as 20100101.ice_daily.nc
Re: Number of history files greater than 9999
Thank you, Arango and Kate; I had a problem restarting my model in the past, so I tried running the model once time.
In this time, I was able to restart without a problem, and the two solutions helped me.
Thank you.
Gabriela
In this time, I was able to restart without a problem, and the two solutions helped me.
Thank you.
Gabriela
Gabriela
CICESE
PhD student
CICESE
PhD student