about the restarting the model

[mashinde]

Dear friends,

I have error during restarting the model.
--------------------------------------------------------------------------
0 1 00:02:00 6.266689E-08 1.830794E+04 1.830794E+04 1.331604E+16
DEF_AVG - inquiring average file: /share/scratch/odc/mahesh/roms_files/output/opti_test/avg_rmed16_grd_e_opt_test_0000.nc

NETCDF_OPEN - unable to open existing NetCDF file:
/share/scratch/odc/mahesh/roms_files/output/opti_test/avg_rmed16_grd_e_opt_test_0000.nc
call from: def_avg.F
--------------------------------------------------------------------------
here is my input file options used for restarting the model.....


! Input/Output parameters.

! NRREC == 0 !for new job
NRREC == -1 !for restart the model

LcycleRST == T

NRST == 1
NSTA == 1

NFLT == 1
NINFO == 1

! Output history, average, diagnostic files parameters.

LDEFOUT == T !for creating new files
! LDEFOUT == F !for appending existing files

NHIS == 1
NDEFHIS == 1

NTSAVG == 1
NAVG == 1
NDEFAVG == 1

NTSDIA == 1
NDIA == 1
NDEFDIA == 1

--------------------------------------------------------------------------
and here are the input files to be used....already replaced ini file with rst file....


GRDNAME == /share/data/odc/mahesh/roms_files/input/rmed16_grd_e/rmed16_grd_etopo5_e.nc
CLMNAME == /share/data/odc/mahesh/roms_files/input/rmed16_grd_e/rmed16_Lclm_e.nc
ITLNAME == /dev/null
IRPNAME == /dev/null
IADNAME == /dev/null
! ININAME == /share/data/odc/mahesh/roms_files/input/rmed16_grd_e/rmed16_ini_e.nc
ININAME == /share/scratch/odc/mahesh/roms_files/output/opti_test/rst_rmed16_grd_e_opt_test.nc
BRYNAME == /share/data/odc/mahesh/roms_files/input/rmed16_grd_e/rmed16_bry_e.nc
FWDNAME == /dev/null
ADSNAME == /dev/null
--------------------------------------------------------------------------
i created following o/p files...

-rw-r--r-- 1 mshinde odc 375974404 Nov 30 18:37 his_rmed16_grd_e_opt_test_0001.nc
-rw-r--r-- 1 mshinde odc 375974404 Nov 30 18:37 his_rmed16_grd_e_opt_test_0002.nc
-rw-r--r-- 1 mshinde odc 382347644 Nov 30 18:37 avg_rmed16_grd_e_opt_test_0001.nc
-rw-r--r-- 1 mshinde odc 1489711400 Nov 30 18:38 dia_rmed16_grd_e_opt_test_0001.nc
-rw-r--r-- 1 mshinde odc 375974404 Nov 30 18:38 his_rmed16_grd_e_opt_test_0003.nc
-rw-r--r-- 1 mshinde odc 382347644 Nov 30 18:38 avg_rmed16_grd_e_opt_test_0002.nc
-rw-r--r-- 1 mshinde odc 1489711400 Nov 30 18:39 dia_rmed16_grd_e_opt_test_0002.nc
-rw-r--r-- 1 mshinde odc 375974404 Nov 30 18:39 his_rmed16_grd_e_opt_test_0004.nc
-rw-r--r-- 1 mshinde odc 382347644 Nov 30 18:39 avg_rmed16_grd_e_opt_test_0003.nc
-rw-r--r-- 1 mshinde odc 1489711400 Nov 30 18:40 dia_rmed16_grd_e_opt_test_0003.nc
-rw-r--r-- 1 mshinde odc 2138528760 Nov 30 18:40 rst_rmed16_grd_e_opt_test.nc
-rw-r--r-- 1 mshinde odc 375974404 Nov 30 18:40 his_rmed16_grd_e_opt_test_0005.nc
-rw-r--r-- 1 mshinde odc 382347644 Nov 30 18:40 avg_rmed16_grd_e_opt_test_0004.nc
-rw-r--r-- 1 mshinde odc 1489711400 Nov 30 18:40 dia_rmed16_grd_e_opt_test_0004.nc
--------------------------------------------------------------------------
i stoped the model at his_rmed16_grd_e_opt_test_0005.nc ....
and now I need to restart the model so it will create avg_rmed16_grd_e_opt_test_0004.nc/dia_rmed16_grd_e_opt_test_0004.nc onwards...........

but it looks the avg_rmed16_grd_e_opt_test_0000.nc file and gives the error.........why this happened.

NOTE: PERFECT_RESTART option is undefined in cppdef.h file
(the same error persist even I checked with defined PERFECT_RESTART option)

I do not understand why it always looking for ......_0000.nc file when I set the options for restart in input file.

I don't want to restart the model from the begining once it stopped due to other reasons.

any suggestions ?


Thanks in advance.

[kate]

Well, you can look in output.F to see what it's trying to do. The options you have picked with:

Code: Select all

NTSAVG == 1
NAVG == 1
NDEFAVG == 1

could well be untested and buggy. Or else it somehow isn't restarting at the time it should be (which is what? step 3? step 5?) I'm puzzled but I'd watch it in the debugger.

[mashinde]

Hi Kate,

Thanks for the reply. I am still checking it.....
is there any connection with mixing schemes...i am using KPP.?
perfect_restart option is undefined.

I am still unable to restart the model to create multiple avg/his/dia files .
It is very important to know how can I restart the model in my application.


any suggestion/comments.


Thanking you

[kate]

Can you restart say the upwelling case after a day, saving at its default frequency?

[mashinde]

dear kate,

well i did some experiments with NHIS/NAVG/NDIA as follows........
----------------------------------------------------------------------------
! Input/Output parameters.

NRREC == -1 !for new job

LcycleRST == T

NRST == 1
NSTA == 1

NFLT == 1
NINFO == 1

! Output history, average, diagnostic files parameters.

LDEFOUT == T !for creating new files
! LDEFOUT == F !for appending existing files

NHIS == 2
NDEFHIS == 1

NTSAVG == 1
NAVG == 2
NDEFAVG == 1

NTSDIA == 1
NDIA == 2
NDEFDIA == 1
-----------------------------------------------------------------------------------------
and here is contents of log file

STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME

DEF_HIS - creating history file: /share/scratch/odc/mahesh/roms_files/output/opti_test/his_rmed16_grd_e_opt_test_0150.nc
DEF_AVG - creating average file: /share/scratch/odc/mahesh/roms_files/output/opti_test/avg_rmed16_grd_e_opt_test_0150.nc
DEF_DIAGS - creating diagnostics file: /share/scratch/odc/mahesh/roms_files/output/opti_test/dia_rmed16_grd_e_opt_test_0150.nc
WRT_RST - wrote re-start fields (Index=1,2) into time record = 0000001
150 1 01:41:20 2.531838E-03 1.830787E+04 1.830788E+04 1.331597E+16
DEF_HIS - creating history file: /share/scratch/odc/mahesh/roms_files/output/opti_test/his_rmed16_grd_e_opt_test_0151.nc
WRT_HIS - wrote history fields (Index=1,1) into time record = 0000001
DEF_AVG - creating average file: /share/scratch/odc/mahesh/roms_files/output/opti_test/avg_rmed16_grd_e_opt_test_0151.nc
WRT_AVG - wrote averaged fields into time record = 0000001
DEF_DIAGS - creating diagnostics file: /share/scratch/odc/mahesh/roms_files/output/opti_test/dia_rmed16_grd_e_opt_test_0151.nc
WRT_DIAGS - wrote diagnostics fields into time record = 0000001
WRT_RST - wrote re-start fields (Index=1,1) into time record = 0000002
151 1 01:42:00 2.557882E-03 1.830787E+04 1.830788E+04 1.331597E+16
DEF_HIS - creating history file: /share/scratch/odc/mahesh/roms_files/output/opti_test/his_rmed16_grd_e_opt_test_0152.nc
DEF_AVG - creating average file: /share/scratch/odc/mahesh/roms_files/output/opti_test/avg_rmed16_grd_e_opt_test_0152.nc
DEF_DIAGS - creating diagnostics file: /share/scratch/odc/mahesh/roms_files/output/opti_test/dia_rmed16_grd_e_opt_test_0152.nc
WRT_RST - wrote re-start fields (Index=1,2) into time record = 0000001
152 1 01:42:40 2.583917E-03 1.830787E+04 1.830787E+04 1.331597E+16
DEF_HIS - creating history file: /share/scratch/odc/mahesh/roms_files/output/opti_test/his_rmed16_grd_e_opt_test_0153.nc
WRT_HIS - wrote history fields (Index=1,1) into time record = 0000001
DEF_AVG - creating average file: /share/scratch/odc/mahesh/roms_files/output/opti_test/avg_rmed16_grd_e_opt_test_0153.nc
WRT_AVG - wrote averaged fields into time record = 0000001
DEF_DIAGS - creating diagnostics file: /share/scratch/odc/mahesh/roms_files/output/opti_test/dia_rmed16_grd_e_opt_test_0153.nc

---------------------------------------------------------------------------------------------------------------
and here is the output files......
-rw-r--r-- 1 mshinde odc 54158100 Dec 2 10:56 his_rmed16_grd_e_opt_test_0150.nc
-rw-r--r-- 1 mshinde odc 54159152 Dec 2 10:57 avg_rmed16_grd_e_opt_test_0150.nc
-rw-r--r-- 1 mshinde odc 54169540 Dec 2 10:57 dia_rmed16_grd_e_opt_test_0150.nc
-rw-r--r-- 1 mshinde odc 439745948 Dec 2 11:02 his_rmed16_grd_e_opt_test_0151.nc
-rw-r--r-- 1 mshinde odc 379160424 Dec 2 11:03 avg_rmed16_grd_e_opt_test_0151.nc
-rw-r--r-- 1 mshinde odc 1489713372 Dec 2 11:07 dia_rmed16_grd_e_opt_test_0151.nc
-rw-r--r-- 1 mshinde odc 54158100 Dec 2 11:09 his_rmed16_grd_e_opt_test_0152.nc
-rw-r--r-- 1 mshinde odc 54159152 Dec 2 11:10 avg_rmed16_grd_e_opt_test_0152.nc
-rw-r--r-- 1 mshinde odc 54169540 Dec 2 11:10 dia_rmed16_grd_e_opt_test_0152.nc
-rw-r--r-- 1 mshinde odc 439745948 Dec 2 11:17 his_rmed16_grd_e_opt_test_0153.nc
-rw-r--r-- 1 mshinde odc 379160424 Dec 2 11:17 avg_rmed16_grd_e_opt_test_0153.nc
-rw-r--r-- 1 mshinde odc 1489713372 Dec 2 11:26 dia_rmed16_grd_e_opt_test_0153.nc
-rw-r--r-- 1 mshinde odc 2132153468 Dec 2 15:39 rst_rmed16_grd_e_opt_test.nc
-----------------------------------------------------------------------------------------
(note: here i just cut & paste last part...
the data are written in alternate file.....pl see the data size...... e.g.,

-rw-r--r-- 1 mshinde odc 379160424 Dec 2 11:03 avg_rmed16_grd_e_opt_test_0151.nc
data is written in this file

-rw-r--r-- 1 mshinde odc 54159152 Dec 2 11:10 avg_rmed16_grd_e_opt_test_0152.nc
data is not written in this file


so why this is happening.......

I am really becoming with restarting the model and creating multiple files......


Please tell some suggestion....

Thanking you

[mashinde]

Dear Kate,

Can you restart say the upwelling case after a day, saving at its default frequency?

yes i did it for upwelling test.....in this case i stored all the records in rst file by changing
LcycleRST == F, so model restart ok for upweling case.


but in my application.....ROMS already completed three years of simulation.....it stops due some queue problem in HPC. In this case rst file has only recent 2 records...so i want to know how can i use this records so i can restart the model..........not from the zero.


Thanks

[kate]

Do you get the same error trying to read file 0000 when restarting at step 150? After three years? I don't know why it's trying to read file 0000, but you can perhaps find out by adding print statements to output.F or watching in the debugger. You can learn a lot with a debugger.

[mashinde]

Dear Kate,

Yes, I got the same error. The only thing I discovered is that I have to store every records in rst file which is not a good idea as it will make rst file too large.
currently, in my application rst file stored recent two records(by default) and I need to know how can I use this latest information to restart the model.

is there any document about how to use debugger. I never used it before.

Thanks

[kate]

Did you change ININAME to point to the rst file?

[mashinde]

Dear Kate,


Yes I replace ininame by rst file while making restart.
I am also trying with upwelling test.


thanks

[mashinde]

Dear kate,

yes....i did it for upwelling test.
it is creating multiple file onwards where it stops..

in this case i just put his file in ininame and restart the model and works


thanks !