Adjoint model blow up
Adjoint model blow up
Greetings,
the last few weeks I am trying to set up a stable 4DVAR setup but I think
that there is something fundamentally wrong on my approach. My application
has relative high resolution - approximately 1.7km - with 30 vertical levels.
The free runs - with or without tides - are completely stable and their solutions
are validated and acceptable. I kept the fields that are necessary for the
background estimation error (5 years) of surface forcing, model, initial
and boundary conditions. I tried the Incremental strong constrain (IS4DAVR)
and dual (PSAS) but in both cases the model blows up at the first step of
of adjoint model (attached log files). My assimilation window is 5 days. I tried
with several timesteps using smaller every time but the result was the same.
My initial thought was that the problem was sea surface height but the model
blows up even if i use only SST observations. To be honest I am struggling to
understand what is wrong in my approach and I am running out of ideas.
The data for observations file comes from AVISO (grdidded product) for SLA in which
I added MDT to be consistent with model's zeta, SST from OSTIA and T/S profiles
from UK Met Office Hadley Centre.
Any help or suggestion will be highly appreciated.
Thanks in advance
Kind Regards
Giannis
the last few weeks I am trying to set up a stable 4DVAR setup but I think
that there is something fundamentally wrong on my approach. My application
has relative high resolution - approximately 1.7km - with 30 vertical levels.
The free runs - with or without tides - are completely stable and their solutions
are validated and acceptable. I kept the fields that are necessary for the
background estimation error (5 years) of surface forcing, model, initial
and boundary conditions. I tried the Incremental strong constrain (IS4DAVR)
and dual (PSAS) but in both cases the model blows up at the first step of
of adjoint model (attached log files). My assimilation window is 5 days. I tried
with several timesteps using smaller every time but the result was the same.
My initial thought was that the problem was sea surface height but the model
blows up even if i use only SST observations. To be honest I am struggling to
understand what is wrong in my approach and I am running out of ideas.
The data for observations file comes from AVISO (grdidded product) for SLA in which
I added MDT to be consistent with model's zeta, SST from OSTIA and T/S profiles
from UK Met Office Hadley Centre.
Any help or suggestion will be highly appreciated.
Thanks in advance
Kind Regards
Giannis
- Attachments
-
- psas_30_sla_mdt.txt
- (3.11 MiB) Downloaded 610 times
-
- is4dvar_30sec.txt
- (3.11 MiB) Downloaded 540 times
- AbhiPasula
- Posts: 9
- Joined: Thu Dec 12, 2019 4:10 pm
- Location: Indian Institute of Science
Re: Adjoint model blow up
Last few days I am setting up a 4DVAR setup.
My NLM free run is working fine. With same CPP options for W4DVAR the prior model itself is blowing up.
I tried with different initial conditions also still, the model is blowing up at the same time step.
My doubt is the NLM prior model configure is same for a free run also, why the model is blowing up.?
Here I am attaching the CPP flags that i used to configure the model. and log files of the free run and the 4dvar run.
My NLM free run is working fine. With same CPP options for W4DVAR the prior model itself is blowing up.
I tried with different initial conditions also still, the model is blowing up at the same time step.
My doubt is the NLM prior model configure is same for a free run also, why the model is blowing up.?
Here I am attaching the CPP flags that i used to configure the model. and log files of the free run and the 4dvar run.
- Attachments
-
- wc13.h
- (5.84 KiB) Downloaded 543 times
- AbhiPasula
- Posts: 9
- Joined: Thu Dec 12, 2019 4:10 pm
- Location: Indian Institute of Science
Re: Adjoint model blow up
I tried different test cases, still model is blowing up for W4DVAR.
but...
the error I am getting every time is:-
------------------------
Found Error: 08 Line: 51 Source: ROMS/Utility/close_io.F
Found Error: 08 Line: 333 Source: ROMS/Utility/close_io.F
------------------------
The dynamic memory used is almost greater than 364000MB for every run.
So, i tried a test case of NTIMES -3 and DT- 150 with a single process. Still, the model got blowup with the above error and used dynamic memory of 363409 MB.
my model resolution is 372x454x70
My doubts are:
1) What is the minimum memory requirement for running a 4DVAR..?
2) For a one day run, the Nonlinear model is running fine with same setup, but for the W4DVAR model Prior model itself is blowing with the above error without any CFL violation.
3) For hour simulations, the prior model is running fine but adjoint model is blowing up with the same error 08.
Is the ROMS blowing up due to lack of computational resources or am I making any errors in the setup shown in the below log file?
but...
the error I am getting every time is:-
------------------------
Found Error: 08 Line: 51 Source: ROMS/Utility/close_io.F
Found Error: 08 Line: 333 Source: ROMS/Utility/close_io.F
------------------------
The dynamic memory used is almost greater than 364000MB for every run.
So, i tried a test case of NTIMES -3 and DT- 150 with a single process. Still, the model got blowup with the above error and used dynamic memory of 363409 MB.
my model resolution is 372x454x70
My doubts are:
1) What is the minimum memory requirement for running a 4DVAR..?
2) For a one day run, the Nonlinear model is running fine with same setup, but for the W4DVAR model Prior model itself is blowing with the above error without any CFL violation.
3) For hour simulations, the prior model is running fine but adjoint model is blowing up with the same error 08.
Is the ROMS blowing up due to lack of computational resources or am I making any errors in the setup shown in the below log file?
- Attachments
-
- log.txt
- (175.92 KiB) Downloaded 577 times
Re: Adjoint model blow up
Greetings
after several weeks i was able to solve the blowup at the first
step of the adjoint model. To be honest I had to redesign
completely my approach in order to run - at some point - the
PSAS model. At the moment I am assimilating only SST. The model
again blows up at the first inner loop. I am getting weird
values for kinetic, potential and overall energy starting from zero
to Infinity as you can see below.
I assume that my observation file is ok because I have check it and created several times
but I cannot say that I am completely confident about that after so many blowups.
Is my observation file wrong or I should look somewhere else?
Any help or suggestion is welcome
Kind Regards
Giannis
after several weeks i was able to solve the blowup at the first
step of the adjoint model. To be honest I had to redesign
completely my approach in order to run - at some point - the
PSAS model. At the moment I am assimilating only SST. The model
again blows up at the first inner loop. I am getting weird
values for kinetic, potential and overall energy starting from zero
to Infinity as you can see below.
Code: Select all
Number of State Observations Processed: ObsTime = 15199.5000, 2010-01-02 12:00:00.00
Variable IstrObs IendObs Count Rejected
temp 770 1538 769 41
Total 769 41
Obs Tally 769 41
AD_HTOBS - Computed adjoint observations forcing, 2010-01-02 12:00:00.00
(Observation records = 0000770 - 0001538, iic = 0002161)
2160 2010-01-02 12:00:00.00 0.000000E+00 0.000000E+00 0.000000E+00 1.669778E+13
2159 2010-01-02 11:59:00.00 2.543052E-13 -6.874194E-12 -6.619889E-12 1.669778E+13
2158 2010-01-02 11:58:00.00 1.897037E-13 -5.423539E-12 -5.233836E-12 1.669778E+13
2157 2010-01-02 11:57:00.00 1.536239E-13 8.617824E-14 2.398021E-13 1.669778E+13
2156 2010-01-02 11:56:00.00 1.326718E-13 1.693501E-09 1.693634E-09 1.669778E+13
2155 2010-01-02 11:55:00.00 1.204015E-13 6.409770E-07 6.409771E-07 1.669778E+13
2154 2010-01-02 11:54:00.00 3.445896E-12 2.417944E-04 2.417944E-04 1.669778E+13
2153 2010-01-02 11:53:00.00 2.442746E-08 9.120887E-02 9.120889E-02 1.669778E+13
2152 2010-01-02 11:52:00.00 1.789652E-04 3.440587E+01 3.440605E+01 1.669778E+13
2151 2010-01-02 11:51:00.00 1.311176E+00 1.297859E+04 1.297990E+04 1.669778E+13
2150 2010-01-02 11:50:00.00 9.606240E+03 4.895783E+06 4.905389E+06 1.669778E+13
2149 2010-01-02 11:49:00.00 7.037941E+07 1.846788E+09 1.917167E+09 1.669778E+13
2148 2010-01-02 11:48:00.00 5.156297E+11 6.966455E+11 1.212275E+12 1.669778E+13
2147 2010-01-02 11:47:00.00 3.777723E+15 2.627887E+14 4.040512E+15 1.669778E+13
2146 2010-01-02 11:46:00.00 2.767721E+19 9.912917E+16 2.777634E+19 1.669778E+13
2145 2010-01-02 11:45:00.00 2.027751E+23 3.739352E+19 2.028125E+23 1.669778E+13
2144 2010-01-02 11:44:00.00 1.485617E+27 1.410559E+22 1.485631E+27 1.669778E+13
2143 2010-01-02 11:43:00.00 1.088426E+31 5.320910E+24 1.088427E+31 1.669778E+13
2142 2010-01-02 11:42:00.00 7.974277E+34 2.007154E+27 7.974277E+34 1.669778E+13
2141 2010-01-02 11:41:00.00 5.842296E+38 7.571390E+29 5.842296E+38 1.669778E+13
2140 2010-01-02 11:40:00.00 4.280315E+42 2.856080E+32 4.280315E+42 1.669778E+13
2139 2010-01-02 11:39:00.00 3.135942E+46 1.077371E+35 3.135942E+46 1.669778E+13
2138 2010-01-02 11:38:00.00 2.297525E+50 4.064060E+37 2.297525E+50 1.669778E+13
2137 2010-01-02 11:37:00.00 1.683265E+54 1.533045E+40 1.683265E+54 1.669778E+13
2136 2010-01-02 11:36:00.00 1.233232E+58 5.782954E+42 1.233232E+58 1.669778E+13
2135 2010-01-02 11:35:00.00 9.035183E+61 2.181446E+45 9.035183E+61 1.669778E+13
2134 2010-01-02 11:34:00.00 6.619561E+65 8.228854E+47 6.619561E+65 1.669778E+13
2133 2010-01-02 11:33:00.00 4.849773E+69 3.104089E+50 4.849773E+69 1.669778E+13
2132 2010-01-02 11:32:00.00 3.553151E+73 1.170925E+53 3.553151E+73 1.669778E+13
2131 2010-01-02 11:31:00.00 2.603190E+77 4.416964E+55 2.603190E+77 1.669778E+13
2130 2010-01-02 11:30:00.00 1.907208E+81 1.666168E+58 1.907208E+81 1.669778E+13
2129 2010-01-02 11:29:00.00 1.397302E+85 6.285119E+60 1.397302E+85 1.669778E+13
2128 2010-01-02 11:28:00.00 1.023723E+89 2.370873E+63 1.023723E+89 1.669778E+13
2127 2010-01-02 11:27:00.00 7.500235E+92 8.943410E+65 7.500235E+92 1.669778E+13
2126 2010-01-02 11:26:00.00 5.494992E+96 3.373634E+68 5.494992E+96 1.669778E+13
2125 2010-01-02 11:25:00.00 4.025866+100 1.272603E+71 4.025866+100 1.669778E+13
2124 2010-01-02 11:24:00.00 2.949522+104 4.800513E+73 2.949522+104 1.669778E+13
2123 2010-01-02 11:23:00.00 2.160945+108 1.810850E+76 2.160945+108 1.669778E+13
2122 2010-01-02 11:22:00.00 1.583201+112 6.830890E+78 1.583201+112 1.669778E+13
2121 2010-01-02 11:21:00.00 1.159921+116 2.576749E+81 1.159921+116 1.669778E+13
2120 2010-01-02 11:20:00.00 8.498074+119 9.720014E+83 8.498074+119 1.669778E+13
2119 2010-01-02 11:19:00.00 6.226052+123 3.666585E+86 6.226052+123 1.669778E+13
2118 2010-01-02 11:18:00.00 4.561472+127 1.383109E+89 4.561472+127 1.669778E+13
2117 2010-01-02 11:17:00.00 3.341929+131 5.217367E+91 3.341929+131 1.669778E+13
2116 2010-01-02 11:16:00.00 2.448440+135 1.968096E+94 2.448440+135 1.669778E+13
2115 2010-01-02 11:15:00.00 1.793832+139 7.424052E+96 1.793832+139 1.669778E+13
2114 2010-01-02 11:14:00.00 1.314238+143 2.800502E+99 1.314238+143 1.669778E+13
2113 2010-01-02 11:13:00.00 9.628667+146 1.056406+102 9.628667+146 1.669778E+13
2112 2010-01-02 11:12:00.00 7.054373+150 3.984974+104 7.054373+150 1.669778E+13
2111 2010-01-02 11:11:00.00 5.168335+154 1.503212+107 5.168335+154 1.669778E+13
2110 2010-01-02 11:10:00.00 3.786543+158 5.670419+109 3.786543+158 1.669778E+13
2109 2010-01-02 11:09:00.00 2.774183+162 2.138996+112 2.774183+162 1.669778E+13
2108 2010-01-02 11:08:00.00 2.032485+166 8.068723+114 2.032485+166 1.669778E+13
2107 2010-01-02 11:07:00.00 1.489085+170 3.043684+117 1.489085+170 1.669778E+13
2106 2010-01-02 11:06:00.00 1.090968+174 1.148139+120 1.090968+174 1.669778E+13
2105 2010-01-02 11:05:00.00 7.992894+177 4.331011+122 7.992894+177 1.669778E+13
2104 2010-01-02 11:04:00.00 5.855936+181 1.633744+125 5.855936+181 1.669778E+13
2103 2010-01-02 11:03:00.00 4.290309+185 6.162812+127 4.290309+185 1.669778E+13
2102 2010-01-02 11:02:00.00 3.143263+189 2.324737+130 3.143263+189 1.669778E+13
2101 2010-01-02 11:01:00.00 2.302889+193 8.769374+132 2.302889+193 1.669778E+13
2100 2010-01-02 11:00:00.00 1.687195+197 3.307984+135 1.687195+197 1.669778E+13
2099 2010-01-02 10:59:00.00 1.236111+201 1.247838+138 1.236111+201 1.669778E+13
2098 2010-01-02 10:58:00.00 9.056278+204 4.707096+140 9.056278+204 1.669778E+13
2097 2010-01-02 10:57:00.00 6.635016+208 1.775611+143 6.635016+208 1.669778E+13
2096 2010-01-02 10:56:00.00 4.861096+212 6.697962+145 4.861096+212 1.669778E+13
2095 2010-01-02 10:55:00.00 3.561447+216 2.526606+148 3.561447+216 1.669778E+13
2094 2010-01-02 10:54:00.00 2.609268+220 9.530866+150 2.609268+220 1.669778E+13
2093 2010-01-02 10:53:00.00 1.911661+224 3.595234+153 1.911661+224 1.669778E+13
2092 2010-01-02 10:52:00.00 1.400565+228 1.356195+156 1.400565+228 1.669778E+13
2091 2010-01-02 10:51:00.00 1.026113+232 5.115838+158 1.026113+232 1.669778E+13
2090 2010-01-02 10:50:00.00 7.517745+235 1.929797+161 7.517745+235 1.669778E+13
2089 2010-01-02 10:49:00.00 5.507821+239 7.279582+163 5.507821+239 1.669778E+13
2088 2010-01-02 10:48:00.00 4.035265+243 2.746005+166 4.035265+243 1.669778E+13
2087 2010-01-02 10:47:00.00 2.956408+247 1.035848+169 2.956408+247 1.669778E+13
2086 2010-01-02 10:46:00.00 2.165991+251 3.907428+171 2.165991+251 1.669778E+13
2085 2010-01-02 10:45:00.00 1.586897+255 1.473960+174 1.586897+255 1.669778E+13
2084 2010-01-02 10:44:00.00 1.162629+259 5.560074+176 1.162629+259 1.669778E+13
2083 2010-01-02 10:43:00.00 8.517915+262 2.097372+179 8.517915+262 1.669778E+13
2082 2010-01-02 10:42:00.00 6.240588+266 7.911708+181 6.240588+266 1.669778E+13
2081 2010-01-02 10:41:00.00 4.572121+270 2.984455+184 4.572121+270 1.669778E+13
2080 2010-01-02 10:40:00.00 3.349731+274 1.125796+187 3.349731+274 1.669778E+13
2079 2010-01-02 10:39:00.00 2.454156+278 4.246731+189 2.454156+278 1.669778E+13
2078 2010-01-02 10:38:00.00 1.798020+282 1.601952+192 1.798020+282 1.669778E+13
2077 2010-01-02 10:37:00.00 1.317306+286 6.042886+194 1.317306+286 1.669778E+13
2076 2010-01-02 10:36:00.00 9.651147+289 2.279498+197 9.651147+289 1.669778E+13
2075 2010-01-02 10:35:00.00 7.070843+293 8.598724+199 7.070843+293 1.669778E+13
2074 2010-01-02 10:34:00.00 NaN 3.243612+202 NaN 1.669778E+13
Found Error: 01 Line: 662 Source: ROMS/Adjoint/ad_main3d.F
Found Error: 01 Line: 884 Source: ROMS/Drivers/w4dpsas_ocean.h
but I cannot say that I am completely confident about that after so many blowups.
Is my observation file wrong or I should look somewhere else?
Any help or suggestion is welcome
Kind Regards
Giannis
Re: Adjoint model blow up
Greetings once more
After several test runs with small to very small dt (30sec to 1 sec and Maximum barotropic Courant Number = 3.56216612E-03)
the problem persists.
Any help or suggestion is more than welcome.
Kind Regards
Giannis
After several test runs with small to very small dt (30sec to 1 sec and Maximum barotropic Courant Number = 3.56216612E-03)
the problem persists.
Any help or suggestion is more than welcome.
Kind Regards
Giannis
Re: Adjoint model blow up
Not sure if it will help with your problem but I suggest you to try increase the number of inner loops and to decrease the horizontal decorrelation length scales.
The STD values for ubar and vbar also look strange. You should check them.
The STD values for ubar and vbar also look strange. You should check them.
- arango
- Site Admin
- Posts: 1367
- Joined: Wed Feb 26, 2003 4:41 pm
- Location: DMCS, Rutgers University
- Contact:
Re: Adjoint model blow up
In our experience, it is challenging for either the tangent linear (TLM) or adjoint models (ADM) to blow-up in the first iteration of the inner loops. Something must be wrong in your step-up. Both models are linearized from the nonlinear model (NLM) state trajectory (prior). The TLM and ADM start from rest (zero state values). The TLM interpolates the linearized state solution at the observation locations. The ADM is forced by the transpose of that observation operator. That's what it is making the ADM to blow-up. It hasn't applied the error covariance model either entered into the minimization solver.
I recommend a few things:
I recommend a few things:
- Run the 4D-Var tutorial documented in WikiROMS
- Run either IS4DVAR or preferable W4DPSAS. I won't recommend using the representer approach W4DVAR
- Make sure that the advection scheme for the TLM and ADM is something simpler and not too highly nonlinear. Recall that the NLM can have different advection schemes.
- Be conservative with the CPP options. Avoid sophisticated algorithms for mixing. Maybe you need to provide your CPP options for us to check
- Examine and plot your observations. Chech output NetCDF file: *_mod.nc
- Start simple and then add complexity is always the right thing to do.
- Check our papers on 4D-Var to see how we configure our applications
Re: Adjoint model blow up
I would suggest
use
Cha for zeta (no Che)
Fla for 2d momentum (no Sch)
Cla for all 3d value including Nonlinear model
use also
Same mixing/viscosity coefficient for NLM ADM.
I saw zero bottom drag coefficient in your ocean.in. Is there a reason for 0 ?
I don't know the reason but I suspect that the Shchepetkin and Chapman explicit option do not go well with 4dvar.
You need to use identical clamped condition for your 3 d value (no radiation condition for 4dvar)
-JH
use
Cha for zeta (no Che)
Fla for 2d momentum (no Sch)
Cla for all 3d value including Nonlinear model
use also
Same mixing/viscosity coefficient for NLM ADM.
I saw zero bottom drag coefficient in your ocean.in. Is there a reason for 0 ?
I don't know the reason but I suspect that the Shchepetkin and Chapman explicit option do not go well with 4dvar.
You need to use identical clamped condition for your 3 d value (no radiation condition for 4dvar)
-JH
Joonho Lee
Re: Adjoint model blow up
Greetings once more
Raquel, Hernan and Joonho I am more than grateful for your
suggestions. Finally I was able to run without any problem
the 4DPSAS model . Raquel you had a point
about my decorrelation scales. I found some literature about
using variograms in order to estimate decorrelation scales
and I adjust them accordingly. I am not sure if this was the
main problem but you got a point. Hernan you was right about
my CPP file. I was using VISC_GRID, DIFF_GRID and
WIND_MINUS_CURRENT keys in my setup. After removing them
and using a moderate dt and ndtfast - 60 seconds and 40 respectively
the model run smoothly all 50 inner loops. Now I am starting
to plot and analyse the results in order first to find the optimal
number of inner loops for my setup from the cost function. The
next step is to use SLA observation and if everything is OK to
use CTD casts. Joonho, I switch from Chapman explicit and
Shchepetkin to Chapman implicit and Flather. Linear and
quadratic drags were set equal to zero because I am using
logarithmic bottom drag (5*10^-4). Also set Clamped boundary
conditions for 3D fields.
Once more many thank you all for your suggestions and your willingness
to help.
Kind Regards
Giannis
Raquel, Hernan and Joonho I am more than grateful for your
suggestions. Finally I was able to run without any problem
the 4DPSAS model . Raquel you had a point
about my decorrelation scales. I found some literature about
using variograms in order to estimate decorrelation scales
and I adjust them accordingly. I am not sure if this was the
main problem but you got a point. Hernan you was right about
my CPP file. I was using VISC_GRID, DIFF_GRID and
WIND_MINUS_CURRENT keys in my setup. After removing them
and using a moderate dt and ndtfast - 60 seconds and 40 respectively
the model run smoothly all 50 inner loops. Now I am starting
to plot and analyse the results in order first to find the optimal
number of inner loops for my setup from the cost function. The
next step is to use SLA observation and if everything is OK to
use CTD casts. Joonho, I switch from Chapman explicit and
Shchepetkin to Chapman implicit and Flather. Linear and
quadratic drags were set equal to zero because I am using
logarithmic bottom drag (5*10^-4). Also set Clamped boundary
conditions for 3D fields.
Once more many thank you all for your suggestions and your willingness
to help.
Kind Regards
Giannis
-
- Posts: 3
- Joined: Mon Nov 07, 2016 7:33 pm
- Location: Institute of Oceanology, Chinese Academy of Scienc
Re: Adjoint model blow up
Dear all,
I am condcting a backward tracking using the Adjoint model. I have run the FWD model successfully. However, the similar blow up at the first step of the adjoint model.
1. as suggested by arango, I have run the 4D-Var tutorial documented in WikiROMS succefully;
2. as sueggested by susonic, we use
Cha for zeta (no Che)
Fla for 2d momentum (no Sch)
Cla for all 3d value including Nonlinear model
Same mixing/viscosity coefficient for NLM ADM.
3. according to the answer by ymamoutos, I have removed the VISC_GRID, DIFF_GRID in my setup.
The blow up still occurs as you can see below
It is also very wired that we only add the passive tracer dye01 in the place we are interested in the initial adjont file, but the temperature and salinity get Infinity in the adjoint model.
Here I am attaching the output file containing the model setup.
At the moment we have no idea what is going wrong here. Any suggestion is welcome
Kind Regards
Lingjing
I am condcting a backward tracking using the Adjoint model. I have run the FWD model successfully. However, the similar blow up at the first step of the adjoint model.
1. as suggested by arango, I have run the 4D-Var tutorial documented in WikiROMS succefully;
2. as sueggested by susonic, we use
Cha for zeta (no Che)
Fla for 2d momentum (no Sch)
Cla for all 3d value including Nonlinear model
Same mixing/viscosity coefficient for NLM ADM.
3. according to the answer by ymamoutos, I have removed the VISC_GRID, DIFF_GRID in my setup.
The blow up still occurs as you can see below
Code: Select all
TIME-STEP YYYY-MM-DD hh:mm:ss.ss KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
14400 0001-02-30 00:00:00.00 0.000000E+00 0.000000E+00 0.000000E+00 5.910055E+15
14390 0001-02-29 23:00:00.00 3.510907E-03 2.184294E-05 3.532750E-03 5.910055E+15
14380 0001-02-29 22:00:00.00 1.411261E-02 8.362980E-05 1.419624E-02 5.910055E+15
14370 0001-02-29 21:00:00.00 1.186302E+15 9.436077E+05 1.186302E+15 5.910055E+15
14360 0001-02-29 20:00:00.00 1.244035E+56 8.586994E+32 1.244035E+56 5.910055E+15
14350 0001-02-29 19:00:00.00 3.495551E+97 1.667387E+60 3.495551E+97 5.910055E+15
14340 0001-02-29 18:00:00.00 7.237327+138 6.757092E+87 7.237327+138 5.910055E+15
14330 0001-02-29 17:00:00.00 9.786195+179 3.613735+115 9.786195+179 5.910055E+15
14320 0001-02-29 16:00:00.00 7.847325+220 1.065016+143 7.847325+220 5.910055E+15
14310 0001-02-29 15:00:00.00 3.538782+261 1.793766+170 3.538782+261 5.910055E+15
14300 0001-02-29 14:00:00.00 NaN 1.843634+197 NaN 5.910055E+15
Found Error: 01 Line: 671 Source: ROMS/Adjoint/ad_main3d.F
Found Error: 01 Line: 260 Source: ROMS/Drivers/ad_ocean.h
Elapsed wall CPU time for each process (seconds):
Here I am attaching the output file containing the model setup.
At the moment we have no idea what is going wrong here. Any suggestion is welcome
Kind Regards
Lingjing
- Attachments
-
- output.dat
- (117.1 KiB) Downloaded 495 times
Re: Adjoint model blow up
Greetings
You are using nested boundary conditions in NLM model
but it seems you didn't enable the NESTING cpp option.
Also why you are using mixed radiation conditions in
NLM model? In general the choice of the boundary conditions
in the NLM model I think is completely inconsistent.
Giannis
You are using nested boundary conditions in NLM model
but it seems you didn't enable the NESTING cpp option.
Also why you are using mixed radiation conditions in
NLM model? In general the choice of the boundary conditions
in the NLM model I think is completely inconsistent.
Giannis
-
- Posts: 3
- Joined: Mon Nov 07, 2016 7:33 pm
- Location: Institute of Oceanology, Chinese Academy of Scienc
Re: Adjoint model blow up
Hi, Giannis
thank you for your reply.
The new output file is attached.
we use the mixed radiation conditions in the NLM model as there are both inflow and outflow at the boundary.
The NLM model has been run succefully. As the adjoint model can has different boundary conditions, I wonder weather the setup in the NLM model would influence the ADM model. I also tried to change the boundary conditions in the NLM model to those in the ADM model (cha.. fla.. cla..) both in the FWD and ADM model. The ADM model still blows up.
Look forward to more suggestions and further discussion.
Lingjing
thank you for your reply.
In fact, we do not use the nested model in our NLM. It was a misleading presentation. what we used in the NLM model areymamoutos wrote: ↑Tue Jul 14, 2020 12:19 pm Greetings
You are using nested boundary conditions in NLM model
but it seems you didn't enable the NESTING cpp option.
Also why you are using mixed radiation conditions in
NLM model? In general the choice of the boundary conditions
in the NLM model I think is completely inconsistent.
Giannis
Code: Select all
Lateral Boundary Conditions: NLM
============================
Variable Grid West Edge South Edge East Edge North Edge
--------- ---- ---------- ---------- ---------- ----------
zeta 1 Chapman Exp Chapman Exp Chapman Exp Chapman Exp
ubar 1 Shchepetkin Shchepetkin Shchepetkin Shchepetkin
vbar 1 Shchepetkin Shchepetkin Shchepetkin Shchepetkin
u 1 Rad + Nud Rad + Nud Rad + Nud Rad + Nud
v 1 Rad + Nud Rad + Nud Rad + Nud Rad + Nud
temp 1 Rad + Nud Rad + Nud Rad + Nud Rad + Nud
salt 1 Rad + Nud Rad + Nud Rad + Nud Rad + Nud
dye_01 1 Clamped Clamped Clamped Clamped
we use the mixed radiation conditions in the NLM model as there are both inflow and outflow at the boundary.
The NLM model has been run succefully. As the adjoint model can has different boundary conditions, I wonder weather the setup in the NLM model would influence the ADM model. I also tried to change the boundary conditions in the NLM model to those in the ADM model (cha.. fla.. cla..) both in the FWD and ADM model. The ADM model still blows up.
Look forward to more suggestions and further discussion.
Lingjing
- Attachments
-
- output_new.dat
- (117.11 KiB) Downloaded 519 times
Re: Adjoint model blow up
Greetings once more
Try to run using the same options
for boundary conditions, i.e. set
NLM obc to be the same with adjoint.
Another suggestion is to unset the
relaxation options (QCORRECTION,
SRELAXATION) and run the nonlinear
model with BULK_FLUXES enable.
Also which of the available
4DVAR schemes you are using? It's
not clear from the logs you posted.
Giannis
Try to run using the same options
for boundary conditions, i.e. set
NLM obc to be the same with adjoint.
Another suggestion is to unset the
relaxation options (QCORRECTION,
SRELAXATION) and run the nonlinear
model with BULK_FLUXES enable.
Also which of the available
4DVAR schemes you are using? It's
not clear from the logs you posted.
Giannis
-
- Posts: 3
- Joined: Mon Nov 07, 2016 7:33 pm
- Location: Institute of Oceanology, Chinese Academy of Scienc
Re: Adjoint model blow up
Hi, Giannisymamoutos wrote: ↑Wed Jul 15, 2020 9:09 am Greetings once more
Try to run using the same options
for boundary conditions, i.e. set
NLM obc to be the same with adjoint.
Another suggestion is to unset the
relaxation options (QCORRECTION,
SRELAXATION) and run the nonlinear
model with BULK_FLUXES enable.
Also which of the available
4DVAR schemes you are using? It's
not clear from the logs you posted.
Giannis
Thank you very much for the suggestions.
In the past few days, I have tried to using the same boundary conditions for the NLM and adjoint model, and carefully check the CPP setup. Finally, I was able to run the adjoint model succefully.
By the way, I did not use the 4DVAR schemes. I just run the I4DVAR in WC13 as a test.
Thanks again for your help.
Cheers,
Lingjing