I'm trying to make sense of the time profiling output for an MPI run of ROMS, with example output below. Is the elapsed time for message passing included in the elapsed time for the non-linear model? For example, does the time elapsed shown for the Model 2D kernel also include the time for 2D halo exchanges? Or do you have to add Model 2D kernel time and 2D halo exchange time to get the total time for 2D stepping?
Nonlinear model elapsed time profile:
Initialization ................................... 478.991 ( 0.1024 %)
Reading of input data ............................ 295.288 ( 0.0631 %)
Processing of input data ......................... 24366.475 ( 5.2101 %)
Processing of output time averaged data .......... 2034.842 ( 0.4351 %)
Computation of vertical boundary conditions ...... 374.829 ( 0.0801 %)
Computation of global information integrals ...... 5998.291 ( 1.2826 %)
Writing of output data ........................... 15187.017 ( 3.2473 %)
Model 2D kernel .................................. 221638.845 (47.3913 %)
2D/3D coupling, vertical metrics ................. 6819.473 ( 1.4582 %)
Omega vertical velocity .......................... 14814.303 ( 3.1676 %)
Equation of state for seawater ................... 14721.860 ( 3.1479 %)
GLS vertical mixing parameterization ............. 74444.107 (15.9178 %)
3D equations right-side terms .................... 3313.498 ( 0.7085 %)
3D equations predictor step ...................... 12071.718 ( 2.5812 %)
Pressure gradient ................................ 4012.504 ( 0.8580 %)
Harmonic mixing of tracers, geopotentials ........ 4142.211 ( 0.8857 %)
Harmonic stress tensor, S-surfaces ............... 1962.384 ( 0.4196 %)
Corrector time-step for 3D momentum .............. 33187.006 ( 7.0961 %)
Corrector time-step for tracers .................. 16517.137 ( 3.5317 %)
Total: 456380.778 97.5844
Nonlinear model message Passage profile:
Message Passage: 2D halo exchanges ............... 198439.255 (42.4308 %)
Message Passage: 3D halo exchanges ............... 79077.268 (16.9085 %)
Message Passage: 4D halo exchanges ............... 29843.343 ( 6.3812 %)
Message Passage: data broadcast .................. 11739.061 ( 2.5101 %)
Message Passage: data reduction .................. 5144.240 ( 1.1000 %)
Message Passage: data gathering .................. 3363.179 ( 0.7191 %)
Message Passage: data scattering.................. 408.240 ( 0.0873 %)
Total: 328014.586 70.1369
All percentages are with respect to total time = 467677.894
MPI time profiling results?
Re: MPI time profiling results?
I was also trying to understand this question. The profile breakdown looks similar (see below). Is this normal for a ROMS run? The run is on 64 processors and uses serial I/O. In addition, is it possible run times are being hampered by the serial I/O/what kind of speed up would be expected by switching to parallel I/O?
Nonlinear model elapsed time profile:
Initialization ................................... 13643.532 ( 1.2258 %)
OI data assimilation ............................. 11.378 ( 0.0010 %)
Reading of input data ............................ 22709.763 ( 2.0403 %)
Processing of input data ......................... 80238.841 ( 7.2088 %)
Computation of vertical boundary conditions ...... 1864.843 ( 0.1675 %)
Computation of global information integrals ...... 358.941 ( 0.0322 %)
Writing of output data ........................... 16781.614 ( 1.5077 %)
Model 2D kernel .................................. 483542.947 (43.4424 %)
2D/3D coupling, vertical metrics ................. 9626.206 ( 0.8648 %)
Omega vertical velocity .......................... 11943.059 ( 1.0730 %)
Equation of state for seawater ................... 16972.353 ( 1.5248 %)
Atmosphere-Ocean bulk flux parameterization ...... 8381.848 ( 0.7530 %)
My2.5 vertical mixing parameterization ........... 138098.676 (12.4070 %)
3D equations right-side terms .................... 12738.116 ( 1.1444 %)
3D equations predictor step ...................... 49374.785 ( 4.4359 %)
Pressure gradient ................................ 3554.001 ( 0.3193 %)
Harmonic mixing of tracers, isopycnals ........... 13814.746 ( 1.2411 %)
Harmonic stress tensor, geopotentials ............ 29147.193 ( 2.6186 %)
Corrector time-step for 3D momentum .............. 24644.077 ( 2.2141 %)
Corrector time-step for tracers .................. 161828.397 (14.5390 %)
Total: 1099275.315 98.7609
Nonlinear model message Passage profile:
Message Passage: 2D halo exchanges ............... 523336.329 (47.0175 %)
Message Passage: 3D halo exchanges ............... 159054.711 (14.2898 %)
Message Passage: 4D halo exchanges ............... 109670.134 ( 9.8530 %)
Message Passage: data broadcast .................. 18941.984 ( 1.7018 %)
Message Passage: data reduction .................. 266.122 ( 0.0239 %)
Message Passage: data gathering .................. 1585.258 ( 0.1424 %)
Message Passage: data scattering.................. 23399.430 ( 2.1022 %)
Message Passage: point data gathering ............ 22.878 ( 0.0021 %)
Total: 836276.845 75.1326
All percentages are with respect to total time = 1113067.298
Nonlinear model elapsed time profile:
Initialization ................................... 13643.532 ( 1.2258 %)
OI data assimilation ............................. 11.378 ( 0.0010 %)
Reading of input data ............................ 22709.763 ( 2.0403 %)
Processing of input data ......................... 80238.841 ( 7.2088 %)
Computation of vertical boundary conditions ...... 1864.843 ( 0.1675 %)
Computation of global information integrals ...... 358.941 ( 0.0322 %)
Writing of output data ........................... 16781.614 ( 1.5077 %)
Model 2D kernel .................................. 483542.947 (43.4424 %)
2D/3D coupling, vertical metrics ................. 9626.206 ( 0.8648 %)
Omega vertical velocity .......................... 11943.059 ( 1.0730 %)
Equation of state for seawater ................... 16972.353 ( 1.5248 %)
Atmosphere-Ocean bulk flux parameterization ...... 8381.848 ( 0.7530 %)
My2.5 vertical mixing parameterization ........... 138098.676 (12.4070 %)
3D equations right-side terms .................... 12738.116 ( 1.1444 %)
3D equations predictor step ...................... 49374.785 ( 4.4359 %)
Pressure gradient ................................ 3554.001 ( 0.3193 %)
Harmonic mixing of tracers, isopycnals ........... 13814.746 ( 1.2411 %)
Harmonic stress tensor, geopotentials ............ 29147.193 ( 2.6186 %)
Corrector time-step for 3D momentum .............. 24644.077 ( 2.2141 %)
Corrector time-step for tracers .................. 161828.397 (14.5390 %)
Total: 1099275.315 98.7609
Nonlinear model message Passage profile:
Message Passage: 2D halo exchanges ............... 523336.329 (47.0175 %)
Message Passage: 3D halo exchanges ............... 159054.711 (14.2898 %)
Message Passage: 4D halo exchanges ............... 109670.134 ( 9.8530 %)
Message Passage: data broadcast .................. 18941.984 ( 1.7018 %)
Message Passage: data reduction .................. 266.122 ( 0.0239 %)
Message Passage: data gathering .................. 1585.258 ( 0.1424 %)
Message Passage: data scattering.................. 23399.430 ( 2.1022 %)
Message Passage: point data gathering ............ 22.878 ( 0.0021 %)
Total: 836276.845 75.1326
All percentages are with respect to total time = 1113067.298