Articles | Volume 7, issue 2
https://doi.org/10.5194/ascmo-7-53-2021
© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/ascmo-7-53-2021
© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
23 Sep 2021
| 23 Sep 2021
Forecast score distributions with imperfect observations
Julie Bessac
CORRESPONDING AUTHOR
Mathematics and Computer Science Division, Argonne National Laboratory, Lemont, IL 60439, USA
Philippe Naveau
Laboratoire de Sciences du Climat et de l'Environnement, IPSL-CNRS, Gif-sur-Yvette, 91191, France
Related authors
Qiuyi Wu, Julie Bessac, Whitney Huang, Jiali Wang, and Rao Kotamarthi
Adv. Stat. Clim. Meteorol. Oceanogr., 8, 205–224, https://doi.org/10.5194/ascmo-8-205-2022, https://doi.org/10.5194/ascmo-8-205-2022, 2022
Short summary
Short summary
We study wind conditions and their potential future changes across the U.S. via a statistical conditional framework. We conclude that changes between historical and future wind directions are small, but wind speeds are generally weakened in the projected period, with some locations being intensified. Moreover, winter wind speeds are projected to decrease in the northwest, Colorado, and the northern Great Plains (GP), while summer wind speeds over the southern GP slightly increase in the future.
Yoann Robin, Mathieu Vrac, Aurélien Ribes, Occitane Barbaux, and Philippe Naveau
EGUsphere, https://doi.org/10.5194/egusphere-2025-1121, https://doi.org/10.5194/egusphere-2025-1121, 2025
Short summary
Short summary
We describe an improved method and the associated free licensed package ANKIALE (ANalysis of Klimate with bayesian Inference: AppLication to extreme Events) for estimating the statistics of temperature extremes. This method uses climate model simulations (including multiple scenarios simultaneously) to provide a prior of the real-world changes, constrained by the observations. The method and the tool are illustrated via an application to temperature over Europe until 2100, for four scenarios.
Romain Pic, Clément Dombry, Philippe Naveau, and Maxime Taillardat
Adv. Stat. Clim. Meteorol. Oceanogr., 11, 23–58, https://doi.org/10.5194/ascmo-11-23-2025, https://doi.org/10.5194/ascmo-11-23-2025, 2025
Short summary
Short summary
Correctly forecasting weather is crucial for decision-making in various fields. Standard multivariate verification tools have limitations, and a single tool cannot fully characterize predictive performance. We formalize a framework based on aggregation and transformation to build interpretable verification tools. These tools target specific features of forecasts, improving predictive performance characterization and bridging the gap between theoretical and physics-based tools.
Cedric Gacial Ngoungue Langue, Helene Brogniez, and Philippe Naveau
EGUsphere, https://doi.org/10.5194/egusphere-2024-3481, https://doi.org/10.5194/egusphere-2024-3481, 2025
This preprint is open for discussion and under review for Atmospheric Measurement Techniques (AMT).
Short summary
Short summary
This work evaluates the representation of total column water vapor and total cloud cover in General Circulation Models, ERA5 reanalysis and satellite data records from the European Space Agency Climate Change Initiative. A new technique, called "multiresolution analysis," is applied to this evaluation, which enables an analysis of model behavior across different temporal frequencies, from daily to decadal scales, including subseasonal and seasonal variations.
Pauline Rivoire, Olivia Martius, Philippe Naveau, and Alexandre Tuel
Nat. Hazards Earth Syst. Sci., 23, 2857–2871, https://doi.org/10.5194/nhess-23-2857-2023, https://doi.org/10.5194/nhess-23-2857-2023, 2023
Short summary
Short summary
Heavy precipitation can lead to floods and landslides, resulting in widespread damage and significant casualties. Some of its impacts can be mitigated if reliable forecasts and warnings are available. In this article, we assess the capacity of the precipitation forecast provided by ECMWF to predict heavy precipitation events on a subseasonal-to-seasonal (S2S) timescale over Europe. We find that the forecast skill of such events is generally higher in winter than in summer.
Manuela Irene Brunner and Philippe Naveau
Hydrol. Earth Syst. Sci., 27, 673–687, https://doi.org/10.5194/hess-27-673-2023, https://doi.org/10.5194/hess-27-673-2023, 2023
Short summary
Short summary
Reservoir regulation affects various streamflow characteristics. Still, information on when water is stored in and released from reservoirs is hardly available. We develop a statistical model to reconstruct reservoir operation signals from observed streamflow time series. By applying this approach to 74 catchments in the Alps, we find that reservoir management varies by catchment elevation and that seasonal redistribution from summer to winter is strongest in high-elevation catchments.
Qiuyi Wu, Julie Bessac, Whitney Huang, Jiali Wang, and Rao Kotamarthi
Adv. Stat. Clim. Meteorol. Oceanogr., 8, 205–224, https://doi.org/10.5194/ascmo-8-205-2022, https://doi.org/10.5194/ascmo-8-205-2022, 2022
Short summary
Short summary
We study wind conditions and their potential future changes across the U.S. via a statistical conditional framework. We conclude that changes between historical and future wind directions are small, but wind speeds are generally weakened in the projected period, with some locations being intensified. Moreover, winter wind speeds are projected to decrease in the northwest, Colorado, and the northern Great Plains (GP), while summer wind speeds over the southern GP slightly increase in the future.
Antoine Grisart, Mathieu Casado, Vasileios Gkinis, Bo Vinther, Philippe Naveau, Mathieu Vrac, Thomas Laepple, Bénédicte Minster, Frederic Prié, Barbara Stenni, Elise Fourré, Hans Christian Steen-Larsen, Jean Jouzel, Martin Werner, Katy Pol, Valérie Masson-Delmotte, Maria Hoerhold, Trevor Popp, and Amaelle Landais
Clim. Past, 18, 2289–2301, https://doi.org/10.5194/cp-18-2289-2022, https://doi.org/10.5194/cp-18-2289-2022, 2022
Short summary
Short summary
This paper presents a compilation of high-resolution (11 cm) water isotopic records, including published and new measurements, for the last 800 000 years from the EPICA Dome C ice core, Antarctica. Using this new combined water isotopes (δ18O and δD) dataset, we study the variability and possible influence of diffusion at the multi-decadal to multi-centennial scale. We observe a stronger variability at the onset of the interglacial interval corresponding to a warm period.
Jakob Zscheischler, Philippe Naveau, Olivia Martius, Sebastian Engelke, and Christoph C. Raible
Earth Syst. Dynam., 12, 1–16, https://doi.org/10.5194/esd-12-1-2021, https://doi.org/10.5194/esd-12-1-2021, 2021
Short summary
Short summary
Compound extremes such as heavy precipitation and extreme winds can lead to large damage. To date it is unclear how well climate models represent such compound extremes. Here we present a new measure to assess differences in the dependence structure of bivariate extremes. This measure is applied to assess differences in the dependence of compound precipitation and wind extremes between three model simulations and one reanalysis dataset in a domain in central Europe.
Cited articles
Anderson, J. L.: A method for producing and evaluating probabilistic forecasts
from ensemble model integrations, J. Clim., 9, 1518–1530, 1996. a
Bessac, J.: Codes for scoring under uncertain verification data, available at: https://github.com/jbessac/uncertainty_scoring, GitHub [code], last access: 8 September 2021. a
Bowler, N. E.: Accounting for the effect of observation errors on verification
of MOGREPS, Meteorol. Appl., 15, 199–205, 2008. a
Bröcker, J. and Ben Bouallègue, Z.: Stratified rank histograms for
ensemble forecast verification under serial dependence, Q. J.
Roy. Meteorol. Soc., 146, 1976–1990,
https://doi.org/10.1002/qj.3778, 2020. a
Bröcker, J. and Smith, L. A.: Scoring probabilistic forecasts: The
importance of being proper, Weather Forecast., 22, 382–388, 2007. a
Candille, G. and Talagrand, O.: Retracted and replaced: Impact of observational
error on the validation of ensemble prediction systems, Q. J.
Roy. Meteorol. Soc., 134, 509–521, 2008. a
Cressie, N. and Wikle, C. K.: Statistics for spatio-temporal data, John Wiley
& Sons, Hoboken, N.J., 2015. a
Daley, R.: Estimating observation error statistics for atmospheric data
assimilation, Ann. Geophys., 11, 634–647, 1993. a
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin,
D. B.: Bayesian data analysis, CRC press, 2013. a
Gneiting, T. and Raftery, A. E.: Strictly proper scoring rules, prediction, and
estimation, J. Am. Stat. Assoc., 102, 359–378,
2007. a
Gneiting, T., Raftery, A. E., Westveld III, A. H., and Goldman, T.: Calibrated
probabilistic forecasting using ensemble model output statistics and minimum
CRPS estimation, Mon. Weather Rev., 133, 1098–1118, 2005. a
Gorgas, T. and Dorninger, M.: Quantifying verification uncertainty by reference
data variation, Meteorol. Z., 21, 259–277, 2012. a
Hamill, T. M.: Interpretation of rank histograms for verifying ensemble
forecasts, Mon. Weather Rev., 129, 550–560, 2001. a
Janjić, T., Bormann, N., Bocquet, M., Carton, J. A., Cohn, S. E., Dance,
S. L., Losa, S. N., Nichols, N. K., Potthast, R., Waller, J. A., and Weston,
P.: On the representation error in data assimilation, Q. J.
Roy. Meteorol. Soc., 144, 1257–1278, 2017. a
Jolliffe, I. T.: Uncertainty and inference for verification measures, Weather
Forecast., 22, 637–650, 2007. a
Jolliffe, T. and Stephenson, D. B.: Forecast verification: A practitioner's
guide in atmospheric science, edited by: Wiley, I., Chichester, Weather, 59, 132–132, https://doi.org/10.1256/wea.123.03,
2004. a, b, c
Kalman, R. E.: A new approach to linear prediction and filtering problems,
Transactions of the ASME, J. Basic Eng., 82, 35–45, 1960. a
Kalman, R. E. and Bucy, R. S.: New results in linear filtering and prediction
theory, J. Basic Eng., 83, 95–108, 1961. a
Kavetski, D., Kuczera, G., and Franks, S. W.: Bayesian analysis of input
uncertainty in hydrological modeling: 1. Theory, Water Resour. Res.,
42, 3, https://doi.org/10.1029/2005WR004368, 2006a. a
Kavetski, D., Kuczera, G., and Franks, S. W.: Bayesian analysis of input
uncertainty in hydrological modeling: 2. Application, Water Resour.
Res., 42, 3, https://doi.org/10.1029/2005WR004376, 2006b. a
Kleen, O.: Measurement Error Sensitivity of Loss Functions for Distribution
Forecasts, SSRN 3476461, https://doi.org/10.2139/ssrn.3476461, 2019. a
Murphy, A. H.: A new vector partition of the probability score, J.
Appl. Meteorol., 12, 595–600, 1973. a
Murphy, A. H. and Winkler, R. L.: A general framework for forecast
verification, Mon. Weather Rev., 115, 1330–1338, 1987. a
Muskulus, M. and Verduyn-Lunel, S.: Wasserstein distances in the analysis of
time series and dynamical systems, Physica D, 240,
45–58, 2011. a
National Centers for Environmental Information, National Oceanic Atmospheric Administration, U.S. Department of Commerce: Automated Surface Observing Systems (ASOS) program, [code], available at: ftp://ftp.ncdc.noaa.gov/pub/data/asos-onemin, last access: 8 September 2021. a
Pappenberger, F., Ghelli, A., Buizza, R., and Bodis, K.: The skill of
probabilistic precipitation forecasts under observational uncertainties
within the generalized likelihood uncertainty estimation framework for
hydrological applications, J. Hydrometeorol., 10, 807–819, 2009. a
Robert, C. and Casella, G.: Monte Carlo statistical methods, Springer Science
& Business Media, 2013. a
Robin, Y., Yiou, P., and Naveau, P.: Detecting changes in forced climate
attractors with Wasserstein distance, Nonl. Process. Geophys.,
24, 393–405, 2017. a
Santambrogio, F.: Optimal transport for applied mathematicians, Vol. 87, Birkhäuser Basel, 2015. a
Scheuerer, M. and Möller, D.: Probabilistic wind speed forecasting on a
grid based on ensemble model output statistics, Ann. Appl.
Stat., 9, 1328–1349, 2015. a
Schuhmacher, D., Bähre, B., Gottschlich, C., Hartmann, V., Heinemann, F.,
Schmitzer, B., Schrieber, J., and Wilm, T.: transport: Computation of
Optimal Transport Plans and Wasserstein Distances, R package
version 0.12-2,
https://cran.r-project.org/package=transport (last access: 8 September 2021), 2020. a
Skamarock, W., Klemp, J., Dudhia, J., Gill, D., Barker, D., Duda, M., Huang,
X.-Y., Wang, W., and Powers, J.: A description of the Advanced Research
WRF Version 3, Tech. Rep., https://doi.org/10.5065/D68S4MVH, 2008. a
Stein, C. M.: Estimation of the mean of a multivariate normal distribution,
Ann. Stat., 9, 1135–1151, https://doi.org/10.1214/aos/1176345632, 1981. a
Taillardat, M., Mestre, O., Zamo, M., and Naveau, P.: Calibrated Ensemble
Forecasts using Quantile Regression Forests and Ensemble Model Output
Statistics, Mon. Weather Rev., 144, 2375–2393,
https://doi.org/10.1175/MWR-D-15-0260.1, 2016. a, b
Taillardat, M., Fougères, A.-L., Naveau, P., and de Fondeville, R.: Extreme
events evaluation using CRPS distributions, arXiv preprint arXiv:1905.04022, available at: https://arxiv.org/abs/1905.04022
(last access: 8 September 2021),
2019. a
Waller, J. A., Dance, S. L., Lawless, A. S., and Nichols, N. K.: Estimating
correlated observation error statistics using an ensemble transform Kalman
filter, Tellus A, 66, 23294, https://doi.org/10.3402/tellusa.v66.23294, 2014. a
Weijs, S. V. and Van De Giesen, N.: Accounting for observational uncertainty in
forecast verification: an information-theoretical view on forecasts,
observations, and truth, Mon. Weather Rev., 139, 2156–2162, 2011. a
Weijs, S. V., Van Nooijen, R., and Van De Giesen, N.: Kullback–Leibler
divergence as a forecast skill score with classic
reliability–resolution–uncertainty decomposition, Mon. Weather Rev.,
138, 3387–3399, 2010. a
Zamo, M. and Naveau, P.: Estimation of the Continuous Ranked Probability Score
with Limited Information and Applications to Ensemble Weather Forecasts,
Math. Geosci., 50, 209–234, 2018. a
Short summary
We propose a new forecast evaluation scheme in the context of models that incorporate errors of the verification data. We rely on existing scoring rules and incorporate uncertainty and error of the verification data through a hidden variable and the conditional expectation of scores. By considering scores to be random variables, one can access the entire range of their distribution and illustrate that the commonly used mean score can be a misleading representative of the distribution.
We propose a new forecast evaluation scheme in the context of models that incorporate errors of...
