Articles | Volume 7, issue 1
https://doi.org/10.5194/ascmo-7-13-2021
© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
https://doi.org/10.5194/ascmo-7-13-2021
© Author(s) 2021. This work is distributed under
the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
Novel measures for summarizing high-resolution forecast performance
Research Applications Laboratory, National Center for Atmospheric Research, Boulder, Colorado, USA
Related authors
Manuela I. Brunner, Eric Gilleland, and Andrew W. Wood
Earth Syst. Dynam., 12, 621–634, https://doi.org/10.5194/esd-12-621-2021, https://doi.org/10.5194/esd-12-621-2021, 2021
Short summary
Short summary
Compound hot and dry events can lead to severe impacts whose severity may depend on their timescale and spatial extent. Here, we show that the spatial extent and timescale of compound hot–dry events are strongly related, spatial compound event extents are largest at
sub-seasonal timescales, and short events are driven more by high temperatures, while longer events are more driven by low precipitation. Future climate impact studies should therefore be performed at different timescales.
Chiara Marsigli, Elizabeth Ebert, Raghavendra Ashrit, Barbara Casati, Jing Chen, Caio A. S. Coelho, Manfred Dorninger, Eric Gilleland, Thomas Haiden, Stephanie Landman, and Marion Mittermaier
Nat. Hazards Earth Syst. Sci., 21, 1297–1312, https://doi.org/10.5194/nhess-21-1297-2021, https://doi.org/10.5194/nhess-21-1297-2021, 2021
Short summary
Short summary
This paper reviews new observations for the verification of high-impact weather and provides advice for their usage in objective verification. New observations include remote sensing datasets, products developed for nowcasting, datasets derived from telecommunication systems, data collected from citizens, reports of impacts and reports from insurance companies. This work has been performed in the framework of the Joint Working Group on Forecast Verification Research (JWGFVR) of the WMO.
Manuela I. Brunner, Eric Gilleland, and Andrew W. Wood
Earth Syst. Dynam., 12, 621–634, https://doi.org/10.5194/esd-12-621-2021, https://doi.org/10.5194/esd-12-621-2021, 2021
Short summary
Short summary
Compound hot and dry events can lead to severe impacts whose severity may depend on their timescale and spatial extent. Here, we show that the spatial extent and timescale of compound hot–dry events are strongly related, spatial compound event extents are largest at
sub-seasonal timescales, and short events are driven more by high temperatures, while longer events are more driven by low precipitation. Future climate impact studies should therefore be performed at different timescales.
Chiara Marsigli, Elizabeth Ebert, Raghavendra Ashrit, Barbara Casati, Jing Chen, Caio A. S. Coelho, Manfred Dorninger, Eric Gilleland, Thomas Haiden, Stephanie Landman, and Marion Mittermaier
Nat. Hazards Earth Syst. Sci., 21, 1297–1312, https://doi.org/10.5194/nhess-21-1297-2021, https://doi.org/10.5194/nhess-21-1297-2021, 2021
Short summary
Short summary
This paper reviews new observations for the verification of high-impact weather and provides advice for their usage in objective verification. New observations include remote sensing datasets, products developed for nowcasting, datasets derived from telecommunication systems, data collected from citizens, reports of impacts and reports from insurance companies. This work has been performed in the framework of the Joint Working Group on Forecast Verification Research (JWGFVR) of the WMO.
Cited articles
Baddeley, A. J.: An error metric for
binary images, in: Robust Computer Vision Algorithms, edited by: Forstner, W. and Ruwiedel, S., Wichmann, 402 pp., 59–78, available at:
https://pdfs.semanticscholar.org/aa50/669b71429f2ca54d64f93839a9da95ceba6b.pdf (last access: 14 May 2020), 1992b. a, b, c, d, e, f
Baddeley, A. and Turner, R.: spatstat: An R Package for Analyzing Spatial
Point Patterns, J. Stat. Softw., 12, 1–42,
available at: http://www.jstatsoft.org/v12/i06/ (last access: 11 February 2021), 2005. a
Baldwin, M. E. and Elmore, K. L.: Objective verification of high-resolution WRF forecasts during 2005 NSSL/SPC Spring Program, in: 21st Conf. on
Weather Analysis and Forecasting/17th Conf. on Numerical Weather Prediction,
Amer. Meteor. Soc., Washington, D.C., 11B4, 2005. a
Becker, R. A., Wilks, A. R., Brownrigg, R., Minka, T. P., and Deckmyn, A.:
maps: Draw Geographical Maps, R package version
3.3.0, available at: https://CRAN.R-project.org/package=maps (last access: 11 February 2021), 2018. a
Brown, B. G., Gilleland, E., and Ebert, E. E.: Forecasts of Spatial Fields,
John Wiley and Sons, Ltd., 95–117, 2011. a
Brunet, D. and Sills, D.: A generalized distance transform: Theory and
applications to weather analysis and forecasting, IEEE T. Geosci. Remote, 55, 1752–1764,
https://doi.org/10.1109/TGRS.2016.2632042, 2017. a
Brunet, D., Sills, D., and Casati, B.: A spatio-temporal user-centric distance
for forecast verification, Meterol. Z., 27, 441–453,
https://doi.org/10.1127/metz/2018/0883, 2018. a
Casati, B., Ross, G., and Stephenson, D. B.: A new intensity-scale approach for
the verification of spatial precipitation forecasts, Meteorol. Appl., 11,
141–154, https://doi.org/10.1017/S1350482704001239, 2004. a, b
Cressie, N. and Suesse, T.: Great expectations and even greater exceedances
from spatially referenced data, Spat. Stat.-Neth., 37, 100420,
https://doi.org/10.1016/j.spasta.2020.100420, 2020. a
Dorninger, M., Mittermaier, M. P., Gilleland, E., Ebert, E. E., Brown, B. G.,
and Wilson, L. J.: MesoVICT: Mesoscale Verification Inter-Comparison over
Complex Terrain, Tech. rep., No. NCAR/TN-505+STR, https://doi.org/10.5065/D6416V21,
2013. a, b
Dorninger, M., Gilleland, E., Casati, B., Mittermaier, M. P., Ebert, E. E.,
Brown, B. G., and Wilson, L.: The set-up of the Mesoscale Verification
Inter-Comparison over Complex Terrain project, B. Am. Meteorol. Soc.,
99, 1887–1906, https://doi.org/10.1175/BAMS-D-17-0164.1, 2018. a, b
Ebert, E. E.: Fuzzy verification of high-resolution gridded forecasts: a review
and proposed framework, Meteorol. Appl., 15, 51–64, 2008. a
Ebert, E. E. and McBride, J. L.: Verification of precipitation in weather
systems: determination of systematic errors, J. Hydrol., 239, 179–202,
https://doi.org/10.1016/S0022-1694(00)00343-7, 2000. a
Gilleland, E.: Spatial Forecast Verification: Baddeley's Delta Metric Applied
to the ICP Test Cases, Weather Forecast., 26, 409–415, 2011. a
Gilleland, E.: SpatialVx: Spatial Forecast Verification, R package
version 0.6-6,
available at: http://www.ral.ucar.edu/projects/icp/SpatialVx (last access: 11 February 2021), 2019. a
Gilleland, E., Ahijevych, D., Brown, B. G., Casati, B., and Ebert, E. E.:
Intercomparison of Spatial Forecast Verification Methods, Weather Forecast.,
24, 1416–1430, 2009. a
Gilleland, E., Skok, G., Brown, B. G., Casati, B., Dorninger, M., Mittermaier, M. P., Roberts, N., and Wilson, L. J.: A novel set of verification test
fields with application to distance measures, Mon. Weather Rev., 148, 1653–1673,
https://doi.org/10.1175/MWR-D-19-0256.1, 2020. a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t
Kain, J. S., Weiss, S. J., Bright, D. R., Baldwin, M. E., Levit, J. J., Carbin, G. W., Schwartz, C. S., Weisman, M. L., Droegemeier, K. K., Weber, D. B., and
Thomas, K. W.: Some practical considerations regarding horizontal resolution
in the first generation of operational convection-allowing NWP, Weather
Forecast., 23, 931–952, https://doi.org/10.1175/WAF2007106.1, 2008. a
Keil, C. and Craig, G.: A displacement-based error measure applied in a
regional ensemble forecasting system, Mon. Weather Rev., 135, 3248–3259, 2007. a
Keil, C. and Craig, G.: A displacement and amplitude score employing an optical
flow technique, Weather Forecast., 24, 1297–1308, 2009. a
Koch, J., Jensen, K. H., and Stisen, S.: Toward a true spatial model evaluation
in distributed hydrological modeling: Kappa statistics, Fuzzy theory, and EOF-analysis benchmarked by the human perception and evaluated against a
modeling case study, Water Resour. Res., 51, 1225–1246,
https://doi.org/10.1002/2014WR016607, 2015. a
Koch, J., Siemann, A., Stisen, S., and Sheffield, J.: Spatial validation of
large-scale land surface models against monthly land surface temperature
patterns using innovative performance metrics, J. Geophys. Res.-Atmos., 121, 5430–5452, https://doi.org/10.1002/2015JD024482, 2016. a
Koch, J., Demirel, M. C., and Stisen, S.: The SPAtial EFficiency metric (SPAEF): multiple-component evaluation of spatial patterns for optimization of hydrological models, Geosci. Model Dev., 11, 1873–1886, https://doi.org/10.5194/gmd-11-1873-2018, 2018. a
Kullback, S. and Leibler, R. A.: On information and sufficiency, Ann.
Math. Stat., 22, 79–86, https://doi.org/10.1214/aoms/1177729694, 1951. a
Marzban, C. and Sandgathe, S.: Optical flow for verification, Weather
Forecast., 25, 1479–1494, https://doi.org/10.1175/2010WAF2222351.1, 2010. a
Mass, C. F., Ovens, D., Westrick, K., and Colle, B. A.: Does Increasing
Horizontal Resolution Produce More Skillful Forecasts?, B. Am. Meteorol. Soc., 83, 407–430,
https://doi.org/10.1175/1520-0477(2002)083<0407:DIHRPM>2.3.CO;2, 2002. a
Meijster, A., Roerdink, J. B. T. M., and Hesselink, W. H.: A General Algorithm for Computing Distance Transforms in Linear Time, in: Mathematical Morphology and its Applications to Image and Signal Processing, edited by: Goutsias, J., Vincent, L., Bloomberg, D. S., Computational Imaging and Vision, Springer, Boston, MA, 18, https://doi.org/10.1007/0-306-47025-X_36, 2000. a
Mittermaier, M. P.: A strategy for verifying near-convection resolving model forecasts at observing sites, Weather Forecast., 29, 185–204, 2014. a
Nychka, D., Furrer, R., Paige, J., and Sain, S.: fields: Tools for spatial
data, R package version 10.0, https://doi.org/10.5065/D6W957CT, available at: https://github.com/NCAR/Fields (last access: 11 February 2021), 2017. a
R Core Team: R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria,
available at: https://www.R-project.org/ (last access: 11 February 2021), 2018. a
Rezacova, D., Sokol, Z., and Pesice, P.: A radar-based verification of
precipitation forecast for local convective storms, Atmos. Res., 83,
221–224, https://doi.org/10.1016/j.atmosres.2005.08.011, 2007. a
Roberts, N. M. and Lean, H. W.: Scale-Selective Verification of Rainfall
Accumulations from High-Resolution Forecasts of Convective Events, Mon.
Weather Rev., 136, 78–97, 2008. a
Rossa, A. M., Nurmi, P., and Ebert, E. E.: Overview of methods for the
verification of quantitative precipitation forecasts, edited by: Michaelides, S. C., Springer, New York, NY, USA, 418–450, 2008. a
Skamarock, W. C., Klemp, J. B., Dudhia, J., Gill, D. O., Barker, M., Wang, W.,
and Powers, J. G.: A description of the Advanced Research WRF version 2,
Ncar tech. rep. ncar/tn-468+str, Boulder, CO, USA, 2005. a
Steinacker, R., Häberli, C., and Pöttschacher, W.: A transparent method
for the analysis and quality evaluation of irregularly distributed and noisy
observational data, Mon. Weather Rev., 128, 2303–2316, 2000. a
Venugopal, V., Basu, S., and Foufoula-Georgiou, E.: A new metric for comparing
precipitation patterns with an application to ensemble forecasts, J. Geophys.
Res., 110, D08111, https://doi.org/10.1029/2004JD005395, 2005. a
Wealands, S. R., Grayson, R. B., and Walker, J. P.: Quantitative comparison of
spatial fields for hydrological model assessment – some promising
approaches, Adv. Water Resour., 28, 15–32, 2005. a
Weniger, M., Kapp, F., and Friederichs, P.: Spatial Verification Using Wavelet
Transforms: A Revie, Q. J. Royal Meteor. Soc., 143,
120–136, https://doi.org/10.1002/qj.2881, 2016. a
Wikle, C. K., Zammit-Mangion, A., and Cressie, N.: Spatio-Temporal Statistics
with R, Chapman and Hall/CRC, Boca Raton, Florida, USA, 396 pp.,
available at: https://spacetimewithr.org (last access: 11 February 2021), 2019.
a
Zhang, J., Craigmile, P. F., and Cressie, N.: Loss function approaches to
predict a spatial quantile and its exceedance region, Technometrics, 50, 216–227, https://doi.org/10.1198/004017008000000226, 2008. a
Zinner, T., Mannstein, H., and Taferner, A.: Cb-TRAM: Tracking and monitoring
severe convection from onset over rapid development to mature phase using
multi-channel Meteosat-8 SEVIRI data, Meteorol. Atmos. Phys., 101, 191–210,
https://doi.org/10.1007/s00703-008-0290-y, 2008. a
Short summary
Verifying high-resolution weather forecasts has become increasingly complicated,
and simple, easy-to-understand summary measures are a good alternative. Recent work has demonstrated some common pitfalls with many such summaries. Here, new summary measures are introduced that do not suffer from these drawbacks, while still providing meaningful information.
Verifying high-resolution weather forecasts has become increasingly complicated,
and simple,...