There is a need for accurate estimates of paleoclimate, especially temperature and precipitation, to better understand how climate has changed in the past. Scientific measurements of temperature and precipitation have been recorded for several hundred years, and in many locations for a much shorter time. Because of long-standing interest in weather, there are a vast number of anecdotal, nonscientific records of weather. However, many reconstructions of paleoclimate using compiled historical records are not amenable to direct statistical analysis because they consist of imprecise measurements of weather reported in letters, newspapers, books, and other documents . The large quantity of historical weather records, combined with appropriate statistical models, has the potential to facilitate the extension of scientific understanding of climate further back in time. Thus, there is a need for a statistical framework that can model historical data compiled from a variety of disparate sources by leveraging climate data from the recent past.

Historical observer weather data are often unreliable, sparse both temporally and spatially, and noisy because these data were recorded before widespread adoption of scientific measurement standards. As a result, historical observer weather data have not been widely used for rigorous statistical reconstructions of climate because these challenges make it difficult to create generic statistical approaches for analysis. Historical observer climate data can occur at hourly, daily, or monthly timescales, and the current-era analog data used to train statistical models can also vary in temporal resolution. Therefore, there is often a change of temporal support between the historical observer and current-era analog data that must be accounted for .

Another complication is that the true target one wishes to predict (the historical, unobserved climate) is never available to evaluate model predictive performance. Moreover, the historical observer data are often of unknown or of varying reliability and are typically sparse, sometimes involving only a few locations per year. The consequences of such data characteristics for evaluating model performance are underexplored; hence, we explore methods to validate historical observer-era model predictions under these sparse data scenarios.

We used spatially and temporally sparse historical observer measurements of temperature recorded at United States (US) military forts and other historical observer stations to reconstruct spatially explicit maps of mean mid-day July temperature by leveraging modern spatially explicit current-era analog data to impute missing spatial structure. We perform the reconstruction within a model framework that accounts for uncertainty in current-era data products and uncertainty in parameter estimation, and properly evaluates predictive skill. We test eight model specifications using a simulation study, generate predictions for mid-day July temperature at approximately 20 000 locations for each year in 1820–1894 with associated uncertainties, and evaluate model performance using a computationally efficient approximation to leave-one-out cross-validation.

We used two datasets we refer to as the historical observer dataset and the current-era analog dataset. The historical observer dataset consists of temperature records from 1820 to 1894 at US forts in the Upper Midwestern US as well as non-military observer stations. These data were compiled as part of the Climate Database Modernization Program (; CDMP 19th Century Forts and Voluntary Observers Database Build Project: http://www.isws.illinois.edu/atmos/clirecord.asp; ). At the observer stations, measurements were recorded with time and date; however, the timing of measurements varied among and within individual observer stations and was often temporally imprecise (“daily min”, “daily max”, “mid-day”, etc.).

Protocols varied across the observer stations through space and time, leading to many irregularities in the historical observer data. Temperature measurements were obtained by a variety of methods: some records report daily minimum and maximum temperatures, others report hourly measurements, and sometimes there are days or weeks with missing measurements. In addition, the number and locations of the observer stations change through time, containing between 1 and 234 locations per year; this variation is due to historical events, including the Civil War and the westward expansion of the US in the late 19th century. Most years have only a few observations and, in general, the number of observer locations per year increases through time. Therefore, the model must align the temporal and spatial scales of the two data sources to reconstruct continuous temperature fields across the Upper Midwest. An example of 4 years of historical data is shown in Fig. a.

Because the historical observer data are spatially sparse, traditional spatial statistical methods, such as Kriging, are not applicable, as these methods require larger sample sizes to produce reasonable predictive surfaces. Thus, we used the current-era analog data to provide spatial structure for the reconstruction. For the current-era analog data, we used the Parameter-elevation Relationships on Independent Slopes Model (PRISM) monthly mean mid-day temperature surfaces created by interpolation of the US Historical Climate Network (USHCN) data over the period 1895–2010 . The PRISM data include 115 years of mean mid-day July temperatures resolved to an 800 m×800 m grid, resulting in almost 20 000 spatial locations of interest in the study region (Fig. b). Unlike the historical observer data, the PRISM data are compiled from the USHCN and consist of commonly used model interpolated temperature records. Other data products are available, including high-quality data from satellite measurements; however, we used PRISM for the current-era analog data due to the longer temporal coverage that provides the model with more examples of the spatial structure of mid-day July temperature. Because PRISM is a data product and not raw data, we account for potential measurement errors in the current-era analog data using our modeling framework.

To evaluate model performance, we apply scoring rules to the estimated posterior predictive distributions. A highly desirable property of a scoring rule is propriety . A scoring rule is proper if the expected score of the optimal prediction is less than or equal to the expected score of any other prediction . Hence, a proper scoring rule, on average, chooses the best prediction from a set of candidate predictions . Often, paleoclimate reconstructions evaluate predictive performance by holding out some of the training set data for use in cross-validation, using skill scores like the coefficient of efficiency (CE) and relative efficiency (RE) . Although these scoring rules are common in the paleoclimate reconstruction community, suggest that scoring rules like CE and RE are improper in general. Because CE and RE are improper, it is possible that the optimal prediction can, on average, have a worse score than a sub-optimal prediction, leading to incorrect inference. Therefore, we focus on three proper scoring rules: mean square prediction error (MSPE), the continuous ranked probability score (CRPS), and a computationally efficient approximation to leave-one-out cross-validation (LOO) using the log score. In general, MSPE is not proper, but because our data models are Gaussian and Student's t, MSPE is proper for predictions of the posterior mean in this case.

The use of MSPE as a scoring rule implies an L2 loss function on the posterior distribution; therefore, our predictions are the posterior predictive means E(ỹt|yt)=∫ỹt[ỹt|yt]dỹt, where [ỹt|yt]=∫[ỹt|θt][θt|yt]dθt is the posterior predictive distribution for model parameters θt. Given out-of-sample observations yoos,t, MSPE is 1T∑t=1T1n-nt∑i∉HtE(ỹit|yt)-yoos,it2, where n-nt is the number of out-of-sample locations for year t and Ht is the set of observed locations in the historical observer data. Because MSPE uses the posterior predictive mean (a point prediction) instead of the full posterior distribution, MSPE ignores much of the information in the posterior distribution gained by performing Bayesian inference. Therefore, MSPE is not an ideal scoring rule for a probabilistic prediction, such as a posterior predictive distribution, even when MSPE is proper. For example, consider two models that give rise to posterior predictive distributions with the same posterior predictive mean but different posterior predictive variances. In this case, it is obvious that the predictive distribution that has better predictive coverage should be preferred, but MSPE would score the two models identically, demonstrating how MSPE loses information by collapsing the posterior distribution into a point estimate.

An alternative to MSPE is the CRPS scoring rule. CRPS is proper, utilizes the full posterior predictive distribution, and allows for a direct comparison of point predictions and probabilistic predictions . CRPS resolves the issue presented in the previously described scenario by including the width of the predictive distribution in the evaluation of the score. Several recent papers presenting climate reconstructions have made use of the CRPS for these reasons . Given a prediction with the cumulative distribution function, Fit, at location i and time t, and out-of-sample observations yoos,t, the CRPS is defined as CRPS({Fit}t=1T,yoos,t)=-∑t=1T∑i∉Ht∫-∞∞Fit(y)-Iy≥yoos,it2dy. show that Eq. () can be written alternatively as CRPS({Fit}t=1T,yoos)=∑t=1T1n-nt∑i∉HtEFityit-yoos,it-12EFityit-yit*, where yit and yit* are independent copies of a random variable with distribution function Fit and the expectation E is with respect to the probability density induced by Fit. The first expectation in Eq. () measures calibration (the absolute error of the prediction relative to the out-of-sample value) and the second expectation rewards predictions that are precise (i.e., narrow prediction intervals).

We can estimate the CRPS after obtaining posterior samples ỹt(k) from the posterior predictive distribution ỹt(k)|yt at each post burn-in iteration k. Then, Eq. () is approximated by CRPS^({F^it}t=1T,yoos)=∑t=1T1n-nt∑i∉Ht1K∑k=1Kỹit(k)-yoos,it-12K2∑k=1K∑ℓ=1Kỹit(k)-ỹit(ℓ). A major disadvantage of both MSPE and CRPS is the need for out-of-sample validation data. For our simulation study, MSPE and CRPS are straightforward to calculate because we simulated the out-of-sample validation data; in practical paleoclimate reconstructions, there are no out-of-sample data. Therefore, MSPE and CRPS must be approximated using cross-validation methods, although these methods are computationally costly and time consuming to implement.

An alternative is to use the approximate leave-one-out cross-validation method (LOO; ). LOO uses a proper scoring rule, the log score, to evaluate predictive skill . We estimate the leave-one-out log pointwise predictive density lpdloo=∑t=1T∑i∈Htlog⁡[yit|y(i)t]=∑t=1T∑i∈Htlog⁡∫[yit|θt][θt|y(i)t]dθt, where y(i)t are the data yt at time t without the ith location. One can calculate Eq. () directly by cross-validation at a high computational cost, or one can approximate Eq. () using importance sampling from post burn-in posterior samples using the full data as described in . Importance ratios with high variance can cause the estimate in Eq. () to be highly unstable and unreliable, and are therefore of practical concern. To test for the presence of large variance of the importance ratios, proposed fitting the generalized Pareto distribution to the upper tail of importance ratios and examining the empirical estimates of the tail shape parameter ξ. If the estimated tail parameter ξ^it is less than 1/2, the variance of the importance ratios is finite and the importance ratios approximating the log posterior score holding out yit can be used directly to approximate LOO. If the estimated tail parameter is 1/2<ξ^it<1, the variance of the importance ratios is infinite but the mean of the importance ratios exists. Hence, propose using smoothed importance ratios. If the estimated tail parameter ξ^it>1, this suggests that the mean and variance of the importance ratios do not exist but that the variance of the smoothed importance ratios is finite, but large, and the use of LOO is sensitive to the held-out observation. Using the smoothed importance weights wit(k), we obtain the Pareto-smoothed importance sampling approximation elpd^PSIS=∑t=1T∑i∈Htlog⁡∑k=1Kwit(k)[yit|θt(k)]∑k=1Kwit(k).

We use the deviance scale and set LOO^=-2elpd^PSIS to make LOO a negatively oriented score (the best model is the one with the lowest score), implementing our score using R package loo .

With paleoclimate data, it is difficult to verify the predictive ability of models using cross-validation. With only a handful of observations in the historical observer data available for each year, cross-validation techniques could be highly biased due to the effects of unusual observations in small sample sizes. This is important because we expect noisy and potentially outlying observations in the historical observer data due to the data collection procedures. Additionally, the high dimensionality of the field we aim to reconstruct and the use of computationally intensive MCMC estimation make cross-validation costly. Instead, we conducted a simulation study to explore the different models for the historical observer station data and evaluate model performance using the scoring rules above. Although we do not simulate from the model that is used for estimation, the simulated data represent a reasonable approximation to mid-day July temperature, providing an environment for model testing and exploration of empirical performance.

We simulate mid-day July temperature in one spatial dimension (we extend to two dimensions using the real data), allowing for faster computation and easier graphical exploration of the spatio-temporal process. We simulate T=50 realizations of a latent surface from the model st=Wβt+ηt, where the matrix W represents fixed influences on climate, such as latitude, elevation, and other covariates that explain much of the temperature surface as well as time-varying components that represent slowly varying global-scale climate processes. To construct patterns that might be seen in climate observations, we simulate temporally varying regression coefficients at different periodicities to represent global-scale climate processes like the Pacific Decadal Oscillation or the Atlantic Multidecadal Oscillation. We do not claim our simulation behaves like any climatological process, only that this example facilitates exploration of complicated patterns potentially seen in climatological data.

We include a spatially correlated random effect ηt∼N0,σηt2Rϕ that smooths the patterns, generating realizations of a one-dimensional climate field (Fig. b). A common choice for the form of Rϕ is the Matérn class of correlation functions. For our simulation, we use the exponential correlation function, a member of the Matérn family. In the exponential correlation function, the i,jth element Rij(ϕ)=exp⁡-dijϕ, where dij represents the Euclidean distance between the ith and jth spatial locations and ϕ is the spatial range parameter.

To create observations that match the temporal irregularities and spatial clustering behavior in the historical observer data, we sample the one-dimensional spatial field using weighted probabilities that generate clustered observations in space, storing the simulated temperature observations at the nt locations in the vector yt. Using this sampling design, we generate noisy realizations for the simulated historical observer data for simulated years t=1,…,25 (Fig. a) using yit=sit+ϵ̃it, where ϵ̃it is independent Student's t error with variance σ̃2=1.5 and degrees of freedom ν=10 that represent uncertainty in historical temperature measurements. To generate the set of simulated current-era analog patterns X that will be used in our regression model, we define for the simulated current-era analog years t=26,…,50 xt=st+ϵ¨t, where, by adding uncorrelated, independent Gaussian noise ϵ¨t with variance σ¨2=0.75, we account for the measurement error of the temperature process during the current-era analog period. The measurements used in the model for the current-era analog period are from PRISM model interpolated data and have measurement error, where this measurement error should be less than that of the historical observer data; hence, σ̃2>σ¨2. We then combine the simulated values into the noisy pattern matrix X≡x26,…,x50 that is used in our model framework. The latent temperature patterns S≡s1,…,s25 are the unobserved target for our reconstruction. Note that, in the real data, S are unavailable; therefore, our scoring rules can use only the noisy observations yt, limiting the ability to improve reconstruction skill. Figure a shows the simulated historical observer period noisy temperature realizations yt and Fig. b shows the simulated historical observer period true, latent temperature field that is the target of our reconstruction, with the x axis representing spatial location. The noisy principal components derived from the simulated current-era analog data are plotted in Fig. c with the simulated latent principal components in Fig. d. Comparing Fig. c and d shows that using the noisy principal components is analogous to the errors-in-covariates framework, where noisy observations of the current-era analog covariates (in our case the principal components) can lead to bias in the regression coefficients, inflated residual variance, and a reduction in prediction skill .

We compare the performance of each model specification using MSPE, CRPS, and LOO scoring rules in our simulation where the best model is the one with the smallest score. We fit the PCR and pPCR models using SSVS and LASSO regularization with both the Gaussian and robust Student's t data models, comparing eight models with the results displayed in Table . Across scores, the models perform similarly, with no consistently best model, although the robust models generally have lower scores than the Gaussian models. The LOO Pareto tail parameter estimates for the robust models show less evidence of misspecification than the traditional Gaussian data models (figure not shown) and the pPCR models have consistently smaller tail parameter estimates than the PCR models, suggesting that the least misspecified models are robust pPCR.

Predicted versus simulated temperatures for the robust PCR and robust pPCR models using LASSO regularization are shown in Fig.  with years represented using different colors. Because the predictions for each year cluster around the 45∘ line, this shows the models reconstruct the annual differences accurately, which are the main features of the simulated data and of primary interest in understanding climate change. However, the models fail to reconstruct much of the within-year spatial variability (the humps and valleys within a year in Fig. b), which is unsurprising given the small sample sizes. Despite having similar predictive scores, there are visible differences in predictions between the two models. The robust PCR predictions predict some of the spatial structure of the mean (the point clouds are generally centered on the 45∘ line), but tend to have unreasonably small predictive standard deviations (not shown). The robust pPCR predictions estimate a spatially averaged annual mean in years with small sample sizes, but predict little spatial structure. Instead, robust pPCR produces predictive standard deviation estimates that account for uncertainty in years with very little data.

After exploring the model framework using a simulation study, we applied our models to the historical observer data. We fit the eight models to the data and present the results from LOO in Table . Examination of the LOO Pareto tail parameter plots in Fig. a and b identified an outlier occurring at data point 1452, corresponding to an unrealistic mean mid-day July temperature measurement of 42 F (7.8 ∘C). After removing the outlier, we fit the models again. Interestingly, the Gaussian model fits without the outlier showed less predictive skill (larger LOO values in Table ) and more model misspecification (a larger tail parameter estimate in Fig. e and f). The decreased performance when removing the outlier is explained by the influence of the outlier on the pooled variance estimate; removing the outlier shrinks the pooled variance estimate and the Gaussian data model is less able to accommodate slightly outlying points with the smaller variance. For the robust models, quality of model fit was slightly improved when fit with the outlier removed (see Table ), but there is little change in model misspecification as defined by large Pareto tail parameter estimates (Fig. c, d, g, and h).

With the outlier removed, the best predictive models are robust PCR and robust pPCR due to having the smallest LOO scores (see Table ). Based purely on LOO, it appears the best overall predictive model is robust PCR, although the increasing sample sizes through time weight the LOO score to predictions of the most recent years. All models still show evidence of misspecification because some Pareto tail parameter estimates are greater than 0.5 (Fig. e, f, g, and h), but the removal of the outlier improved model fit in general. Figure g and h suggest that the robust pPCR model might be preferable to the robust PCR model farther back in time because, for that time period, a much greater proportion of Pareto tail parameter estimates are less than 0.5 in the robust pPCR model than the robust PCR model. Hence, there is some evidence that the choice of model is dependent on the desired inference. If inference over the entire period is desired, both robust PCR and robust pPCR predict with skill. If inference is desired on the years furthest back in time, Fig.  suggests that robust pPCR predictions are preferable.

To visualize our results, we plot reconstructions of 4 years of the historical temperature surfaces using the robust PCR model (Fig. ) and the robust pPCR model (Fig. ). Visual comparison of the reconstructions illustrates differences in the two models. The robust PCR model assumes the climate patterns in the observational data are without error; the stronger influence of these patterns is seen in the posterior predictive mean surface (Fig. a), particularly for year 1847. In comparison, the robust pPCR model shows less influence of these climate patterns in the posterior predictive mean (Fig. a), particularly for year 1847. Because the robust pPCR model includes the assumption that some of the pattern is noise, the reconstructions have less spatial structure in years with little data. Differences in posterior predictive standard deviations are also evident (Figs. b and b), where the robust PCR model shows reduction of uncertainty in the spatial locations near observations and higher uncertainty away from the observations, while the robust pPCR model has posterior predictive standard deviations that are more spatially diffuse and perhaps more realistic given the lack of spatial predictive ability seen in the simulation study. The reconstructed temperature surfaces for both the robust PCR and robust pPCR models for every year are included in the Supplement.

By using the spatial structure in the current-era analog data, we generated temperature predictions at unobserved locations with corresponding uncertainties. We chose four locations, Champaign, Illinois, Detroit, Michigan, Madison, Wisconsin, and Minneapolis, Minnesota, and show the time series of temperature predictions in Fig. a. From these time series, we see smaller standard deviations for the years with more historical observer period observations (sample sizes are shown in Fig. b), with greater uncertainty the further we go back in time. We can also see a general trend for the standard deviations of the robust PCR model to be smaller than the robust pPCR model (the red intervals are more often inside the blue intervals). The robust PCR model is also more likely to have structure in the mean that may not be warranted when the sample size is low (i.e., the spike at Champaign and Detroit that is not present in Madison or Minneapolis in 1847 and 1848 with sample sizes of 3). The two models show evidence of a bias/variance trade-off, with the robust PCR tending toward spatially structured predictions with smaller variance and the robust pPCR model providing less spatially structured predictions with larger variance.

There are many challenges inherent in modeling paleoclimate data. Due to the lack of direct measurements of climate, paleoclimate reconstructions must rely on sparse, noisy proxies of climate. The nuances of paleoclimate data often require specialized modeling techniques and careful investigation into modeling assumptions and performance. In addition, care is needed to properly validate paleoclimate reconstruction skill. In summary, we extended principal component regression methods, applied regularization techniques to choose important principal components, developed robust models to account for the presence of outliers, and explored the use of a probabilistic principal component model to account for measurement uncertainty in the spatially rich current-era analog data. By rigorously evaluating the predictive skill of our models, we were able to explore our extensions of PCR for climate reconstruction, laying the groundwork for future developments with more complex climate data than PRISM temperature surfaces. The models presented in this paper would be good candidates for modeling climate variables that are strongly non-stationary and non-Gaussian (e.g., wind speed or precipitation), but these extensions are the subject of ongoing research.

Within our modeling framework, we presented a simulation study for evaluating paleoclimate reconstructions using proper scoring rules. By using proper scoring rules and exploring model performance in a simulation framework, we have stronger support for the quality of the reconstruction. We presented three statistical scoring rules and explored their strengths and weaknesses. MSPE is a commonly used and easy to understand scoring rule, but is not proper in general and only uses a point prediction, ignoring the probabilistic inference that is gained by using Bayesian techniques. The CRPS is proper and allows for direct comparison of point predictions and probabilistic predictions, but requires out-of-sample validation data or computationally expensive cross-validation. The use of MSPE and CRPS scoring rules allowed for exploration of the empirical properties of the computationally efficient LOO approximation to leave-one-out cross-validation. Our use of LOO to score the historical observer period model predictions not only enabled us to perform model selection, but also aided in diagnosing an outlying observation and refining model fit.

The methods presented in this paper could be applied to other historical datasets at different locations around the world, further extending the spatially explicit empirical record of climate further back in time while rigorously accounting for uncertainties. The methods we presented could also be extended to model temperature and precipitation for each month of the year by including a seasonal component in the calibration model and by modeling dynamics at appropriate timescales. There are many datasets that could be used within this framework as the current-era analog, including modern satellite data. The different options of current-era analog datasets present a trade-off between the number of records available as analogs and the quality of the data. If there are occasionally rare climate processes that occur, it seems that a longer record of climate analogs would be preferred. If the climate processes are relatively stable in time but vary highly in space, a shorter and more precise modern dataset that is not model interpolated might be preferred. In addition, use of highly precise current-era analog data could reduce or eliminate the need to account for measurement error in the current-era analog data.

Ultimately, our temperature reconstructions extend the climatological record in the Upper Midwestern US further into the past. These temperature reconstructions, with their associated uncertainties, can be used to gain better understanding of the influences of climate on the biological and ecological processes observed in the region. By backcasting mean mid-day July temperature with our models, we gain the potential to better understand how climate has changed, and this knowledge could be used to improve future climate reconstructions. Many of the techniques and methods we used – modeling principal components with a probabilistic model, hierarchical pooling to borrow strength among years with sparse and dense data, model selection and regularization, and proper model evaluation – can be adapted and used in future climate reconstruction problems.

The historical observer data are available from http://www.isws.illinois.edu/atmos/clirecord.asp and the current-era analog data are available from http://www.prism.oregonstate.edu/. Compiled data and code can be accessed on gitHub at https://github.com/jtipton25/observer (10.5281/zenodo.242996).