Time Series Forecasting with Conformal Prediction Intervals: Scikit-Learn is All you Need | by Marco Cerliani | Dec, 2022

By Jessie Hobb On Dec 14, 2022

Accurate Uncertainty Quantification with MAPIE and TSPIRAL

When carrying out a time series forecasting task we are used to developing solutions that produce point-wise estimations of future observations. That’s correct and, if properly validated, they may positively impact business results. Is it possible to do better? Can we provide more detailed forecasts by simply adding further information?

Enriching our forecasts with prediction intervals is the key. Practically speaking a prediction interval is represented by a couple of numbers. These values are respectively a lower and an upper bound where future observations are likely to occur. The likelihood that the future values fall in the given bound is expressed by a floating number (alpha) between 0 and 1. Where an alpha near one means that we are more confident that this happen.

It’s straightforward to understand the added value of attaching a prediction interval to our forecasts. Providing an uncertainty estimation is an old need that can be solved in various ways. All methods are awesome if properly adopted, but some are better. Let’s try to elaborate.

Everyone knows bootstrapping as a resampling technique. It’s common to see bootstrapping applied to forecasting residuals to obtain an uncertainty measure for future forecasts. Despite residual bootstrapping may be a good starting point when approaching for the first time uncertainty quantification, it may lead to poor performance since it only takes care of data uncertainty. Another source of uncertainty exists. We are referring to the modeling uncertainty. Modeling uncertainty takes care of possible lack of knowledge encountered in the training phases, which may influence forecasts. A good forecasting uncertainty measure should encompass both data and modeling uncertainties.

In this post, we introduce conformal prediction as a technique to estimate uncertainty. Specifically, we demonstrate how to produce prediction intervals in a time series forecasting scenario. Using tspiral (a python package for time series forecasting with scikit-learn estimators) in conjunction with MAPIE (a scikit-learn-compatible module for estimating prediction intervals), we show how to solve a time forecasting task providing accurate uncertainty estimation without moving outside the scikit-learn ecosystem.

The first step for producing prediction intervals consists in choosing the forecasting model to use. That seems not reasonable but it’s one of the main benefits of conformal prediction since it’s a model-agostic technique (i.e. it can be used in any context with any predictive algorithms).

For this post, we focus on two of the most famous forecasting techniques adopted in the machine learning ecosystem. We are referring to recursive and direct forecasting. They are both known methodologies with proper benefits and drawbacks, and accessible in the scikit-learn format using tspiral (to learn more about the topic I suggest one of my previous articles).

Simulated Sinusoidal Series separated into Train, Validation, and Test [image by the author]

Let’s imagine having produced the forecasting depicted below using a simulated sinusoidal series. How is it possible to add reliable prediction intervals to our predictions?

Comparing Recursive (blue) and Direct (orange) Forecasting on Test [image by the author]

To solve the problem we can use conformal prediction. Conformal prediction intervals are built by studying the distributions of the residuals. There isn’t any secret receipt or any sort of magic.

alpha = 0.95conformity_scores = np.abs(np.subtract(y_val, y_pred_val))
estimated_distributions = np.add(y_pred_test[:, None], conformity_scores)
lower_q, upper_q = np.quantile(
estimated_distributions, 
[(1-alpha)/2, 1-(1-alpha)/2], axis=1
)

Given the predictions and the true values on a validation set, we have to:

compute the absolute residuals (conformity_scores).
add the conformity scores to the test predictions. This generates distributions for each point-wise test forecast (estimated_distributions).
compute upper and lower quantiles for each point-wise forecast distribution to obtain prediction intervals.

Point-wise Recursive Conformity Score Distributions (upper). Quantiles on Recursive Conformity Score Distributions (lower). [image by the author]

Despite its simplicity, it’s possible to automate the calculation of conformal prediction intervals using MAPIE. Let’s see it in action for both recursive and direct forecasting.

Conformal Intervals with Recursive Forecasting

forecaster = ForecastingCascade(
Ridge(), 
lags=range(1,24+1),
use_exog=False,
)forecaster.fit(None, y_train)
model = MapieRegressor(
forecaster, cv="prefit",
).fit(X_val, y_val)
forecaster.fit(None, y_train_val)
model.single_estimator_ = forecaster
forecasts = model.predict(X_test, alpha=0.05)

Recursive Forecasting plus Conformal Prediction Intervals. [image by the author]

Conformal Intervals with Direct Forecasting

forecaster = ForecastingChain(
Ridge(),
n_estimators=len(y_test),
lags=range(1,24+1),
use_exog=False,
)forecaster.fit(None, y_train)
model = MapieRegressor(
forecaster, cv="prefit",
).fit(X_val, y_val)
forecaster.fit(None, y_train_val)
model.single_estimator_ = forecaster
forecasts = model.predict(X_test, alpha=0.05)

In this naive form, MAPIE can estimate prediction intervals simply given a validation set and a fitted model (as depicted above). To grant more robustness, it’s possible carrying out the uncertainty estimation using a cross-validated approach or more sophisticated techniques.

Direct Forecasting plus Conformal Prediction Intervals. [image by the author]

Conformal Intervals with Recursive Forecasting plus CrossValidation

forecaster = ForecastingCascade(
Ridge(),
lags=range(1,24+1),
use_exog=False,
)model = MapieRegressor(
forecaster, 
cv=TemporalSplit(20, test_size=len(y_test)),
method='base', agg_function=None, n_jobs=-1,
).fit(X_train_val, y_train_val)
forecasts = model.predict(X_test, alpha=0.05, ensemble=False)

Recursive Forecasting plus Conformal Prediction Intervals using CrossValidation. [image by the author]

Conformal Intervals with Direct Forecasting plus CrossValidation

forecaster = ForecastingChain(
Ridge(),
n_estimators=len(y_test),
lags=range(1,24+1),
use_exog=False,
)model = MapieRegressor(
forecaster, 
cv=TemporalSplit(20, test_size=len(y_test)),
method='base', agg_function=None, n_jobs=-1,
).fit(X_train_val, y_train_val)
forecasts = model.predict(X_test, alpha=0.05, ensemble=False)

Direct Forecasting plus Conformal Prediction Intervals using CrossValidation. [image by the author]

Conformal prediction generates trustable prediction intervals since it’s proved that data and modeling uncertainties are taken into account in the estimation process. Other methodologies show good responses for disentangling uncertainty sources (an example with linear models is reported in one of my previous articles). However, the adaptability of conformal prediction, and its accessibility through MAPIE, makes the technique a must when approaching uncertainty quantification.

In this post, we discovered the power of conformal prediction for estimating prediction intervals. We focused on a time series forecasting task to add prediction intervals to our predictions. Adding trustable conformal prediction intervals to forecasts generated through recursive or direct forecasting it’s possible and straightforward. Thanks to the combined usage of tspiral and MAPIE it’s possible to make time series forecasting and uncertainty quantification all in one shot simply using scikit-learn.