Putting Your Forecasting Model to the Test: A Guide to Backtesting | by Eryk Lewinson | Nov, 2023

Learn how to properly evaluate the performance of time series models through backtesting

Evaluating time series models is not a simple task. In fact, it is quite easy to make serious errors while evaluating forecasting models. While these errors may not break the code or prevent us from obtaining some output numbers, they can significantly affect the accuracy of such performance estimates.

In this article, we will demonstrate how to properly evaluate time series models.

The simplest way to evaluate the performance of a machine learning model is to split the dataset into two subsets: training and test sets. To further improve the robustness of our performance estimate, we may want to split our dataset multiple times. This procedure is called cross-validation.

The following diagram represents one of the most popular types of cross-validation — the k-fold approach. In the case of 5-fold validation, we first divide the dataset into 5 chunks. Then, we train the model using 4 of these chunks and evaluate its performance on the 5th chunk. This process is repeated 4 more times, each time holding out a different chunk for evaluation.