Out-of-Sample testing is a necessary practice to avoid curve-fitting during the optimisation of a trading system. Walk-Forward testing improves on the idea of out-of-sample data testing and is designed as an on-going, adaptive approach. Its invention is mostly credited to Robert Pardo (read more about it in his book)
The way it works is fairly simple. It is a combination of multiple cycles of “in-sample optimisation” with “out-of-sample verification”.
Background on optimisation and out-of-sample testing
The reason for performing out-of sample verification tests is to check whether the in-sample data optimisation resulted in curve-fitting (over-optimisation) or in a robust system parameters selection.
If the parameters derived from optimisation perform much worse in the out-of-sample verification test, it most likely means that the parameter values were (over-) optimised for the specific in-sample dataset (curve-fitted). If the system performs similarly, it should mean that the system parameters are robust and validate the approach taken for optimisation.
Walk-Forward Process: how it works
Walk-Forward testing is an on-going and dynamic process to determine whether parameters optimisation just curve fits the price and noise or produces statistically valid out-of-sample results. Here is how it works:
Let’s say we have 10 years of data from 1999 to 2009. Optimisation period is three years (in-sample data) and Verification period is one year (out-of-sample data). To begin, you start by optimising your system using only the first three years of data – in this example, 1999-2001. When the system is optimised, record the optimal parameter values and use them in the test with new data (out-of-sample) starting with 2002.
Slide the three-year window of data forward (2000-2002) and perform the same process. Once you have processed all the data available, you can collate the performance of all out-of-sample tests and compare those to in-sample optimisation runs. If the comparison shows that the system is sufficiently robust to be traded live, you simply continue the walk-forward process in real time by re-optimising every year.
Walk-Forward is an adaptive process which re-optimises the system on a continuous basis to adapt its parameters to the most recent market conditions.
The premise of performing several optimisation/verification steps over time is that the recent past is a better environment for selecting system parameters than the distant past. This is an assumption you need to consider when choosing whether to use Walk-Forward testing or not but this is a useful tool in your Systems development arsenal.
As discussed earlier changing system parameters based on recent market conditions could result in a system chasing its tail. The next post on this topic will be an actual comparison of a basic system’s performance when optimised in a “standard way” and when optimised using “Walk-Forward”.