A few months ago, I got quite interested when Trading Blox announced that they introduced a new walk-forward functionality in their latest version. I just got round to upgrading, and giving that walk-forward testing a go. Amongst other things, some of the chart features have been improved – as can be seen in the eye candy above.
You can check this previous post for an explanation of how walk-forward works, from a general point of view. Robert Pardo (usually credited with the invention of walk-forward) has also written his bookfew books on the subject.
This new feature is a combination of small enhancements to the core application (ie. silent tests, dynamic date and starting equity settings, etc.) and a semi-custom script implementing the walk-forward testing procedure. It feels more like a (good) hack leveraging the core stepping functionality, rather than a functionality built from the ground up, but it does the job rather well (I looked at the underlying code and its ingenuity is actually pretty cool).
A minor issue is that you have to work out the number of optimization/out-of-sample cycles. If you get that wrong, the test will not cover the desired time interval and it makes the process less automated. Apart from that, the settings are pretty straight-forward, you can choose lengths of optimization and out-of-sample phases as well as other options:
At the end of the optimization phase of each cycle, the system picks the best set of parameters to be used in the out-of-sample phase. However there are many ways to determine the best system. Do you use raw CAGR, MAR, Sharpe ratio, etc.?
There is a need to define the objective function (also called bliss function, as explained here, which will be used to determine the best system performance.
Trading Blox calculates standard statistics (MAR, CAGR, Drawdown, etc.) which can all be used as the objective function, but also allows the implementation of custom statistics calculations. These custom statistics can then be used as the objective function.
This is where you realise the functionality is not fully integrated into the app. The system shows the individual results of each out-of-sample runs:
The stats calculated relate to each out-of-sample run (1 year in my example), whereas ideally these should all be spliced together, to form one system with its own stats. Same applies to equity curves, which are charted year by year (as individual charts). It is still possible to access the Daily Equity log file to do this yourself but it would be nice to have it better automated. The fact that it is a script and not compiled core code might make it possible to customize it to implement this automation.
Another log file also allows you to investigate the results of each optimization run and check their performance:
Starting optimization run ,225,2005-01-03, to ,2010-01-01, with starting equity ,166470686.069881530, Run ,225, of ,256, Stepped Parameters: , Name,Step Value,Goodness Measure of ,64.497285767, Run ( Index ),8.000000000, Optimization Run,1.000000000, Entry Breakout (days),20.000000000, Exit Breakout (days),10.000000000, Starting optimization run ,226,2005-01-03, to ,2010-01-01, with starting equity ,166470686.069881530, ----------------------------------------, Best goodness of ,64.497285767, was on run ,1.000000000, ----------------------------------------, Starting out of sample test ,241,2010-01-01, to ,2010-09-07, with starting equity ,166470686.069881530,
For my first “test ride” of the walk-forward functionality, I used the good ol’ Donchian system.
The system parameters being tested are the Entry and Exit breakout lengths ranging from 20 to 50 days for the entry and 10 to 25 days for the exit.
The parameters for the walk-forward procedure are a 5-year optimization phase and 1-year out-of-sample phase with the plain CAGR stat being used as the objective/bliss/goodness function. The dates of the test go from 1998 to 2010 (but bear in mind that the first out-of-sample result is from 2003, ie. 1998 to 2002 are used as first optimization period).
The results (year by year) are the ones shown in the section above.
For an interesting comparison, I ran a standard stepped test using the same Donchian system and parameter values. Check below how the walk-forward system performance stacks up against the range of outputs from the stepped test:
In hindsight, it would have been better to pick other systems that performed better, but the advantage of the walk-forward process is that it adapts itself to pick a system.
The walk-forward functionality is definitely not a fully-fledged feature of Trading Blox (yet?) – however it does allow for some automation of the process. The fact that it is “open” is probably a good thing as it can be tinkered with and improved easily. A nice addition to the product…