Systematic Trading research and development, with a flavour of Trend Following
Au.Tra.Sy blog – Automated trading System header image 2

Walk-Forward in Trading Blox: Back-Testing Adaptive Trading

September 8th, 2010 · 7 Comments · Backtest, Software

surface-chart

A few months ago, I got quite interested when Trading Blox announced that they introduced a new walk-forward functionality in their latest version. I just got round to upgrading, and giving that walk-forward testing a go. Amongst other things, some of the chart features have been improved – as can be seen in the eye candy above.

How it works

You can check this previous post for an explanation of how walk-forward works, from a general point of view. Robert Pardo (usually credited with the invention of walk-forward) has also written his bookfew books on the subject.

This new feature is a combination of small enhancements to the core application (ie. silent tests, dynamic date and starting equity settings, etc.) and a semi-custom script implementing the walk-forward testing procedure. It feels more like a (good) hack leveraging the core stepping functionality, rather than a functionality built from the ground up, but it does the job rather well (I looked at the underlying code and its ingenuity is actually pretty cool).

A minor issue is that you have to work out the number of optimization/out-of-sample cycles. If you get that wrong, the test will not cover the desired time interval and it makes the process less automated. Apart from that, the settings are pretty straight-forward, you can choose lengths of optimization and out-of-sample phases as well as other options:

In the setting above, there are 8 cycles, each having a 5-year optimization phase and 1-year out-of-sample phase

In the settings above, there are 8 cycles, each having a 5-year optimization phase and 1-year out-of-sample phase

Objective Function

At the end of the optimization phase of each cycle, the system picks the best set of parameters to be used in the out-of-sample phase. However there are many ways to determine the best system. Do you use raw CAGR, MAR, Sharpe ratio, etc.?

There is a need to define the objective function (also called bliss function, as explained here, which will be used to determine the best system performance.

Trading Blox calculates standard statistics (MAR, CAGR, Drawdown, etc.) which can all be used as the objective function, but also allows the implementation of custom statistics calculations. These custom statistics can then be used as the objective function.

Output

This is where you realise the functionality is not fully integrated into the app. The system shows the individual results of each out-of-sample runs:

wf-results

The stats calculated relate to each out-of-sample run (1 year in my example), whereas ideally these should all be spliced together, to form one system with its own stats. Same applies to equity curves, which are charted year by year (as individual charts). It is still possible to access the Daily Equity log file to do this yourself but it would be nice to have it better automated. The fact that it is a script and not compiled core code might make it possible to customize it to implement this automation.

Another log file also allows you to investigate the results of each optimization run and check their performance:

Starting optimization run ,225,2005-01-03, to ,2010-01-01, with starting equity ,166470686.069881530,
 
Run ,225, of ,256, Stepped Parameters: ,
Name,Step Value,Goodness Measure of ,64.497285767,
Run ( Index ),8.000000000,
Optimization Run,1.000000000,
Entry Breakout (days),20.000000000,
Exit Breakout (days),10.000000000,
Starting optimization run ,226,2005-01-03, to ,2010-01-01, with starting equity ,166470686.069881530,
 
 
----------------------------------------,
Best goodness of ,64.497285767, was on run ,1.000000000,
----------------------------------------,
 
Starting out of sample test ,241,2010-01-01, to ,2010-09-07, with starting equity ,166470686.069881530,

Test Results

For my first “test ride” of the walk-forward functionality, I used the good ol’ Donchian system.

The system parameters being tested are the Entry and Exit breakout lengths ranging from 20 to 50 days for the entry and 10 to 25 days for the exit.

The parameters for the walk-forward procedure are a 5-year optimization phase and 1-year out-of-sample phase with the plain CAGR stat being used as the objective/bliss/goodness function. The dates of the test go from 1998 to 2010 (but bear in mind that the first out-of-sample result is from 2003, ie. 1998 to 2002 are used as first optimization period).

The results (year by year) are the ones shown in the section above.

For an interesting comparison, I ran a standard stepped test using the same Donchian system and parameter values. Check below how the walk-forward system performance stacks up against the range of outputs from the stepped test:

walk-forward in yellow

All test results with the walk-forward result in yellow

In hindsight, it would have been better to pick other systems that performed better, but the advantage of the walk-forward process is that it adapts itself to pick a system.

In Closing

The walk-forward functionality is definitely not a fully-fledged feature of Trading Blox (yet?) – however it does allow for some automation of the process. The fact that it is “open” is probably a good thing as it can be tinkered with and improved easily. A nice addition to the product…

Related Posts with Thumbnails

Tags: ·

7 Comments so far ↓

  • Troy S

    The walk forward feature should allow the automatic averaging of the parameter values of say the top 10 strategies/permutations (by CAGR, by Sharpe, etc.) to determine the “best system” after an in-sample optimization. This extra check for system robustness added to the splicing of all out-of-sample run stats into “one strategy” that you mention above don’t seem that hard to do, but are still things that Amibroker (and Trading Blox?) sorely lack… does anyone know of an Amibroker plugin for these features? if not, time to breakout the Amibroker SDK…

  • Jez Liberty

    Troy – yes that would be ideal and adding some sort of robustness measure is something I discussed here
    I’ll try and check how easy/hard it would be to do some extra processing or update the TB script to get there…

  • Scott

    Jez,
    I enjoy your blog. I have built a trading system and am happy with the past results I got the data from tradingblox/csi. My question is how do I move forward and trade it real time? As the historical data is not the same price wise. Do I need to build another module with real price data and executions and have all the price adjustments from the historical system? Your help would be appeciated.

  • Jez Liberty

    Hi Scot – thanks and welcome to the blog.
    I am assuming you’re referring to the fact that continuous contract data gets out of synch because of the stitching algorithm?
    If so you could look at generating CSI continuous contracts with the “Generate Forward” option unticked so that it starts with the most recent contract (ie recent prices are the same) and works its way back towards the past.
    Then your order generation from TB should have the price levels you want.

  • Stefan

    Jez,
    What walk-forward SW do you like; traders Studio? I am looking for one and would appreciate your insight.
    Thanks,
    Stefan

  • Jez Liberty

    Stefan,
    I used TraderStudio but did not like it very much and moved to Trading Blox. The WF testing functionality is not very mature in it, but I don’t regret moving at all….

  • LuckyF00L

    Jez,
    I am just trying to clarify something in my mind. I understand how walk forward testing has straightforward value when we are backtesting an adaptive system but i am slightly unclear about it’s use when it comes to systems that use static parameters. I’ve read a few articles about WF and not a lot of people seem to be stating explicitly how it’s best used in each case. One of the articles i;ve read by Mr Murray Ruggeiro (if i understood correctly) seems to be hinting towards using WF to see whether the optimized parameters come close to the static parameters u get when backtesting over the whole sample of data, as a check for parameter robustness. Can u share ur thoughts on this pls?

Leave a Comment