Systematic Trading research and development, with a flavour of Trend Following
Au.Tra.Sy blog – Automated trading System header image 2

Robustness? What robustness?

December 17th, 2009 · 8 Comments · Backtest

Orchid: a robust flower - photo by purplemattfish@flickrStepping back from the story of the cockroach and the cheetah, it seems that robustness is all about survival. For an automated trading system, this means the system “survives and thrives” after you decide to put it “live”.

However there are different ways to look at robustness in a mechanical trading system.

Different definitions for robustness

Trading system robustness usually implies that the system exhibits similar performance when subject to slight variations. However thinking about it, you realise that this can mean several things:

  • Robustness to future prices (the survival aspect)
  • Robustness to internal changes (i.e. variation in system parameters)
  • Robustness to external changes (i.e. variation in price data)
  • Robustness in system design
  • Robustness in system testing

Also check that Trading Blox forum post for more discussion on robustness (some of which directly inspired ideas in this post).

There are 2 assumptions that can be made:

The first: a system robust to changes (internal and external) will likely be robust to future prices: this is because markets always evolve and changes – and your system should be able to cope with these changes.

The second one is that creating a system using robust concepts and procedures, both in design and testing, should ensure that the system will show robustness.

Robust system design

This generally starts with a trading strategy that makes sense, with few parameters, not many bells and whistles (we do not want the Ferrari that looks good and go fast round the track but which can not go over speed bumps or underground car parks… Well, I do really…but you get the point! ;-)

Another aspect to consider when aiming to build a robust trading system is illustrated by this quote from Bill Eckhardt who aims to use robust tools and components in his systems:

We developed all our systems exclusively in-house so there aren’t any sort of publicly recognizable indicators I could mention. We definitely use non-linear systems and non-linear indicators. Linear indicators, such as filters with moving averages, have been mined dry.

In statistics, the median is a more robust tool than the mean/average (it is les subject to underlying data changes such as outliers). Possibly, using median instead of average in a trading strategy is more robust. A moving “median” crossover system would show more robustness than using a classic moving average crossover system – this is something I am planning to test out.

Robust system testing

The main aspect of robust system testing is to ensure that the back-test is realistic and that no over-fitting takes place. We will not go too much in detail as these have been well documented (I recommend Perry Kaufman’s book which has a whole chapter on System Testing and robustness).

Important points are good quality data, in-sample vs. out-of-sample data, realistic assumptions (costs, slippage, etc.), logic in the strategy.

Robustness to internal changes

This is when you change the parameters of the trading system. It is probably fairly easy to test and measure. Assume you have a Donchian Channel Breakout system with Channel length at 20 days and ATR-based stop at 30 days with a multiplier of 2.

A robust system would exhibit very similar performance with slightly different parameters: the robustness could be quantified by measuring the overall difference/standard deviation in system’s performance when varying parameters (for example in a range of +/- 10%), e.g. Donchian Channel Breakout system with Channel length at 22 days and ATR-based stop at 33 days with a multiplier of 1.8, etc.

Robustness to external changes

Here, this is mainly the “parameters” of the price data that can be amended to test for robustness. There are quite a few things that can be changed such as the set of instruments traded (small permutations should not affect performance drastically), testing period (system performs similarly over all time periods), or even random slight amendments in actual prices.

Robustness to future prices

Unfortunately future price data is pretty hard to come by (CSI are pretty good but they only provide historical data! Feel free to email me if you have found a good provider ;-), so you obviously cannot test your system for future price robustness before putting it live.

Arguably, the aim of focusing on robustness with the four previous points is to ensure that the system performs equally well on future prices. However these four robustness aspects in a trading system can only act as a proxy for future prices robustness.

Although we, as trading system developers, like to gain some sort of “comforting” certainty, we have to keep in mind that the future will be unpredictable, markets will change, and therefore designing a system to trade future prices will always contain some degree of uncertainty. It is up to us to decide (and predict) what parameters and proxies are important to ensure robustness in the future.

This might sound like a paradox, but designing an automated trading system involves more discretion and prediction than one would think.

Related Posts with Thumbnails

Tags: ··

8 Comments so far ↓

  • Milktrader

    Nice article on robustness, a term that has been used so much and so loosely that it has started to lose its meaning.

    Also, that was an interesting quote about how one trader has abandoned moving averages and develops his indicators in-house. Your idea about median vs mean is an interesting one. Another variable that can be tweeked (and should withstand some stress) is the definition of price. We normally use the close as the definition of price. Buy why not the high or the low, depending on how the market is trending? Or how about the concept of average price (HLC/3)?

  • George

    It is an interesting idea to break down robustness into components. I am not sure if I think about it in the same way.

    Here’s how we think about it. Robustness is the ability of the system to perform well in changing conditions.

    Since the future conditions are never exactly the same as past conditions, a more robust system will make money in real trading than a less robust system.

    So, robustness is a measure of the likelihood that the system will perform similarly in real trading compared to development.

    Your analysis does bring up an interesting question. If we can make our system robust to external changes, then how do we measure those changes?

  • Jez

    @milk – good idea about these price variations. As long as you are inventive (as you did with your excellent breakfast spread ;-), you might get off the “beaten tracks” and find something interesting.

    @George, I think I was trying to expres a similar concept: if the system you develop is robust to changes (such as parameter changes or price, instruments changes) it might be robust on future prices (as these will always change)… But this contains an assumption that changes in the future will be similar (in terms of impact on your system) to the changes you test on your system during development.
    In terms of measuring some robustness, I would think you can design an objective function to evaluate your trading systems and measure the standard deviation of this objective function when subjecting your system to changes.

  • Nizar

    Jez,

    I agree with your concluding statement that systems testing and design is far from an exact science.

    Though we cannot test on “future” data, I think out-of-sample testing is as good a tool as possible.

    In addition to Kaufman’s excellent book, I would also recommend Robert Pardo’s Design, Testing, and Optimisation of Trading Systems.

    Turns out Robert Pardo is also a top trader.
    http://www.pardocapital.com/docs/PardoCapital10yearRanking.pdf

    Nizar.

  • George

    Jez,

    Yes, that makes sense. I would like to design an experiment or measure the”assumption that changes in the future will be similar”.

  • Jing

    Hi, I once researched a system for forex. It can make money in each of 7 major currency pairs from 2000 to 2010 with very small maximum drawdown. Even after changing the parameters somewhat, it is still very profitable. The problem is it doesn’t perform well from 1990 to 2000′s currency market, and it is just breakeven during that time. I guess the reason is the introduction of Euro and the market price pattern changed somewhat after 2000, so this system works well only at 2000-2010. Do you think this kind of systems which can only perform well on recent 10-year period is robust enough and should I confidently use it?

  • Jez Liberty

    Well – the problem with this kind of approach is that you are anticipating that the markets will be similar in the future to the 2000-2010 period. I think I prefer an approach that can be “robust” to whatever conditions the market might throw at you, which includes a return to the 1990-2000 conditions. Usually that would mean lower performance on 2000-2010 period though…
    Now, even in system development there is some discretion involved and if you believe that conditions will stay as they are, you might want to stay with a system that works well only on the recent market conditions. You could also potentially keep developing and monitoring your system to try and adapt it to changing conditions.

  • Jing

    I think your suggestion is very reasonable. This system is a pull-back system. It follows the long-term trend, and wait the market has a pullback, then enter the market when the price exceed previous level. From my observation of 2000-2010 currency market, for each significant resistance level, there will be many limit/stop orders around it. The result is if the price successfully surpass a significant resistance, it has a higher chance to continue higher. But it seems this is not the case in 1990-2000. In that period, many times the price just exceed an important resistance by a small amount and then quickly revert back. So this system didn’t performance well at that time.
    I am not sure whether future price patterns will be like in 2000-2010, so I may prefer a system which can perform well in both time periods even with lower performance on 2000-2010. Your suggestion to keep developing and monitoring the system to try and adapt to changing conditions is very helpful! Thanks!

Leave a Comment