Systematic Trading research and development, with a flavour of Trend Following
Au.Tra.Sy blog – Automated trading System header image 2

The Cheetah and the Cockroach: a robust story

December 4th, 2009 · 8 Comments · Backtest, Strategies

Anybody researching automated trading systems quickly comes to realise that robustness is very much a required aspect of any system. Robust systems are sturdy and able to withstand all that the markets can throw at them without collapsing in ruin.

To explore this further, let’s study the animal world with the Cheetah and the Cockroach – bearing in mind how they would compare to trading systems.

The Animal Metaphor

A running cheetahA Cheetahs is a highly specialised and elegant predator. It is a fascinating animal, the fastest on this Earth. However, despite this status, its survival is in jeopardy…
A cheetah mostly hunts gazelles (ideal size/speed ratio) in vast expanses of land (so as to use their speed). Changes in their environment (types of prey, habitat, terrain, competition from other predators) makes them lose their advantage (edge) and they struggle to hunt anything and survive.

cockroachWhereas cockroaches suffer from a vile and dirty reputation, they are very tough little critters, being able to survive in a vast array of unhospitable conditions with barely any food (they are rumoured to be able to survive a nuclear war – although that seems to be an urban myth).

Back to Trading Systems

You will have guessed it: the Cockroach “trading system” is more likely to be robust than the Cheetah “trading system”. Today it might not be pretty compared to the Cheetah, but the Cockroach can adpat to changing conditions. In 100,000 years, it will still probably be around unlike Cheetahs.

What this means is that you might back-test a system that looks very exciting (a Cheetah – maybe a curve-fitted backtested system) but chances are: it will not survive very long, unlike a trading system that might look a “bit dull” in comparison but offers great robustness (the Cockroach).

How do you know if your trading system is robust

As we could see robustness in a trading system means that changing the conditions under which the system runs should not have significant impact on its performance. In practice this means that by changing either of:

  • Date range: all the returns should not come from a specific period
  • Type of markets: if it works on Grains and Metals, it should also work on Interest Rates and Currencies
  • Parameters of the trading decision tools (i.e. indicators): if a MA of 20 is chosen, a MA of 18, 19, 21 or 22 for example should not alter system performance significantly

the characteristics of the trading system should stay similar

To illustrate this point in the real world, listen to Michael Clarke’s (a Trend Following Wizard) description of his trading programs:

CCM’s philosophy when developing a model is to test it using a large pool of commodity interests, approximately 105, with data as far back as 1945. In order for the model to be accepted into CCM’s portfolio of available models, it must trade all 105 markets using the same rules and parameters and the results should indicate excellent performance characteristics for the vast majority (at least 90%) of the markets and for the group of markets as a whole. Also no model is accepted unless it shows stability of performance during tests involved with shifting parameters and altering rules. Much effort has been expended in developing tools to assist in this effort to assure robustness of the models. I consider the software we have developed in this area to be one of our edges in the markets.

In closing

I hope this metaphor helps you thrive for robustness: Don’t try to build a Cheetah, better build a Cockroach! It will have better chances of survival – even though it does not look too pretty.

I believe a trading system robustness is pivotal to its success; and will carry on with a series of posts covering this topic: How to measure robustness? How to make your system robust? Can walk-forward be robust?

PS: Just as I was getting ready to publish this post, I saw this post from Mebane Faber relaying a study from BlackStar funds showing that long-time survivor CTAs all exhibit a Sharpe ratio below 1 (not so attractive) whereas all Sharpe ratios above 1 “disappear” after time. Quite a useful chart proving the point! All the cockroaches seem to out-live the cheetahs…

Related Posts with Thumbnails

Tags: ···

8 Comments so far ↓

  • Riskcog

    A possible corollary is that the best performing systems over a given time frame are likely to be cheetahs in some respect. Do you have any examples of Cockroach strategies in mind from books or elswhere?

    In 2008 among diversified mutual funds PRPFX came out on top. This may be because it is based on research into how to survive civil unrest, depressionnary economic collapse, hyperinflation etc… Maybe an example of a cockroach within the genre that fund occupies.

  • Jez

    @cog
    no specific examples, this was just rather a reflexion on how to evaluate/backtest trading systems (ie it is easy to lure yourself with a back-tested strategy that looks good in the past but which will not be robust enough to survive in the long term). Trend following seems to be such robust strategy (and hence not attractive at all times, big drawdowns, etc. But that’s the “price to pay” for robustness.

  • Milktrader

    The opinion that a system ought to work across all markets is a view that Murray Ruggiero holds, unlike Bob Pardo who believes that some markets are better suited to a trading system than others. But the view that a single parameter set should work across all markets seems a bit much, quite frankly.

    There is also another class of systems known as intermarket systems that are not generic market systems, but specifically tailored to a particular relationship between correlated markets. Once I get my Bug Collection (generic market systems) backtested, optimized and walk-forward, I’ll be focusing on this field as I find it particularly intriguing.

  • Jez

    Milk,
    There are quite a few strategies (especially technical analysis ones) that are supposed to work with the same parameters on all markets, actually most of the basic ones I think (they might not be that succesful though…)- look at Tom Demark ones for example (if you have started the book ;-) I am pretty sure a Donchian Channel breakout system wuld perform relatively well on a wide collection of sectors.
    I guess some could argue that overall trading market personality/psychology should apply in the same way for any instrument whereas I could also understand the point of view that different markets each have their character…
    Jez

  • Milktrader

    I did start DeMark and I’m intrigued. It’s a generic market type of system that includes potential for counter-trending, which could be useful in conjunction with LTTF systems.

    DeMark is interesting to me because it uses price action and is dynamic (support/resistance are based on recent levels)

  • Jez

    DeMark is intriguing indeed… I feel quite ambivalent about it – the method (TD Sequential for example) sounds one of the most curve-fitted (in the way the rules are so specific with excat numbers for SetUp, CountDown, etc.) so I am quite uncomfortable with this. However I wanted to back-test it as an addition to a LTTF system (ie to anticpate trend reversals and take partial profits) but the indicators are a a bit of a project in itself to code up. As I am not convinced of the robustness of the approach, it has dropped down the “list of my priorities…”

    I met a prop trader a few months ago and he mentioned how he loved using DeMark and wished that he’d found out about it so I thought I might investigate that further… Just heard from him a few weeks ago.. And he’s looking for a job! ;-)

  • Milktrader

    Perl mentions that he has not backtested TD Sequential, so I guess it’s up to us. I found some Amibroker code but not TradersStudio yet.

    I think there is something there with DeMark, though I am also suspicious that maybe there isn’t. You can convince yourself you see something when their is nothing there, and that’s what I ask myself when I follow along with the indicators. (I downloaded the code for MetaTrader and have taken a look on how it acts on the Euro).

    Your friend’s lack of trading success may have less to do with DeMark and more to do with other things that contribute to the lack of success. What I love about trading systems is that you can throw the code out into the public with almost no fear that anyone will trade it properly.

  • Nizar

    Hi Jez,

    Just a few thoughts.

    First of all I agree on your points re:Robustness.

    Your mention of cheetahs reminded me of a quote by Mark Weinstein (A high percentage trader profiled in Market Wizards) which has no direct relevance to this post but here it is:

    Although the cheetah is the fastest animal in the world and can catch any animal on the plains, it will wait until it is absolutely sure it can catch its prey. It may hide in the bush for a week waiting for just the right moment. It will wait for a baby antelope, and not just any baby antelope, but preferably one that is also sick or lame. Only then, when there is no chance it can lose its prey, does it attack. That, to me, is the epitome of professional trading – Mark Weinstein.

    My opinion on a system having to work well on all markets is as follows. If a system does perform well in other markets besides that which it was designed and tested on, then yes, that does add an element of robustness.

    However, if it does not, this does not take anything away from the system. Why?
    Because some markets have inherently different characteristics in terms of the way they behave.

    For example, from my experience with equity systems, I have found the US markets (the S&P500) are mean reverting in nature, while the ASX is more trend following.

    Nizar.

Leave a Comment