Comments on: Robustness? What robustness?

By: Jing

Jing — Sun, 07 Nov 2010 23:47:20 +0000

I think your suggestion is very reasonable. This system is a pull-back system. It follows the long-term trend, and wait the market has a pullback, then enter the market when the price exceed previous level. From my observation of 2000-2010 currency market, for each significant resistance level, there will be many limit/stop orders around it. The result is if the price successfully surpass a significant resistance, it has a higher chance to continue higher. But it seems this is not the case in 1990-2000. In that period, many times the price just exceed an important resistance by a small amount and then quickly revert back. So this system didn’t performance well at that time.
I am not sure whether future price patterns will be like in 2000-2010, so I may prefer a system which can perform well in both time periods even with lower performance on 2000-2010. Your suggestion to keep developing and monitoring the system to try and adapt to changing conditions is very helpful! Thanks!

By: Jez Liberty

Jez Liberty — Sun, 07 Nov 2010 23:09:13 +0000

Well – the problem with this kind of approach is that you are anticipating that the markets will be similar in the future to the 2000-2010 period. I think I prefer an approach that can be “robust” to whatever conditions the market might throw at you, which includes a return to the 1990-2000 conditions. Usually that would mean lower performance on 2000-2010 period though…
Now, even in system development there is some discretion involved and if you believe that conditions will stay as they are, you might want to stay with a system that works well only on the recent market conditions. You could also potentially keep developing and monitoring your system to try and adapt it to changing conditions.

By: Jing

Jing — Sun, 07 Nov 2010 02:02:50 +0000

Hi, I once researched a system for forex. It can make money in each of 7 major currency pairs from 2000 to 2010 with very small maximum drawdown. Even after changing the parameters somewhat, it is still very profitable. The problem is it doesn’t perform well from 1990 to 2000′s currency market, and it is just breakeven during that time. I guess the reason is the introduction of Euro and the market price pattern changed somewhat after 2000, so this system works well only at 2000-2010. Do you think this kind of systems which can only perform well on recent 10-year period is robust enough and should I confidently use it?

By: George

George — Fri, 15 Jan 2010 14:48:03 +0000

Jez,

Yes, that makes sense. I would like to design an experiment or measure the”assumption that changes in the future will be similar”.

By: Nizar

Nizar — Sun, 20 Dec 2009 02:08:06 +0000

Jez,

I agree with your concluding statement that systems testing and design is far from an exact science.

Though we cannot test on “future” data, I think out-of-sample testing is as good a tool as possible.

In addition to Kaufman’s excellent book, I would also recommend Robert Pardo’s Design, Testing, and Optimisation of Trading Systems.

Turns out Robert Pardo is also a top trader.
http://www.pardocapital.com/docs/PardoCapital10yearRanking.pdf

Nizar.

By: Jez

Jez — Thu, 17 Dec 2009 22:52:34 +0000

@milk – good idea about these price variations. As long as you are inventive (as you did with your excellent breakfast spread ;-), you might get off the “beaten tracks” and find something interesting.

@George, I think I was trying to expres a similar concept: if the system you develop is robust to changes (such as parameter changes or price, instruments changes) it might be robust on future prices (as these will always change)… But this contains an assumption that changes in the future will be similar (in terms of impact on your system) to the changes you test on your system during development.
In terms of measuring some robustness, I would think you can design an objective function to evaluate your trading systems and measure the standard deviation of this objective function when subjecting your system to changes.

By: George

George — Thu, 17 Dec 2009 15:53:50 +0000

It is an interesting idea to break down robustness into components. I am not sure if I think about it in the same way.

Here’s how we think about it. Robustness is the ability of the system to perform well in changing conditions.

Since the future conditions are never exactly the same as past conditions, a more robust system will make money in real trading than a less robust system.

So, robustness is a measure of the likelihood that the system will perform similarly in real trading compared to development.

Your analysis does bring up an interesting question. If we can make our system robust to external changes, then how do we measure those changes?

By: Milktrader

Milktrader — Thu, 17 Dec 2009 13:05:52 +0000

Nice article on robustness, a term that has been used so much and so loosely that it has started to lose its meaning.

Also, that was an interesting quote about how one trader has abandoned moving averages and develops his indicators in-house. Your idea about median vs mean is an interesting one. Another variable that can be tweeked (and should withstand some stress) is the definition of price. We normally use the close as the definition of price. Buy why not the high or the low, depending on how the market is trending? Or how about the concept of average price (HLC/3)?