Yes, that makes sense. I would like to design an experiment or measure the”assumption that changes in the future will be similar”.

I agree with your concluding statement that systems testing and design is far from an exact science.

Though we cannot test on “future” data, I think out-of-sample testing is as good a tool as possible.

In addition to Kaufman’s excellent book, I would also recommend Robert Pardo’s Design, Testing, and Optimisation of Trading Systems.

Turns out Robert Pardo is also a top trader.
http://www.pardocapital.com/docs/PardoCapital10yearRanking.pdf

Nizar.

@George, I think I was trying to expres a similar concept: if the system you develop is robust to changes (such as parameter changes or price, instruments changes) it might be robust on future prices (as these will always change)… But this contains an assumption that changes in the future will be similar (in terms of impact on your system) to the changes you test on your system during development.
In terms of measuring some robustness, I would think you can design an objective function to evaluate your trading systems and measure the standard deviation of this objective function when subjecting your system to changes.

Here’s how we think about it. Robustness is the ability of the system to perform well in changing conditions.

Since the future conditions are never exactly the same as past conditions, a more robust system will make money in real trading than a less robust system.

So, robustness is a measure of the likelihood that the system will perform similarly in real trading compared to development.

Your analysis does bring up an interesting question. If we can make our system robust to external changes, then how do we measure those changes?

Also, that was an interesting quote about how one trader has abandoned moving averages and develops his indicators in-house. Your idea about median vs mean is an interesting one. Another variable that can be tweeked (and should withstand some stress) is the definition of price. We normally use the close as the definition of price. Buy why not the high or the low, depending on how the market is trending? Or how about the concept of average price (HLC/3)?