Comments on: How can Walk-Forward testing keep your system a step ahead?

By: Jez Liberty

Jez Liberty — Tue, 05 Oct 2010 08:50:04 +0000

Agree with that point Alex, however the ideal Walk-Forward process would only have one start date for indicator calculation (ie start of the data being tested). For each optimisation cycle, you would look at the performance of all the systems (which started from the beginning of the test) but only for that period of optimisation (ie letting all systems/parameter combinations run in parallel from the beginning and looking at them in separate time-windows (corresponding to the optimisations and out-of-sample phases). You would not reactivate each system for each different cycle. Unfortunately this is what TB does and one reason it is not ideal.

By: Alex

Alex — Fri, 01 Oct 2010 14:57:53 +0000

With start-date dependency I mean the following: consider a double exponential moving average system; if you start to compute the fast and the slow EMA at a certain date you get some entry and exit signals. If you start to compute on a month later for example, with the same parameter, you get different signals and different results as well. In walk forward analysis you have this problem every cycle.

By: Jez Liberty

Jez Liberty — Thu, 30 Sep 2010 17:35:44 +0000

Hi Alex – thanks for the comment.
Not 100% sure I understand your question but for walk-forward, I would have a collection of paramter combinations I want the system to run with. All these combinations would run from the start of the back-test and at each interval period their performance would be measured and the best performing ones would be chosen for trading…

By: Alex

Alex — Thu, 30 Sep 2010 14:48:48 +0000

Hi Jez, I like your blog.
Walk forward is intresting but how can you deal with start date dependency? I mean, with a simple EMA cross system, when do you start to compute the fast and slow lag for each cycle?

By: dew

dew — Thu, 29 Jul 2010 00:26:37 +0000

This blog entry helped me finally understand Walk Forward Testing. The picture alone is worth a thousand words.

Much appreciated.

Also, I referenced this blog post on the NinjaTrader forum

By: Jez

Jez — Thu, 12 Nov 2009 17:37:23 +0000

That sounds definitely interesting as I’ll probably move into shorter timeframes at some point. And its always good to hear about problems and solutions to learn the platform..

By: Dave

Dave — Thu, 12 Nov 2009 14:16:40 +0000

Jez, This is a point in Amibroker that may make one work a little harder in CBT. In regular backtester, Tomasz closes all trades at end of OOS period. There is no carryover. For longer EOD runs this may not be critical. But it plays havoc with shorter periods. Bruce Robinson has work around started using “state” function…but it requires some manipulation. I may be able to dig up Bruces AFL if you’re interested.

By: Jez

Jez — Fri, 06 Nov 2009 17:05:29 +0000

Milk,
I think there are an infinite number of bliss functions…
The bliss function serves the purpose of quantify your trading Nirvana (bliss, nirvana: I am staying in the same theme here…).

The bliss function is your OWN way to rank how well or bad the performance of a system has done – based on your own personal parameters. For example, profit might be proportinally more important to you than drawdowns.. Why not use
(net profit)^2 / Max DD (or 2 * ln(CAR) – ln(MaxDD) ) – ie you put more emphasis on profit
but then you could be more fancy adding more parameters in the equation and possibly some logic:
if CAR > 15% then 2 * ln(CAR) – ln(MaxDD) – ln(Pct. time in market) – 3 * ln(average portfolio heat)
if CAR < 15% then 2 * ln(CAR) – 3 * ln(MaxDD) – ln(Pct. time in market) – 5 * ln(average portfolio heat)
etc.

I find Ed Seykota's website pretty good (I'll probably add it to my blogroll) and you could check that link:
http://seykota.com/tribe/faq/2004_Oct/Oct_11/index.htm
look for bliss function – it raises other questions too (ie net profit/Max DD can give you a value of 8 with 4% profit and .5% DD. Is that better than 40% profit and 10% DD?)

But in any case, I am not sure how you could add a predictive element to it – that does not seem to fit its purpose (which is to quantify with one single number how good the back-test results are). Now I think it might be possible to define a robust approach which allows to pick the parameter set that might perform best in the future (ie compare all optimisation runs and their bliss functions but do not necessarily pick the highest number – potentially a spike – but rather a local stable maximum).

By: Milktrader

Milktrader — Fri, 06 Nov 2009 16:03:44 +0000

Yes, objective = fitness = bliss function. It is the arbiter of best parameters. And I would suggest they are predictive of the robustness of a system. We probably would doubt the value of Net Profit as a good judge of best parameters. Net Profit/ Max Drawdown is better. What is better than that?

Perhaps, and this is just a thought, there is a way to measure how well a fitness function picks the best parameter set.
Perhaps th

By: Jez

Jez — Fri, 06 Nov 2009 09:50:12 +0000

That point about comparing perfect profit with the system performance (equity) would be one interesting way of answering the question “Is the system not performing any more or are the markets not offering any opportunities?” (ie if performance is not good but perfect profit stays good, the system might have an issue)

Regarding the metric for the effectiveness of a fitness function (I take it you mean objective/bliss function) – I am not convinced this is a valid approach as I dont see the objective function’s role to have any “predictive” power (i.e. it should just be a way to express the overall performance of a system under your ideal criteria). Although I see your point that this is the input used to select which parameter set are carried forward (and therefore some sort of predictability value might be useful)…

Or maybe I mis-understood your point: would you have an objective function as the output metric and a fitness function trained (by neural nets) to determine the best predictability between the optimisation and verification test (ie the fitness function would be used to work out the best/most robust parameter set between optimisation and walk-forward back-test, ie where the ratio of objective functions are most constant and ideally fairly high). This sounds a bit complicated.

I am not so keen on neural networks anyway (black-box approach vs KISS, etc.)