Comments on: How can Walk-Forward testing keep your system a step ahead?

By: dew

dew — Thu, 29 Jul 2010 00:26:37 +0000

This blog entry helped me finally understand Walk Forward Testing. The picture alone is worth a thousand words.

Much appreciated.

Also, I referenced this blog post on the NinjaTrader forum

By: Jez

Jez — Thu, 12 Nov 2009 17:37:23 +0000

That sounds definitely interesting as I’ll probably move into shorter timeframes at some point. And its always good to hear about problems and solutions to learn the platform..

By: Dave

Dave — Thu, 12 Nov 2009 14:16:40 +0000

Jez, This is a point in Amibroker that may make one work a little harder in CBT. In regular backtester, Tomasz closes all trades at end of OOS period. There is no carryover. For longer EOD runs this may not be critical. But it plays havoc with shorter periods. Bruce Robinson has work around started using “state” function…but it requires some manipulation. I may be able to dig up Bruces AFL if you’re interested.

By: Jez

Jez — Fri, 06 Nov 2009 17:05:29 +0000

Milk,
I think there are an infinite number of bliss functions…
The bliss function serves the purpose of quantify your trading Nirvana (bliss, nirvana: I am staying in the same theme here…).

The bliss function is your OWN way to rank how well or bad the performance of a system has done – based on your own personal parameters. For example, profit might be proportinally more important to you than drawdowns.. Why not use
(net profit)^2 / Max DD (or 2 * ln(CAR) – ln(MaxDD) ) – ie you put more emphasis on profit
but then you could be more fancy adding more parameters in the equation and possibly some logic:
if CAR > 15% then 2 * ln(CAR) – ln(MaxDD) – ln(Pct. time in market) – 3 * ln(average portfolio heat)
if CAR < 15% then 2 * ln(CAR) – 3 * ln(MaxDD) – ln(Pct. time in market) – 5 * ln(average portfolio heat)
etc.

I find Ed Seykota's website pretty good (I'll probably add it to my blogroll) and you could check that link:
http://seykota.com/tribe/faq/2004_Oct/Oct_11/index.htm
look for bliss function – it raises other questions too (ie net profit/Max DD can give you a value of 8 with 4% profit and .5% DD. Is that better than 40% profit and 10% DD?)

But in any case, I am not sure how you could add a predictive element to it – that does not seem to fit its purpose (which is to quantify with one single number how good the back-test results are). Now I think it might be possible to define a robust approach which allows to pick the parameter set that might perform best in the future (ie compare all optimisation runs and their bliss functions but do not necessarily pick the highest number – potentially a spike – but rather a local stable maximum).

By: Milktrader

Milktrader — Fri, 06 Nov 2009 16:03:44 +0000

Yes, objective = fitness = bliss function. It is the arbiter of best parameters. And I would suggest they are predictive of the robustness of a system. We probably would doubt the value of Net Profit as a good judge of best parameters. Net Profit/ Max Drawdown is better. What is better than that?

Perhaps, and this is just a thought, there is a way to measure how well a fitness function picks the best parameter set.
Perhaps th

By: Jez

Jez — Fri, 06 Nov 2009 09:50:12 +0000

That point about comparing perfect profit with the system performance (equity) would be one interesting way of answering the question “Is the system not performing any more or are the markets not offering any opportunities?” (ie if performance is not good but perfect profit stays good, the system might have an issue)

Regarding the metric for the effectiveness of a fitness function (I take it you mean objective/bliss function) – I am not convinced this is a valid approach as I dont see the objective function’s role to have any “predictive” power (i.e. it should just be a way to express the overall performance of a system under your ideal criteria). Although I see your point that this is the input used to select which parameter set are carried forward (and therefore some sort of predictability value might be useful)…

Or maybe I mis-understood your point: would you have an objective function as the output metric and a fitness function trained (by neural nets) to determine the best predictability between the optimisation and verification test (ie the fitness function would be used to work out the best/most robust parameter set between optimisation and walk-forward back-test, ie where the ratio of objective functions are most constant and ideally fairly high). This sounds a bit complicated.

I am not so keen on neural networks anyway (black-box approach vs KISS, etc.)

By: Milktrader

Milktrader — Thu, 05 Nov 2009 21:08:01 +0000

How about we formulate a metric for the effectiveness of a fitness function. Maybe something logarithmic with values between zero and 1. Each fitness function can be treated as its own neural network and graded on its ability to guess the right answer.

Not sure how to do this though.

By: Milktrader

Milktrader — Thu, 05 Nov 2009 21:01:17 +0000

Yes, that is the essence of Pardo’s formula. Another cool idea he has is equity correlation with perfect profit. It shows that a system makes money when the market offers it and may make less when the perfect profit is less.

Instead of creating a composite fitness function, how about we run walk forward under a group of them to determine which illuminates the best about future profitability. That would be quite an undertaking, but an interesting exercise.

By: Jez

Jez — Thu, 05 Nov 2009 20:42:47 +0000

Hi Milk – for some reason your last comment was “caught” in my “spam nets”…

I assume the above PROM formula (from Pardo I believe) would be your objective function to optimise a trading system?

I have not given much thought to an objective function yet but I have always imagined having a “big” formula trying to compromise the good and bads of the trading system, ie you would want to maximise:
- Profit (MAR, CAGR)
- Sharpe ratio
etc.
while you want to maximise:
- average portfolio heat
-margin-to-equity ratio
- drawdowns (ie maximise Ed Seykota lake ratio).
- time in market
etc.

One thing I would like to try is to formulate all these aspects and prioritise them in one formula.

The idea of reducing the good aspects of the trading system (# of wins) and increasing the bad ones (# of losses) in the PROM formula above is quite interesting also – Thanks!

By: Milktrader

Milktrader — Thu, 05 Nov 2009 20:31:51 +0000

Pessimistic Return on Margin (PROM)

Function FF_PROM2 (MarginRequired As Integer)

Dim AdjWin As Double
Dim AdjLoss As Double
Dim MarginValue As Integer
Dim PROM As Double

Dim SlowAve As BarArray
Dim FastAve As BarArray

AdjWin = wins – Sqr(wins)
AdjLoss = losses + Sqr(losses)

MarginValue = MarginRequired

If SlowAve < FastAve Then

PROM = 0

End If

PROM = ((AdjWin * AvgWin) + (AdjLoss * AvgLoss))/MarginValue

'this next relationship is what allows the factor to be passed into report

FF_PROM2 = PROM

End Function