I recently found a 30-page paper introducing the ideas and principles of his Leverage Space Model. I thought reading it might be a good way to get back into Vince’s material.

Follows a summary of the paper (PDF download link), which is maths-less (only concepts and principles are discussed). It is a good introduction to Ralph Vince theories.

Vince’s Optimal f

This is what Vince is famous for. It is basically a way to determine trading quantity (aka leverage) using the probability distributions of the trade outcomes.

If f represents the fraction of capital to wager (risk) on each bet (trade), the optimal value is the one which optimises the geometric growth of the bankroll (account balance).

In his previous books, Vince has defined the formula to determine the optimal f. The first part of the paper discusses the optimal f concept and is a good introduction for the non-initiated (showing how over-betting on a game with positive expectancy can and will result in a loss).

Leverage Space Model promises

The optimal f section discusses a single-component approach whereas the Leverage Space Model deals with multiple components portfolio.

It is presented as an improvement on the Modern Portfolio Theory, briefly discussed. This is based on the following advantages:

1. Risk is defined as drawdown (instead of variance in the MPT)
2. The fallacy and danger of correlation is eliminated
3. Valid for any distributional form – Fat-tails are addressed
4. The model is all about Leverage, which is not addressed in the MPT model.

Return aspect

The model starts by building a multi-dimensional terrain, drawing the overall expected return, based on multiple combinations of components in the portfolio and their respective f-values.

In this example the model builds the terrain for 2 simultaneous coin-toss with a payoff of 2:1. The x and y axis represent the respective f-values (leverage) for each of the bets/trades - while the z-axis (vertical) represents the expected return

The maximum portfolio growth is located at the peak of the terrain, resulting from the specific corresponding f-values combination. The terrain construction does not take into account correlation between the instruments – instead, the model uses the joint probability of two scenarios occurring simultaneously, dictated by the price data history.

The Risk Aspect

So far, the model has only looked at returns. To introduce the risk component, you must determine your maximum allowed drawdown. This is a hard and fast rule: no combination should breach that limit.

Using a derivation of the risk of ruin, the model computes the risk of maximum drawdown for each set of f-values (for a specific timeline – as, in the long run, the risk of drawdown tends to 100%). If the risk of drawdown is too high, the specific f-values combination is ignored.

In practice, the initial terrain is truncated: by removing all points breaching the maximum drawdown threshold.

The terrain has been truncated: all areas deemed too risky, from a drawdown perspective, have been removed.

The Algorithm

Vince implements a genetic algorithm to calculate the terrain, by initially calculating the expected return for each set of f-values, and secondly by running the maximum drawdown test on this same set. Once the whole set combination has been run through, the terrain is built (including truncations). The the aim is then to find the optimal set (highest return with lowest f-values).

Comments and Extra Info

The paper is rather short and does not deal with any of the maths behind the models. For this you’d have to get yourself a copy of the Handbook of Portfolio Mathematics which introduces the model in more detail or Vince’s latest book, dedicated to the Leverage Space Model, which has had a not-so-positive review by Max Dama.

The ideas in the paper are an interesting take on position sizing. Vince uses a simple objective/bliss function (CAGR with a binary risk/drawdown filter) to evaluate all possible scenarios of portfolio allocation/leverage. It might be interesting to use the concepts of the model with your own bliss function recipe.

One of Vince’s claim that the MPT does not address leverage sounds a bit simplistic – surely the percentage of Cash as an asset in the portfolio is an implicit measure of leverage. On the other hand, the approach on correlation/joint probability of scenarios sounds interesting and seems to go in the right direction. As Vince says:

Counting on correlation fails you when you need it the most.

Another point that seems missed out is how the model handles non-stationarity of the market. Vince mentions the chronomorphism of market prices distributions (i.e. they change with time) and even draws a betting comparison with blackjack – in which the optimal f curve changes for each card dealt. However there is no mention of how the model takes an adaptive approach to these chronomorphic distributions.

Vince’s homepage contains a link to the java software that implements his model (needs to register/leave email to download) and another one with a spreadsheet example. I have not had time to take a serious look at all those. Please let me know your feedback if you do.

Joshua Ulrich – blogger and reader of this blog (hello there: finally got round to adding you to the blogroll!) – is collaborating with Ralph Vince to port the Leverage Space Model to the R platform. His FOSS Trading blog is definitely worth a read too.

Dennis did so to set up an old argument with fellow trader Bill Eckhardt on whether trading could be taught or not (not unlike the story in classic movie Trading Places).

The experience was succesful in proving Dennis right (trading could be taught), with his turtle traders making him $100 million.

Why turtles?

The Turtles were nicknamed as such because of an analogy with how Richard Dennis expected to “grow” traders in the same way Singapore farms grow turtles. Dennis taught his students a mechanical Trend Following system and let them trade with his own capital. After being kept secret for more than a decade, the rules were revealed and floated on the internet for a while. Two enjoyable books have now been published on the topic (Complete Turtle Trader – featuring the actual turtle rules and The Way of the Turtle written by Curtis Faith, a former Turtle) if you are interested in learning more about it.

The turtle system

The Turtle system did not contain “magical” components. It was basically a combination of 2 different breakout systems with specific rules for money management, including position sizing, pyramiding, correlation limits and cutting down position size during drawdowns (a quick “Turtle trading rules” google search should yield some results for the exact rules).

System kaput?

Now that the rules have been made public, it is possible to backtest them and see how they would have performed on the recent markets. Such test result can be found on the Trading Blox forum.

click to zoom in

It basically shows that the CAGR drops from 216% (!!) from 1970 to 1986 (when Dennis and Eckhardt were developing the system, and also when the students traded the system with real money) to barely double digits (10.5%) in the last 23 years (1986 to 2009), with a completely flat period from 1996 to 2009.

As a side note check the crazy amount that compounding generates at that rate of 216%!

One could argue that Dennis & co were just lucky to trade the system during what appears to be its golden period.

System overheating warning

During the Turtle experiment, Dennis came to the realisation that their position sizing rules were such that:

you have been trading as much as twice as big as we thought

Here is a snapshot of the memo that Dennis sent to all traders asking them to cut their position size in half.

As Dennis said:

We must be living right

Another way of saying “We have been very lucky”?…

What does this mean?

Well, some say that the Turtle performance was a fluke – that the Turtles were actually the proverbial monkeys writing Hamlet (see the Infinite Monkey Theorem). I guess these people would be in the EMH camp (Efficient Market Hypothesis).

Some say that Trend Following is dying/dead and the Turtle system under-performance is an illustration of that. Problem is: these people seem to celebrate the demise of “simpleton” Trend Following strategy every so often (during major drawdown peiods), only for Trend Following to come back roaring again (think 2008).

Some might also say that market conditions are changing, and systems need to adapt to these changing conditions.

Although, some also say that this argument is specious, citing Bill Dunn as an example of a CTA claiming to use the same rules as when he started in the 70′s.

Reconciling it all

Instead of concluding this post quickly here – on what it does or does not means – I thought it might be more interesting to expand and spend more time on the possible interpretations above in a post of its own. Part 2 will take this discussion further.

Stay tuned, I will try to touch on whether or not Trend Following should, and can, adapt to changing market conditions – update: post here.

Master Chef Paul Fraser - photo: Peter-Duke@flickr

Well, “building” your bliss function is a bit like making up your own recipe: you first have to choose the main ingredients (return, variance, risk, etc.) and mix them in a way that makes sense and in the right proportions. Most great chefs perfect their recipes through trial and error. The process can equally be applied to the bliss function.
 
Please note that my bliss function recipe is not finished. I am merely floating the idea of how I am planning to establish my bliss equation. Please feel free to comment if you have any comments on super ingredients you know about: at the Au.Tra.Sy restaurant, everybody can help in the cuisine!

Main ingredients and proportions

First, you have to decide what will be in your recipe. The base ingredients will probably be return, variance and risk. These can be declined in several flavours which you must choose. In the previous bliss function post, we looked at the following comparison of 2 different systems:

MAR = CAGR / MaxDD (MaxDD = worst drawdown)

This led us to conclude that neither CAGR nor MAR were a good performance and variance measure on their own. From that observation came the idea of using:

ln(CAGR + 1) x MAR

We then added some more ingredients to complete the recipe such as:

Bliss = ln(CAGR + 1) x MAR - 0.5 x ME - 0.1 x TimeInMkt%

This would be our base recipe. The process here is just to use common sense (based on your preferences) to build a tentative bliss function equation

tinkering with ingredients and proportions

This is the stage where great recipes are made!

If you set out to invent a dish, you’ll pick up the ingredients that you want to use and mix them with common sense (as we have done above). However, your first attempt will surely need some adjustments (that chocolate cake never turns perfect the first time!)

Test your equation on any system equity curve and stats that you can put your hands on. Use your back-testing software output, real-life fund performance reports, or just make them up: you just want some data to run your equation on.

You should have some ideas of which systems you prefer naturally. Your mission is simply to run the equation on all these systems and make sure that the bliss function ranks them as per your natural preferences. If it does not, you need to tinker with the recipe/equation.

What caused the divergence between the function output and your natural choice? How can you adjust the formula to rectify this? You’ll probably have to change some coefficients, add some smoothing functions, add a minor ingredient, add some extra logic, etc.

The downside: it is a long, on-going process, maybe a process that does not appear very scientific: but we are not trying to find a universal truth here – rather your own truth. And unless you can formulate it right off the bat, you have no alternative but trial and error

An extra ingredient: robustness

That could be chef Jez Liberty’s signature ingredient – that makes people love or hate that recipe…

What we want to measure here is how the system is robust. We’ve seen that there are multiple ways to define and measure robustness. For this, let’s concentrate on robustness to slight parameter variation. The process would be three-fold (assuming you are already happy with one robustness-less bliss function equation):

• Run a backtest of your system with different parameter values
• Calculate the bliss value for each of the sub-system/parameter value generated in your backtest
• Establish a robustness score for each sub-system/parameter value based on the variance of its own bliss value compared to the bliss values of its neigbouring parameter values

A picture will illustrate this more clearly. Imagine that the following is a chart of our bliss values (for each parameter value tested):

The 16th parameter value produces the best bliss value, but its neighbouring values are pretty low: its robustness score would penalise it. On the other hand, parameter #11 seems fairly close to its four closest neighbours and it should get a positive score.

One robustness coefficient (Rob), calculated for each parameter value, could be:
Rob = sd(all) / sd(param,n_neigh)
where:
sd(param,neigh) would be the standard deviation of the bliss values across the parameter observed and its n neighbours (e.g. n=4 would consider a group of 5 bliss values)
sd(all) would be the standard deviation across all parameters

The new robust bliss value would be adjusted by Rob such as:
Blissnew = Blissold x Rob

Of course you would have to tinker with this Rob formula, possibly change the formula a fair bit. But this gives you a base to start working with and test against sample backtest results.

one main problem

That robustness measure is only good intra-system, for taking into account which parameter sets are more robust.

This does not allow to compare multiple systems from a robustness point of view. But for this, you just have to start the same thought process again: how do you want to measure robustness for a given system and compare it against another one?

A starting point might be to apply some variations to the trading systems, maybe a 10% variation on the parameter values, a bit of light price “scrambling” (i.e. update your price series by adding 0.1% to the close), etc. Measure the variance of the bliss values across these variations for different systems and update the bliss value by this variance score.

the proof in the pudding?

At the end of the day, the bliss function is trying to aggregate many aspects of a trading system (performance, variance, risk, robustness, etc.) into one single value. The concept is attractive but the recipe might fail to capture all the subtleties of the ingredients.

Imagine mixing together a banoffee pie with ratatouille nicoise, an entrecote, some Maroilles cheese and a Corn and Clam Chowder all rinsed down with some Kaoliang… An extreme comparison, you may say but you get the idea (ps: try some of these dishes – individually – they are very nice!)

But what is “best”?

How do you measure best results, though? Compound Annual Growth Rate (CAGR) is one of the first metric that comes to mind (it is the simplest and most direct one). However consider this:

System A produces a greater CAGR – at the cost of much larger variability in the results. I don’t know about you, but “to a point”, I would rather obtain a smoother equity curve and slightly worse CAGR (i.e. choosing System B).

This reasoning leads to a bunch of other metrics and ratio dealing another measure: the variance of the return (Sharpe ratio, Sterling ratio, Sortino ratio, Treynor ratio, Lake ratio, Calmar and MAR ratios, etc.).

Introducing YOUR Bliss function

In short, there are many ways to quantify how good a system is. One of the first step in the optimization process is to decide on the objective function – which will be used to measure and compare the performance of each system tested in the optimization run (the Sharpe ratio is one such objective function, widely used to compare fund managers).

Ed Seykota coined the term Bliss Function, which suits very aptly: the system that performs best on your set of criteria will give you the most blissful results (wealth increase, peace of mind).

Now consider this:

MAR = CAGR / MaxDD (MaxDD = worst drawdown)

System A is very consistent but its return is very unexciting. There might be a threshold where you are willing to accept more risk variance for more reward (i.e. choose to trade system B despite its lower return-to-variability ratio value, like the MAR chosen here). And this is the main point of building an objective/bliss function:

The bliss function should be your own “goodness measure” of a trading system, using your own formula and criteria. Choose a formula that will produce the best results/pick the best systems for you. And you get to be as original as you want!

For the case above, you might decide to complicate the formula of the Bliss function by adding an arbitrary adjustment multiplying function AM to penalise low-returns systems such as:

Bliss = AM(CAGR) x MAR

with:
AM(x<10%) = 0 (ignore any systems producing less than 10%)
AM(10%<=x<20%) = 0.5 (penalising low-returns systems)
AM(20%<=x<60%) = 1 (standard value)
AM(x>=60%) = 1.5 (rewarding high-returns systems)

A continuous function with the same impact would be the logarithmic function.

Variance != Risk

The shortcut taken by many is to equate return variance with risk. If that is the case, any of your preferred ratio above would be an elegant way to measure risk-adjusted returns.

However, risk and variance are two complete different animals (LTCM had an outstanding Sharpe ratio... until they blew up!)

So, how can you quantify hidden risk? Raw performance return does not allow you to identify it. It really depends on what you consider as Risk.

Hidden risk can be contained in all of these aspects of a strategy that expose you to "risk situations" (think Black Swans) even if they do not materialise.

Time in the Market might be relevant here: all things being equal, you might prefer a strategy in the market 30% of the time rather than 75% - less chance to be hit by this 6-sigma event).

Leverage has a direct influence on the quantitative impact of such event. You might want to introduce measures such as average margin-equity ratio (ME) into your Bliss formula.

Adding the 2 components with arbitrary coefficients the formula would become:

Bliss = ln(CAGR + 1) x MAR - 0.5 x ME - 0.1 x TimeInMkt%

additional parameters

So far, we have covered fairly standard metrics for trading systems. But you might want to add your own specific criteria.

One such criteria, which could be interesting to add, is some form of "robustness measure". This will be the subject of a follow-up post.

In Closing

Coming up with your own bliss function is basically mixing up the various metrics of your trading system (absolute performance, variance, risk and possibly other factors such as robustness) that matter to you. This post tries to highlight the concept and thought process rather than a complete or exact Bliss function.

To expand on the concept of the bliss function, concepts such as walk-forward testing can leverage the bliss function and use it as the objective/selective function.

One can argue that there are as many bliss functions as there are traders. The difficulty lies in finding and formulating its formula...

However there are different ways to look at robustness in a mechanical trading system.

Different definitions for robustness

Trading system robustness usually implies that the system exhibits similar performance when subject to slight variations. However thinking about it, you realise that this can mean several things:

• Robustness to future prices (the survival aspect)
• Robustness to internal changes (i.e. variation in system parameters)
• Robustness to external changes (i.e. variation in price data)
• Robustness in system design
• Robustness in system testing

Also check that Trading Blox forum post for more discussion on robustness (some of which directly inspired ideas in this post).

There are 2 assumptions that can be made:

The first: a system robust to changes (internal and external) will likely be robust to future prices: this is because markets always evolve and changes – and your system should be able to cope with these changes.

The second one is that creating a system using robust concepts and procedures, both in design and testing, should ensure that the system will show robustness.

Robust system design

This generally starts with a trading strategy that makes sense, with few parameters, not many bells and whistles (we do not want the Ferrari that looks good and go fast round the track but which can not go over speed bumps or underground car parks… Well, I do really…but you get the point! ;-)

Another aspect to consider when aiming to build a robust trading system is illustrated by this quote from Bill Eckhardt who aims to use robust tools and components in his systems:

We developed all our systems exclusively in-house so there aren’t any sort of publicly recognizable indicators I could mention. We definitely use non-linear systems and non-linear indicators. Linear indicators, such as filters with moving averages, have been mined dry.

In statistics, the median is a more robust tool than the mean/average (it is les subject to underlying data changes such as outliers). Possibly, using median instead of average in a trading strategy is more robust. A moving “median” crossover system would show more robustness than using a classic moving average crossover system – this is something I am planning to test out.

Robust system testing

The main aspect of robust system testing is to ensure that the back-test is realistic and that no over-fitting takes place. We will not go too much in detail as these have been well documented (I recommend Perry Kaufman’s book which has a whole chapter on System Testing and robustness).

Important points are good quality data, in-sample vs. out-of-sample data, realistic assumptions (costs, slippage, etc.), logic in the strategy.

Robustness to internal changes

This is when you change the parameters of the trading system. It is probably fairly easy to test and measure. Assume you have a Donchian Channel Breakout system with Channel length at 20 days and ATR-based stop at 30 days with a multiplier of 2.

A robust system would exhibit very similar performance with slightly different parameters: the robustness could be quantified by measuring the overall difference/standard deviation in system’s performance when varying parameters (for example in a range of +/- 10%), e.g. Donchian Channel Breakout system with Channel length at 22 days and ATR-based stop at 33 days with a multiplier of 1.8, etc.

Robustness to external changes

Here, this is mainly the “parameters” of the price data that can be amended to test for robustness. There are quite a few things that can be changed such as the set of instruments traded (small permutations should not affect performance drastically), testing period (system performs similarly over all time periods), or even random slight amendments in actual prices.

Robustness to future prices

Unfortunately future price data is pretty hard to come by (CSI are pretty good but they only provide historical data! Feel free to email me if you have found a good provider ;-), so you obviously cannot test your system for future price robustness before putting it live.

Arguably, the aim of focusing on robustness with the four previous points is to ensure that the system performs equally well on future prices. However these four robustness aspects in a trading system can only act as a proxy for future prices robustness.

Although we, as trading system developers, like to gain some sort of “comforting” certainty, we have to keep in mind that the future will be unpredictable, markets will change, and therefore designing a system to trade future prices will always contain some degree of uncertainty. It is up to us to decide (and predict) what parameters and proxies are important to ensure robustness in the future.

This might sound like a paradox, but designing an automated trading system involves more discretion and prediction than one would think.

In the last few posts I have been exploring the e-ratio and wrote the code in TradersStudio. I also tried to implement it in AmiBroker AFL language mostly to check how it performed speed-wise – as I found TradersStudio quite slow.

While testing AmiBorker, it actually felt so fast that I decided to perform a more formal speed comparison test.

Conditions of the test

I exported the CSI data via the API for a proportionally back-adjusted Corn contract going back to 1949. The data was imported in both AmiBroker and TradersStudio via their ASCII Import.

The system tested is a simple 20-day Donchian Channel Breakout (Buy-only) and the ATR used to normalise the MAE/MFE is 20-day also.
The optimiser in both systems were used to generate the e-ratio for 50 51 different trade durations (from 10 to 50 days).

One of the condition to validate the results of the test was that the trades generated by both systems are similar (to double-check I did not make a coding mistake resulting in simpler/faster process for one of the platforms).

The computer I was running the test on is a quad-core CPU (2.4 GHz) with 3.25 GB of addressable RAM. Only one platform was running at the time it was tested.

Results are in!

And they look astoninglishly good for AmiBroker:

• The TradersStudio test was run first and took 4 mins 15 sec to complete and produce the custom report (which needs to be manipulated in Excel to calculate the e-ratio). Furthermore, it appeared that every incremental run in the optimisation process took slightly longer than the previous one.
• The AmiBroker test ran in (…drumrolls…) 10 sec! And the e-ratio was readily available from the results grid:
  
  

  AmiBroker optimisation results showing the e-ratio. Click to expand

Both apps maxed out their allocated CPU (i.e. overall CPU usage of 25% = a quarter of the quad-core CPU available).

I then decided to run the AmiBroker code over 5 markets to get a feel of how long it would take. I added Crude Oil (going back to 1985), Cotton (1968), Gold (1975) and Yen (1972).
AmiBroker ran the same code + optimisation in 2 mins 20 sec, twice as fast as one market in TradersStudio!

Applying a simple proportional calculation to derive the time it would take in TradersStudio for the same dataset would give us a completion time of 1 hour (59 min 30 sec exactly…). Since the performance seems to degrade over the course of the optimisation in TradersStudio (i.e. the first back-test is quicker than the last one of the optimisation run) we could assume that it would actually take longer, which is what I experienced when I was testing on similar data in the last few weeks.

So here you have it: AmiBroker is 25 times faster than TradersStudio and it gives you the results in a much more friendly format.

Trade Reconciliation

As mentioned earlier, to ensure that the test was valid, the set of trades generated by both systems should be compared. I ran one back-test of the system over the Corn data for a single optimisation step (e-ratio for 20 days duration) in both systems.

I could find that 296 trades out of 340 presented a very good match (some matches had one day difference on the Entry date and rounding differences on the prices). But overall the trade comparison was good enough to give assurance that the same systems and results were being tested.

I’ll keep investigating for myself where these errors are coming from (and post a further code update if warranted) but we can assume that the differences are trivial with regards to performance comparison.

Did I mention that AmiBroker is 3 times as cheap with a much wider following (many more code samples, better docs, more forums, etc.)?…

So I decided to give the “legendary fast” Amibroker a test to see if it could better TradersStudio’s performance. After some “playing and learning”, I have finalised the code to compute the e-ratio. My big thanks go to the ASX gorilla whose own version forms a large part of my code.

Below is the code explanation and downloadable afl file.

Directly from the ASX Gorilla’s website as a prelude to the code:

My implementation of the Edge Ratio involves two profound Amibroker fudges. The first is the use of the AddToComposite function to create a composite ticker symbol in which to hold the ATR array of a given stock for later retrieval within the Custom Back Tester via the Foreign function.

The second fudge is the use of the VarSet/VarGet function to create a quasi array. This was necessary to overcome the limitation where array elements cannot exceed in number the value of barcount-1.

The first part is to actually code up your Buy signal (in our case a Donchian Channel Breakout). The Sell signals are tested separately:

//BUY RULES: implemented with Buy Stop on Upper Donchian Channel(17)
BuyStop = Ref(HHV(High, 17),-1);
Buy = Cross( High, BuyStop );
BuyPrice = Max( BuyStop, Low ); // make sure buy price >= Low

Exiting positions is done on a fixed duration basis:

//Never Sell so that the position is stopped out after N bar instead
Sell = 3 > 5;
//Stop the positon and close it after N bars
//(eratio = N that we step from 1 to 100 in optimisation)
ApplyStop( stopTypeNBar, stopModeBars, eratio );

The first fudge mentioned above to store the ATR:

Normaliser = ATR(17);
AddToComposite(Normaliser, "~atr_"+Name(), "C", 1+2+8);

And the “meat” of the code: the chunk that implements the custom back-testing to:

1. Loop through the signals and store the Entry ATR value.
2. Loop through all trades and retrieve MFE, MAE and ATR.
3. Compute the e-ratio based on values from all trades.

SetCustomBacktestProc(""); //activate the custom backtester
if(Status("action") == actionPortfolio) //called when backtesting/optimising
{
	bo = GetBacktesterObject();
	bo.PreProcess(); // run default backtest procedure
	TradeATR = NumTrades = ATRArr = 0; //init variables
	for( bar=0; bar < BarCount-1; bar++)
	{
		bo.ProcessTradeSignals(bar);
 
		for ( sig=bo.GetFirstSignal(bar); sig; sig=bo.GetNextSignal(bar) )
		{
			if (sig.isEntry())
			{
				NumTrades++;
				ATRArr = Foreign("~atr_"+sig.Symbol, "C");
				VarSet("TradeATR" + NumTrades, ATRArr[bar]);
 
				_TRACE("Symbol " + sig.Symbol + " ATR: " + VarGet("TradeATR" + NumTrades));
			}
		}
	}
 
	AvgMAE = AccumMAE = AvgMFE = AccumMFE = NumTrades = 0;
 
	// iterate through closed trades
	for( trade = bo.GetFirstTrade(); trade; trade = bo.GetNextTrade() )
	{
		NumTrades++;
		EntryATR = VarGet ("TradeATR" + NumTrades);
		if ( EntryATR != 0 )
		{
			_TRACE("EntryATR: " + WriteVal(EntryATR));
			_TRACE("AccumMAE : " + WriteVal(AccumMAE));
			AccumMAE = AccumMAE + (trade.GetMAE()*trade.EntryPrice/(100*EntryATR));
			AccumMFE = AccumMFE + (trade.GetMFE()*trade.EntryPrice/(100*EntryATR));
		}
 
		trade.AddCustomMetric("My MAE", trade.GetMAE()*trade.EntryPrice/100);
		trade.AddCustomMetric("My MFE", trade.GetMFE()*trade.EntryPrice/100);
		trade.AddCustomMetric("Entry ATR", EntryATR*10000);
	}
 
	AvgMAE = AccumMAE / NumTrades;
	AvgMFE = AccumMFE / NumTrades;
 
	_TRACE(WriteVal(AccumMAE ));
	_TRACE(WriteVal(NumTrades));
	_TRACE(WriteVal(AvgMAE));
 
	Eratio = abs(AvgMFE/AvgMAE);
 
	_TRACE(WriteVal(Eratio));
 
	bo.AddCustomMetric( "Avg MAE", AvgMAE );
	bo.AddCustomMetric( "Avg MFE", AvgMFE );
	bo.AddCustomMetric( "Eratio", Eratio);
 
	bo.PostProcess();
}

If you want to run this code, you can download the e-ratio “gorilla” afl file and simply update the BUY signals to whatever you fancy testing.

The next post will be a direct speed comparison between TradersStudio and Amibroker for computing the e-ratio on the same underlying data and with the same signal. I expect Amibroker to win the fight hands-down as it appeared “way” faster!

The way it works is fairly simple. It is a combination of multiple cycles of “in-sample optimisation” with “out-of-sample verification”.

Background on optimisation and out-of-sample testing

The reason for performing out-of sample verification tests is to check whether the in-sample data optimisation resulted in curve-fitting (over-optimisation) or in a robust system parameters selection.

If the parameters derived from optimisation perform much worse in the out-of-sample verification test, it most likely means that the parameter values were (over-) optimised for the specific in-sample dataset (curve-fitted). If the system performs similarly, it should mean that the system parameters are robust and validate the approach taken for optimisation.

Walk-Forward Process: how it works

Walk-Forward testing is an on-going and dynamic process to determine whether parameters optimisation just curve fits the price and noise or produces statistically valid out-of-sample results. Here is how it works:

Let’s say we have 10 years of data from 1999 to 2009. Optimisation period is three years (in-sample data) and Verification period is one year (out-of-sample data). To begin, you start by optimising your system using only the first three years of data – in this example, 1999-2001. When the system is optimised, record the optimal parameter values and use them in the test with new data (out-of-sample) starting with 2002.

Walk Forward from 1999 to 2009

Slide the three-year window of data forward (2000-2002) and perform the same process. Once you have processed all the data available, you can collate the performance of all out-of-sample tests and compare those to in-sample optimisation runs. If the comparison shows that the system is sufficiently robust to be traded live, you simply continue the walk-forward process in real time by re-optimising every year.

In Closing

Walk-Forward is an adaptive process which re-optimises the system on a continuous basis to adapt its parameters to the most recent market conditions.

The premise of performing several optimisation/verification steps over time is that the recent past is a better environment for selecting system parameters than the distant past. This is an assumption you need to consider when choosing whether to use Walk-Forward testing or not but this is a useful tool in your Systems development arsenal.

As discussed earlier changing system parameters based on recent market conditions could result in a system chasing its tail. The next post on this topic will be an actual comparison of a basic system’s performance when optimised in a “standard way” and when optimised using “Walk-Forward”.