Master Chef Paul Fraser - photo: Peter-Duke@flickr

Well, “building” your bliss function is a bit like making up your own recipe: you first have to choose the main ingredients (return, variance, risk, etc.) and mix them in a way that makes sense and in the right proportions. Most great chefs perfect their recipes through trial and error. The process can equally be applied to the bliss function.
 
Please note that my bliss function recipe is not finished. I am merely floating the idea of how I am planning to establish my bliss equation. Please feel free to comment if you have any comments on super ingredients you know about: at the Au.Tra.Sy restaurant, everybody can help in the cuisine!

Main ingredients and proportions

First, you have to decide what will be in your recipe. The base ingredients will probably be return, variance and risk. These can be declined in several flavours which you must choose. In the previous bliss function post, we looked at the following comparison of 2 different systems:

MAR = CAGR / MaxDD (MaxDD = worst drawdown)

This led us to conclude that neither CAGR nor MAR were a good performance and variance measure on their own. From that observation came the idea of using:

ln(CAGR + 1) x MAR

We then added some more ingredients to complete the recipe such as:

Bliss = ln(CAGR + 1) x MAR - 0.5 x ME - 0.1 x TimeInMkt%

This would be our base recipe. The process here is just to use common sense (based on your preferences) to build a tentative bliss function equation

tinkering with ingredients and proportions

This is the stage where great recipes are made!

If you set out to invent a dish, you’ll pick up the ingredients that you want to use and mix them with common sense (as we have done above). However, your first attempt will surely need some adjustments (that chocolate cake never turns perfect the first time!)

Test your equation on any system equity curve and stats that you can put your hands on. Use your back-testing software output, real-life fund performance reports, or just make them up: you just want some data to run your equation on.

You should have some ideas of which systems you prefer naturally. Your mission is simply to run the equation on all these systems and make sure that the bliss function ranks them as per your natural preferences. If it does not, you need to tinker with the recipe/equation.

What caused the divergence between the function output and your natural choice? How can you adjust the formula to rectify this? You’ll probably have to change some coefficients, add some smoothing functions, add a minor ingredient, add some extra logic, etc.

The downside: it is a long, on-going process, maybe a process that does not appear very scientific: but we are not trying to find a universal truth here – rather your own truth. And unless you can formulate it right off the bat, you have no alternative but trial and error

An extra ingredient: robustness

That could be chef Jez Liberty’s signature ingredient – that makes people love or hate that recipe…

What we want to measure here is how the system is robust. We’ve seen that there are multiple ways to define and measure robustness. For this, let’s concentrate on robustness to slight parameter variation. The process would be three-fold (assuming you are already happy with one robustness-less bliss function equation):

• Run a backtest of your system with different parameter values
• Calculate the bliss value for each of the sub-system/parameter value generated in your backtest
• Establish a robustness score for each sub-system/parameter value based on the variance of its own bliss value compared to the bliss values of its neigbouring parameter values

A picture will illustrate this more clearly. Imagine that the following is a chart of our bliss values (for each parameter value tested):

The 16th parameter value produces the best bliss value, but its neighbouring values are pretty low: its robustness score would penalise it. On the other hand, parameter #11 seems fairly close to its four closest neighbours and it should get a positive score.

One robustness coefficient (Rob), calculated for each parameter value, could be:
Rob = sd(all) / sd(param,n_neigh)
where:
sd(param,neigh) would be the standard deviation of the bliss values across the parameter observed and its n neighbours (e.g. n=4 would consider a group of 5 bliss values)
sd(all) would be the standard deviation across all parameters

The new robust bliss value would be adjusted by Rob such as:
Blissnew = Blissold x Rob

Of course you would have to tinker with this Rob formula, possibly change the formula a fair bit. But this gives you a base to start working with and test against sample backtest results.

one main problem

That robustness measure is only good intra-system, for taking into account which parameter sets are more robust.

This does not allow to compare multiple systems from a robustness point of view. But for this, you just have to start the same thought process again: how do you want to measure robustness for a given system and compare it against another one?

A starting point might be to apply some variations to the trading systems, maybe a 10% variation on the parameter values, a bit of light price “scrambling” (i.e. update your price series by adding 0.1% to the close), etc. Measure the variance of the bliss values across these variations for different systems and update the bliss value by this variance score.

the proof in the pudding?

At the end of the day, the bliss function is trying to aggregate many aspects of a trading system (performance, variance, risk, robustness, etc.) into one single value. The concept is attractive but the recipe might fail to capture all the subtleties of the ingredients.

Imagine mixing together a banoffee pie with ratatouille nicoise, an entrecote, some Maroilles cheese and a Corn and Clam Chowder all rinsed down with some Kaoliang… An extreme comparison, you may say but you get the idea (ps: try some of these dishes – individually – they are very nice!)

I wanted to test the MMDI as a follow-up to Moving Median: a better indicator than Moving Average?. This also provided a good opportunity to test Trading Blox (which I am thinking of buying).

MMDI: What is it?

In short, this is an indicator very similar to the MACD, except that the short moving average of the MACD is replaced by a moving median.

Portfolio Filter: Trade with the trend

One concept often used to improve the edge of a trading system is to look at 2 or more timeframes. The main timeframe (shorter one) is used for triggering trading signals (eg Donchian Channel breakouts), and the longer timeframe is used to determine the direction of the main trend. The filter rules prevent any trade signal to be taken if it goes against the main trend.

Trading Blox: a componentized testing framework

A great feature of Trading Blox is that it provides you with a skeleton workflow that forms the framework for the backtesting process. What this means is that Trading Blox implements and runs its logical workflow in the simulation loop (ie read data, update indicators, check entry signals, check exit signals, post-simulation scripts, etc.) but provides you with hooks at every step (about 35 hooks per simulation loop) where you can write your own code for customisation (with access to Trading Blox internal objects).

Next is the concept of blocks, which represent the different components of a trading systems (Entry signals, Money Management, Risk Management, Portfolio Filter, etc.). These blocks are easily reusable in any system and implement the functionality required via the code contained in their scripts.

One such block we are interested in for today is the MACD Portfolio Filter:

MACD Portfolio Filter Blox

This block stops the system from opening new trades in the opposite direction to the trend (the direction of the trend is derived from the MACD value).

MMDI Portfolio Filter

It was easy to use the standard Donchian channel system that ships with Trading Blox and replace its MACD Portfolio filter block by an implementation of the MMDI Portfolio filter. All it took was a copy of that block and an update of some of the scripts to implement the MMDI indicator and the filtering based on its value.

Indicator calculation:

mmdiIndicator=Median(ohlcDiv4,mmdiShort)-emaIndicator

Filtering code (long side):

[...]
' If positive, then allow long trades
IF ( mmdiIndicator > 0 ) THEN
	instrument.AllowLongTrades
ENDIF
[...]

And applying the new block (MMDI Portfolio Filter) to the system in the system editor screen:

System Editor

Test Scenario

The test is a comparison of the standard Donchian Channel breakout Tend Following system with MACD Portfolio Filter against its variation using the MMDI Portfolio Filter.

In order to get more data points (and to test Trading Blox parameter stepping), the comparison was run over a combination of system parameters:
- Long MMDI Moving Average: 200, 250 and 300
- Short MMDI Moving Median: 50, 62 and 74
- Donchian Channel Length (Entry): 20, 30 and 40
- Donchian Channel Length (Exit): 15
- Stop level: 2 x ATR(40)
- Slippage: 15% of ATR (+3% at rollover)
- Commissions: $12.50 per contract
- Dates: 01/01/2001 to 09/30/2010
- Instruments: 28 liquid futures (currencies, commodities, financials)

Test Results

Here are the two results tables produced by Trading Blox, showing a few stats for each system tested:

MACD Portfolio Filter Results - click to expand

MMDI Portfolio Filter results - click to expand

Looking at the CAGR and Sharpe ratio on aggregate, here is the how the systems compare:

Conclusions

First on Trading Blox: it was fairly straight-forward to code up this new indicator and system. The stepped parameter tests were also really quick to run (<1 min). Still pretty pleased and feeling at ease with it.

The test results show a small improvement in the MMDI favour. A possible explanation might be that the more volatile nature of the moving median (as illustrated in the moving median indicator post) allows it to pick up changes in trend faster (and get in them at an earlier, better price).

The whipsawing produced in the Moving Median crossover run would normally take place during range-bound markets, where few Donchain breakouts would happen, therefore cancelling the extra noise and losses associated with them.

Might be worth investigating further…
 
 
PS: David’s code for MMDI on TradeStation is available for free on his dvindicators.com website

PS2: As a comparison, and to illustrate the impact of a portfolio filter, here are the results for the same system without any Portfolio Filter:

Only 3 sets of results as the MACD/MMDI stepped parameters have been removed - click to expand

Without the trend filter applied to the portfolio, the Donchian channel breakout system now exhibits a negative performance (-2.68% on average across the 3 backtests). The trend is definitely your friend!

Robustness in trading is a tough beast to tame and understand. The more “robust” the research and development process, the better (read: robust) the results ought to be, right? With this in mind, I decided to test robust “tools” within the actual mechanical trading strategy itself.

The moving average indicator is so ubiquitous in trading that most folks (me included) use it without second thoughts. Its legacy probably dates from the era of expensive and complicated computing (it is relatively inexpensive to compute), so I wanted to revisit its hegemony – and give it a run for its money: by pitching it against a moving median indicator (on the basis of better statistical robustness for the latter).

Could it be that a moving median is actually a better indicator than the moving average?…

The experiment

To find out I used a basic and simple mechanical trading strategy: the Moving Average Crossover. This trading systems is always in the market, buys when the fast moving average crosses over the slow moving average and sells short when the fast average crosses under the slow average.

The second system would be a Moving Median Crossover. You guessed it: the same system, but replacing the average by the median.

The markets tested were a random collection of 17 Futures daily prices (proportionally back-adjusted contracts) – all going back as far as CSI history goes (1920′s for Wheat!):

 
The money management for both systems is to trade each instrument in a separate independent sub-account, fully funded (i.e. no leverage used) with profit re-invested. All commissions or slippage are ignored.

The main interest of the experiment is the robustness of each indicator. To quantify this, each system is run over 9 combinations of parameters for the Golden Cross (fast indicator values: 45, 50 and 55 days; slow indicator values: 180, 200 and 220 days). A measure of the robustness of the indicator is the uniformity of the results over the 9 combinations.

The results

Below are the total returns for both systems over each parameter set:

 
At first glance, it appears that the Moving Median indicator significantly under-performs the standard Moving Average indicator for these crossover systems.

Let’s look a bit deeper at the results with some basic statistical analysis:

 
The calculation confirms the under-performance of the Moving Median Crossover system. What about robustness you ask? Well, the Moving Median still scores worse than the Moving Average on both measures of uniformity/dispersion: the standard Coefficient of Variation (0.64 v 0.68) and its alternative cousin based on Median and Median Absolute Deviation (0.36 v 0.45): The Moving Average System produces more uniform (robust?) results!

Below is also a histogram of all 288 individual returns (per market per parameter combination, i.e. Wheat 180/45, Wheat 200/50, Silver 200/50, etc.):

There is clearly more blue presence on the left side of the chart and more red one on the right side…

A potential explanation

Using some inductive logic (warning: this might be dangerous when dealing with data from Extremistan^*), I started eye-balling the charts in search for some clues as to why the Median under-performs the Average. Below is an example of what I found:

Crossovers for Cocoa 2004-2010

The chart above only shows Moving Averages and Moving Medians (the prices have been removed to make the picture clearer). Average and Median seem to closely follow each other both on slow and fast sides. Indeed the bulk of the trades take place at roughly the same time (i.e. they both detect large trends fairly similarly).

However, if we zoom in over that red-circled congested area:

Zoomed-in portion of Cocoa chart

We can see that the Median Crossover system generates more signals than the Moving Average one (9 v 5). Trend following systems notoriously make big bucks in large moves but lose money in trend-less, range-bound markets – like the one being zoomed into. If the Moving Median Crossover system is more active in these sort of markets it will generate more losing trades while capturing similar big winners to the Moving Average Crossover system.

Intuitively, it could be hypothesized that the Moving Average evolves in a smoother manner and will generate smoother curves with less erratic moves and consequently less losing trades during trend-less markets – while the Moving Median does not generate a significant edge in detecting large trends.

One test is hardly enough to provide siginificant evidence, however this should give us some insights in the nature of the Moving Median indicator. The first insights being no increase in robustness and a drop in performance (when comparing total returns).

 
 
^*Extremistan: concept popularised by Nassim Taleb to describe the “province” where the total can be conceivably impacted by a single observation (e.g. financial data, wealth distribution). The opposite is Mediocristan: the province dominated by the mediocre, with few extreme successes or failures. No single observation can meaningfully affect the aggregate (e.g. human height and weight distribution). The bell curve is grounded in Mediocristan. There is a qualitative difference between Gaussians and scalable laws, much like gas and water.

For more info on Taleb’s terms, check his Fooled by Randomness glossary (PDF).

However there are different ways to look at robustness in a mechanical trading system.

Different definitions for robustness

Trading system robustness usually implies that the system exhibits similar performance when subject to slight variations. However thinking about it, you realise that this can mean several things:

• Robustness to future prices (the survival aspect)
• Robustness to internal changes (i.e. variation in system parameters)
• Robustness to external changes (i.e. variation in price data)
• Robustness in system design
• Robustness in system testing

Also check that Trading Blox forum post for more discussion on robustness (some of which directly inspired ideas in this post).

There are 2 assumptions that can be made:

The first: a system robust to changes (internal and external) will likely be robust to future prices: this is because markets always evolve and changes – and your system should be able to cope with these changes.

The second one is that creating a system using robust concepts and procedures, both in design and testing, should ensure that the system will show robustness.

Robust system design

This generally starts with a trading strategy that makes sense, with few parameters, not many bells and whistles (we do not want the Ferrari that looks good and go fast round the track but which can not go over speed bumps or underground car parks… Well, I do really…but you get the point! ;-)

Another aspect to consider when aiming to build a robust trading system is illustrated by this quote from Bill Eckhardt who aims to use robust tools and components in his systems:

We developed all our systems exclusively in-house so there aren’t any sort of publicly recognizable indicators I could mention. We definitely use non-linear systems and non-linear indicators. Linear indicators, such as filters with moving averages, have been mined dry.

In statistics, the median is a more robust tool than the mean/average (it is les subject to underlying data changes such as outliers). Possibly, using median instead of average in a trading strategy is more robust. A moving “median” crossover system would show more robustness than using a classic moving average crossover system – this is something I am planning to test out.

Robust system testing

The main aspect of robust system testing is to ensure that the back-test is realistic and that no over-fitting takes place. We will not go too much in detail as these have been well documented (I recommend Perry Kaufman’s book which has a whole chapter on System Testing and robustness).

Important points are good quality data, in-sample vs. out-of-sample data, realistic assumptions (costs, slippage, etc.), logic in the strategy.

Robustness to internal changes

This is when you change the parameters of the trading system. It is probably fairly easy to test and measure. Assume you have a Donchian Channel Breakout system with Channel length at 20 days and ATR-based stop at 30 days with a multiplier of 2.

A robust system would exhibit very similar performance with slightly different parameters: the robustness could be quantified by measuring the overall difference/standard deviation in system’s performance when varying parameters (for example in a range of +/- 10%), e.g. Donchian Channel Breakout system with Channel length at 22 days and ATR-based stop at 33 days with a multiplier of 1.8, etc.

Robustness to external changes

Here, this is mainly the “parameters” of the price data that can be amended to test for robustness. There are quite a few things that can be changed such as the set of instruments traded (small permutations should not affect performance drastically), testing period (system performs similarly over all time periods), or even random slight amendments in actual prices.

Robustness to future prices

Unfortunately future price data is pretty hard to come by (CSI are pretty good but they only provide historical data! Feel free to email me if you have found a good provider ;-), so you obviously cannot test your system for future price robustness before putting it live.

Arguably, the aim of focusing on robustness with the four previous points is to ensure that the system performs equally well on future prices. However these four robustness aspects in a trading system can only act as a proxy for future prices robustness.

Although we, as trading system developers, like to gain some sort of “comforting” certainty, we have to keep in mind that the future will be unpredictable, markets will change, and therefore designing a system to trade future prices will always contain some degree of uncertainty. It is up to us to decide (and predict) what parameters and proxies are important to ensure robustness in the future.

This might sound like a paradox, but designing an automated trading system involves more discretion and prediction than one would think.

To explore this further, let’s study the animal world with the Cheetah and the Cockroach – bearing in mind how they would compare to trading systems.

The Animal Metaphor

A Cheetahs is a highly specialised and elegant predator. It is a fascinating animal, the fastest on this Earth. However, despite this status, its survival is in jeopardy…
A cheetah mostly hunts gazelles (ideal size/speed ratio) in vast expanses of land (so as to use their speed). Changes in their environment (types of prey, habitat, terrain, competition from other predators) makes them lose their advantage (edge) and they struggle to hunt anything and survive.

Whereas cockroaches suffer from a vile and dirty reputation, they are very tough little critters, being able to survive in a vast array of unhospitable conditions with barely any food (they are rumoured to be able to survive a nuclear war – although that seems to be an urban myth).

Back to Trading Systems

You will have guessed it: the Cockroach “trading system” is more likely to be robust than the Cheetah “trading system”. Today it might not be pretty compared to the Cheetah, but the Cockroach can adpat to changing conditions. In 100,000 years, it will still probably be around unlike Cheetahs.

What this means is that you might back-test a system that looks very exciting (a Cheetah – maybe a curve-fitted backtested system) but chances are: it will not survive very long, unlike a trading system that might look a “bit dull” in comparison but offers great robustness (the Cockroach).

How do you know if your trading system is robust

As we could see robustness in a trading system means that changing the conditions under which the system runs should not have significant impact on its performance. In practice this means that by changing either of:

• Date range: all the returns should not come from a specific period
• Type of markets: if it works on Grains and Metals, it should also work on Interest Rates and Currencies
• Parameters of the trading decision tools (i.e. indicators): if a MA of 20 is chosen, a MA of 18, 19, 21 or 22 for example should not alter system performance significantly

the characteristics of the trading system should stay similar

To illustrate this point in the real world, listen to Michael Clarke’s (a Trend Following Wizard) description of his trading programs:

CCM’s philosophy when developing a model is to test it using a large pool of commodity interests, approximately 105, with data as far back as 1945. In order for the model to be accepted into CCM’s portfolio of available models, it must trade all 105 markets using the same rules and parameters and the results should indicate excellent performance characteristics for the vast majority (at least 90%) of the markets and for the group of markets as a whole. Also no model is accepted unless it shows stability of performance during tests involved with shifting parameters and altering rules. Much effort has been expended in developing tools to assist in this effort to assure robustness of the models. I consider the software we have developed in this area to be one of our edges in the markets.

In closing

I hope this metaphor helps you thrive for robustness: Don’t try to build a Cheetah, better build a Cockroach! It will have better chances of survival – even though it does not look too pretty.

I believe a trading system robustness is pivotal to its success; and will carry on with a series of posts covering this topic: How to measure robustness? How to make your system robust? Can walk-forward be robust?

PS: Just as I was getting ready to publish this post, I saw this post from Mebane Faber relaying a study from BlackStar funds showing that long-time survivor CTAs all exhibit a Sharpe ratio below 1 (not so attractive) whereas all Sharpe ratios above 1 “disappear” after time. Quite a useful chart proving the point! All the cockroaches seem to out-live the cheetahs…