The last post on “Monkey-Style” trading with random entries and trailing stop exits generated a fair bit of interest, comments and questions. In this follow-up post, I want to explore the concept further and address some of the points raised in the discussions resulting from that study.

Before I get started, I want to clarify that I am not advocating the use of random trading in real-life system design. I simply used random logic, as per David Harding’s quote, to remove any impact from the entry signals and solely focus on exit and money management.

The Other Side of The Coin

David Harding’s quote (“If you put in stops and run your profits and trade randomly you make money; and if you put in targets and no stops, and you trade randomly you lose money. So the old saw about cutting losses and running profits has some truth to it.”) was really two-sided. The previous post looked into the first part of the quote with random entries and trailing stops. Let’s now look at the other part of the quote with random entries and profit targets.

The overall concept of the system is identical to the one tested in the last post: entries in random directions every time a position is closed, with a fixed fractional money management system over a portfolio of diversified futures. The only difference has to do with exits, which are now triggered by a volatility-adjusted take-profit order. At the time of trade entry, the ATR is calculated as a measure of volatility and a limit order is entered at a distance equal to a multiple of the ATR value.

As the test contains a random element, I ran several hundreds of them in order to average them to detect a “central tendency”. The ATR-multiple also stepped through different values from 2 to 10, each of these being run 200 times to generate 1,800 tests in total.

Well, David Harding seems to be proven right again as the average performance of these 1,800 runs generated a near 85% loss over the 20 years of the back-test (and this is excluding commissions – more on that later).

I have run other tests and generated one chart comparing all outcomes. The “Random Entries with Target Profits” equity curve is plotted on that chart further below.

Note that the Win ratio for this system with Profit Targets comes close to 95%. This is logical: all winning trades are exited when they hit their target profit but losing trades are never exited (except at the end of the test); this shows that focusing on single stats like this one can be misleading (how many times do you hear market gurus claiming 90% accuracy rate?).

More Randomness

I decided to “hire another monkey” to help with another instance of a random system: random entries and random exits.

My first monkey is still in charge of “tossing a coin” to decide of the direction of each new trade, but instead of having a deterministic exit, the second monkey decides randomly of the duration of each trade: monkey picks 69 and the system holds the position for 69 bars. I did assist the monkey a bit, by giving him some “hindsight bias”, so that the average trade duration was roughly equal to the trade duration of the systems in the “Random Entries, Trailing Stops” scenario (for example 13 days for 2xATR, 63 days for 5xATR or 220 days for 10xATR).

This is a very random system, which can go either short or long, and we would therefore expect for it to perform neutrally when tested without any trade frictions. And this is exactly what happens. Check the chart further below: the equity curve is a near flat-line. This goes some way to prove that averaging the results of 1,800 runs with random trading logic does generate a result close to the theoretical/expected result. And we can clearly see the impact of a trailing stop exit (positive) and a profit target exit (negative) compared to a full random system.

Retiring the Entries Monkey

Some readers did comment on the last randomness post, pointing out that entries were as important as exits and such good results could also be obtained with random exits and “good entries”. This was my next test: using a standard Moving Average cross-over for entry generation, I left my “exit monkey” in charge of exiting the trades after a random period of time (again, with the help of hindsight, so that the average trade duration for this random system was similar to the equivalent MA cross-over system). Note that the system only entered on the actual cross-over of moving averages (i.e. when a position exits, the system waits for the next cross-over instead of re-entering in the direction of the moving averages).

The system only returned 64% over 20 years, with relatively high volatility, not really displaying more than what could be described as “random oscillations”. Replacing random entries with MA cross-over entries can not conclusively be said to have improved the system, as the signal-to-noise ratio is too high to assert whether the positive performance of MA entries is significant or not.

MA Entries with Profit Targets

Another way to compare random and MA cross-over entries would be to take the “random entries with profit targets” system and replace the entry generation by a moving average cross-over. This is the next (and final) test in this post.

The MA Entries with Profit Target Exits system variation performs very similarly to the one with random entries, even performing slightly worse, by losing more than 90% over the time span of the back-test (20 years). Again not a strong evidence for the superiority of MA entries over random entries.

Putting it All Together

Below is the chart of each of the 5 scenarios tested, exhibiting a wide variation in results obtained.

 
Some pretty clear trends in there. Note that the Drawdown figures are calculated on a monthly basis. Drawdowns based on daily performance would therefore be larger, as mentioned in this post, discussing a short Drawdown paper by David Harding.

Of course, it is not possible to generalise the results of these findings. The entry logic chosen for these tests for example is specific and some other entry logic, with a different average trade duration might have yielded different results.

This was indeed the case in some previous tests discussed on the Trading Blox forum (thanks to Pumpernickel for pointing these threads out in the last post comment), where similar discussions/tests took place. The results found there seemed to indicate profitability for Trend Following entries and random exits (with a slightly different logic) and even some random entry/exit combination (different logic from above again: direction of positions were not random). Check these two threads for more info and results.

Note on averaging and result normalization

Because most of the tests involved some random logic, I deemed it necessary to run a fairly large sample for each case in order to average the results and obtain an overall tendency.
Some readers have pointed out that averaging the results hides the distribution of the outcomes and might hide the fact some the average performance is skewed by some outliers.

These are valid points to raise, which I want to address by displaying the shape of the distribution of CAGR for all instances of the “Random Entries with Trailing Stops” system (with a 2-ATR stop).

There is of course some variation around the average (average CAGR = 17.03% with a standard deviation of 4.96%), but I feel the average is still a good indicator for the central tendency of the system concepts being tested here. I actually decided to check this by estimating the statistical significance of this result by using the bootstrap method. With the null hypothesis that the average return is zero, the p-value for obtaining an average of 17.03% is actually 0.00 (100,000 resamples).

The other aspect of this comparison test is the normalization of results: every family of tests was normalized so that the standard deviation of the monthly returns was identical for all scenarios, allowing for a better, apples-for-apples comparison.

Notes on Slippage and Commissions

I usually do not include trade friction on the tests published on the blog, one of the reasons being to avoid making assumptions on commissions and slippage amounts, which are variable for every trader based on size, broker, market, etc. (and which can definitely have a non-negligible impact as previously discussed in here and here).
So, I usually prefer to leave this out of the equation (at the risk of making the “raw” results less realistic) and let the readers interpret the results based on their friction costs assumptions. Moreover, we are not really testing a system here, we are using a system back-test to check the impact of entries and exits.

Nevertheless, I have re-run the “Random Entries with Trailing Stops” system including commissions ($10 per contract) and slippage (5%). Of course, these extra costs have an impact: the CAGR drops from 16.14% to 10.16%, as can be seen in the chart below:

And here are some additional figures that should help estimate trading costs:

 
I hope this (long) post has clarified some points/questions raised from the last “randomness” post. If not, we can always continue the discussion in the comments section…

In order to avoid bad surprises in live trading, the maxim forming this post title should apply to as much of a trading system as possible. The goal is to ensure back-testing results are as realistic as they can.

Input data is one component of a trading system and the area this post focuses on, more specifically roll-over methodologies for futures contracts.

Real Monthly Trading Also Means Monthly Roll-Over

One of the triggers for this post was a comment on the Monthly Trading post with regards to the futures data used for the test. In order to have comparable results between the daily and monthly instances of the systems tested, I simply and mechanically reused the same portfolio of instruments…

However, futures contracts in the portfolio roll on Open Interest. What this practically means, is that a trader using this methodology would have to monitor OI every day to decide whether to roll to another contract. Not much of a monthly process.

I still think that test had some value, but it is important to realise that the monthly instance in that test was not representative of a pure monthly process. Instead we were actually simulating a trading strategy with monthly signal taking but daily roll-over monitoring process.

It highlights the fact that it is important to understand what the data used for testing actually represents, and to roll futures contract data in the same way as they are rolled in real-life.

This post is a good opportunity to revisit the concepts of generating continuous futures data and some of the older posts I had written on the subject using CSI (here and here).

ETF: Outsource your Contract Rolling Process

Another reason why I think that the monthly trading test has some value is because the individual monthly trader might decide to trade ETFs instead of futures.

ETFs can be seen as simple wrappers of futures contracts: they trade continuously and their managers take care of rolling their underlying positions with a predefined methodology and timing, charging you a fee in the process.

Of course, this is not futures Trend Following any more, and using ETFs lacks the inherent leverage embedded in futures instruments. But it is possible to obtain decent performance with Trend Following on ETFs (check this ETF Trading book by Anthony Garner for example). An assumption in that book (which I will attempt to verify in a next post) is that futures can act as proxies for ETFs/ETCs.

Contract Rolling and Testing

Rolling contracts is a process necessary to generate a continuous time-series for back-testing futures instruments. There are many different ways to perform this operation (some of which are covered just below). What is important is that the continuous contract should be a realistic representation of one’s trading.

Proportional vs Point-Based Adjustments

At the time of writing these posts on continuous contracts, I was leaning towards the idea that proportional adjustment was the way to go:

“This ensures a constant relative (percentage-wise) relationship between any prices across the trading history“, as opposed to the point-based adjustment, with which “you lose relative price difference (shifting all prices by an absolute amount has that effect on percentage changes: 10 to 11 is +10% but add 100 to both for an extreme case and 110 to 111 becomes less than +1%)“.

When back-testing with Trading Blox, the standard functionality expects the prices to be point-based adjusted, to calculate trade PnL:
(ExitPrice – EntryPrice) * BigPointValue * NumberContracts
With a point-based adjustment, the number of points between entry and exit will represent the actual points the market has moved during the life of the trade, as opposed to proportional adjustment (which would give wrong PnL calculations).

But point-based adjustments prevents us from using relative price difference, such as “check if close today is 5% higher than close yesterday” as an example. This might be an issue for some systems, but there are workarounds for this in Trading Blox though.

We do not have to limit ourselves to one or the other methodology, but we have to understand what the rolling algorithm represents and which data best suit each test.

Of course, you can even go with a more fancy approach to rolling (if this represents what your trading is), like one we had covered here last year with a rolling methodology attempting to optimize roll yield.

Back-Adjusted or Forward-Adjusted?

Another switch on the rolling methodology dashboard (in CSI) is the possibility to perform back-adjustments or forward adjustments. This only determines which price is fixed and in which direction the “adjustment drift” will propagate through the time-series. Forward-adjustment picks the earliest price as the reference price and moves each following contract by the required adjustment amount/ratio to fill the “rolling gap”. Backward-adjustment is the opposite: reference price is last price and adjustment propagates backward.

In order to get recent prices in line with real price levels, most traders prefer back-adjustment. From a back-testing point of view, it should not have much impact. However, the chart time-series will look quite different.

To illustrate this, here is an example chart plotting the same underlying (Crude Oil) since 2007, with four different standard CSI rolling methodologies:

• Proportional, Forward Adjustment
• Proportional, Backward Adjustment
• Point-based, Forward Adjustment
• Point-based, Backward Adjustment

Mix of Point and Proportional, Back and Forward Adjustments

Going back to the monthly trading test, I did take my data “for granted” as it works for most cases… but not really for this one. It does emphasise the point that “Trade what you Test and Test what you Trade” should also apply to data and rolling methodologies, which need to be checked/reviewed for each new system.
 
 

Long-Term Trend Following is often billed as a superior and simple alternative to Buy-and-Hold for individual traders.

For those investors wanting to take a DIY/self-trading approach, an active strategy might bring better performance – but requires a far more active involvement. In the case of an End-of-Day Trend Following system trading a global portfolio, the requirements to check the markets and system every day, potentially several times a day, might be incompatible with individual investors’ lifestyle (job, family, etc.).

A more practical – monthly – system might suit some individual traders’ requirements better. But can these monthly systems achieve decent performances?

Daily vs Monthly Data

This post will be looking at the results of one system traded on two different frequencies:

• Daily
• Monthly

One common approach when testing a monthly/weekly system is to use the corresponding timeframe for the data fed to the back-testing system. This can sound intuitively correct but testing a system on monthly data has some drawbacks.

We still want to be able to “see what happened” in between each monthly trading decision (i.e. chart the intra-month equity curve), which is not possible with monthly data. Moreover, the reporting frequency does affect the system statistics (an illustration with Max Drawdown being described in this paper by David Harding [pdf] – covered in this post).

In order to perform an apples-to-apples comparison of both frequencies on the same system, daily data needs to be used in each case.

System Rules

A viable monthly system needs to be relatively long term: no point in trading a 3-day swing trading system on a monthly basis…

For the purpose of this post, the classical Golden Cross system is used (a.k.a. the 200-50 Moving Average Cross-Over system – featured in the State of Trend Following report).

In its daily incarnation, the system is traded exactly as per the monthly report: Buy when Short MA > Long MA, Sell when Short MA < Long MA. No stops are used, in combination with a fixed fractional money management logic (position size based on volatility). The same portfolio is used over the period 2000-2010.

The monthly incarnation is near-identical, but for an additional monthly trading condition: a trade can only take place on the last trading day of the month. If the MA cross-over occurs on the 3rd of the month: the position only changes at the end of the month. Intuitively, we can see how this could be an issue; by possibly staying on the wrong side of the trade during a good chunk of a move.

Note that if the MAs cross back before the end of the month, the position will stay unchanged at month-end, which could prevent some whipsawing.

Results: The Monthly System Holds Up

Maybe in a sigh of relief, the budding DIY trader/blog reader will look at the following results, which seem to indicate little difference between the daily and monthly systems:

 
After all, it might be possible to run a Trend Following system on a monthly basis with a decent performance!

Below are the graphical comparisons of both monthly and daily systems:

The first chart represents the equity curves of both systems, with the difference between the two plotted on the second chart (the second plot represents the rolling difference in CAGR between each system). The monthly system actually over-performs for most of the testing period (daily over-performance is negative), until 2008, when the daily system takes over.

A Shorter-Term System

Intrigued by the results? I was. And wanted to investigate further how a system could hold up being traded on a monthly basis.

A shorter-term system should logically be more affected by the monthly trading condition, so I decided to stretch the test by using the 50-20 Moving Average Cross-Over and apply the same treatment to it.

Below are the results:

 
While the CAGR is comparable (even slightly better for the monthly system), the Max Drawdown figure is substantially worse (by 22.4%). Looking at the graphical results, we can see that the Max DD occurred during the crash of 2008 – where it paid up to follow your positions closely (ie on a daily basis). Note that this was the same for the Golden Cross system, where 2008 saw the daily system perform much better than the monthly one, during the crash.

Daily and Monthly Equity Curves

CAGR Difference between both systems

Note that comparing two different systems is always a subjective affair, depending on which stats you decide to focus on. For example, the monthly system performs worse on MaxDD but actually better on average drawdown amount (added on the last line of performance stats table), which is also reflected in the higher Sharpe ratio.

Is Monthly Trading Worth It?

As always, the same disclaimer applies regarding the significance of the test using one system only.
But the fact that trading a daily system on a monthly frequency did not dramatically impact performance is quite encouraging.
Moreover, the fact that no adjustments were made to the monthly systems suggests that there might be some room for further improvement (a logic to better handle fast-reversing periods like 2008 might be an interesting area to look at).

Monthly Trend Following systems might not be the best or optimal solution; but they could offer a nice cost/reward solution to the individual trader.
 
 

Many combinations...

Following the last study on diversification using a “portfolio randomizer”, I wanted to further explore further the concept of hindsight bias in portfolio selection and how it can impact performance. This short post is an illustration of that concept.

51 Instruments, 40 Picks, How Many Combinations?

The answer is… a lot: there are 47,626,016,970 ways of picking a portfolio of 40 instruments from a total set of 51 (formula is: 51! / [40! x 11!]). Running so many simulations was not practical (!) so I decided to run a fraction of the possible combination (5,000 random iterations). We get a similar set of results as the last study:

This is still for a run of the 20/50 Moving Average cross-over system running from 1990.
Note that the results are slightly different from the run with 40 instruments from the last post, as the position size was greater here (and therefore the “cloud” of CAGR/MaxDD results has moved towards the top-right).

The average performance figures for the systems are as follows:

Isolating portfolios

The next step was to isolate the worst and best cases (circled on the scatter plot) and identify the difference in portfolio composition between the two. Since this is a combination of 40 elements from a set of 51, there can only be 11 differences at a maximum between each portfolio.

Indeed, the “worst” and “best”portfolios share 32 of their 40 components but still exhibit a wide difference in their performance figures (the MAR ratio for example is 3.5 times better for the best case: 1.02 v. 0.29). For reference, the “good part” of the best portfolio (ie its instruments that do not overlap with the other portfolio) are:

• Cattle-Feeder-CME(Floor+Electronic Combined)
• CopperHG-COMEX(Floor Trading Only)
• Euro(Floor+Electronic Combined)-CME
• Gas Oil(Combined)-ICE(IPE)
• IBEX 35 Index-MEFF
• Natural Gas-Henry Hub-NYMEX(Floor+Electronic Combined)
• OMX Helsinki 25-EUREX
• Palm Oil-Crude-MDEX

On the other hand, the “bad part” of the portfolio is:

• Azuki Beans-Red-TGE
• Cocoa-CSCE
• DJ Euro STOXX 50 Index-EUREX
• EURIBOR-3 Mth-EURONEXT(LIFFE)
• Gold-COMEX(Floor Trading Only)
• MSCI Taiwan Index-SGX(SIMEX)
• Silver-COMEX(Floor Trading Only)
• Soybeans-CBT(Floor Trading Only)

To take the comparison to a further extreme, I ran the exact same system over both bad and good portfolios by themselves.

The difference in results is fairly striking:

Good portfolio

Bad portfolio

Robustness Testing

If anything, I believe this illustrates that testing over several portfolio sets might be a good way to identify robustness in system results. If the system only shows good results on specific portfolios, it might simply be a “lucky” outlier.

On Diversification

Finally, there were a few comments on the last post with regards to how to implement diversification. I had only focused on diversification from a portfolio point of view. However I believe that ideally one would diversify with a large set of instruments to trade, different systems covering different timeframes.

One problem is that diversification on all these levels bring on an increase in required starting capital (one likely reason why most Trend Following Wizards have a minimum managed account size in the millions).

So you might have to make a choice in how to apply diversification.

As diversification is really beneficial thanks to the non-correlation it brings, diversifying across different systems could also be a good idea, as systems can be more or less “engineered” to be un-correlated to each other (ie a Trend Following system with a Mean Reversion system).
It is also possible to “trade” a large portfolio without taking all the signals (by using filters or an overall risk/size limiter).
After all, this is something that the Turtles used to do (when they were at full position size they had to skip signals)
Some ideas to think about…
 
 
Credits: Thanks to Trading Blox forum member sluggo for the reminder of how the Turtles used to skip trades, and how filtering trades could enable small accounts to trade a large portfolio – on this thread.

As the cliche goes:

Diversification is the only free lunch on Wall Street.

This is a concept equally shared by Modern Portfolio Theorists and Trend Following Wizards, who usually emphasise the concept – and are often quoted as trading around 100 different types of instrument, if not more.

The State of Trend Following report contains a decent level of diversification with around 50 instruments and I wanted to use this as a base to check the impact of diversification on performance.

The idea is to run the same strategy using a subset of the portfolio (ie less instruments = less diversification) and see how it performs.

The problem, though, in selecting a subset of instruments out of the 51 in the original portfolio is that it could affect performance in the same way as any portfolio selection can (ie you could obtain vastly different results in trading the same system with two different sets of instruments, just by virtue of a “lucky” pick of strong performers).

Historical Performance

First, let’s get a reference point and look at the historical performance of the system chosen for this test: the 20-50 Moving Average system. Below is the performance back-test of that system over the last 20 years with the original portfolio:

Tests with Less Diversification

As mentioned below, the idea is to work on a subset of instruments and compare the results with the initial portfolio. To avoid any sort of data mining/hindsight bias in the portfolio selection, I decided to run a Monte-Carlo-like approach to test the system with multiple instrument subset combinations: instead of picking a single portfolio subset of 25 instruments, I’ll run the system over 1,000 different sub-portfolios, chosen randomly.

In order to get an idea of how gradually diversification affects the performance, I ran the test in three steps:

• sub-portfolio of 15 instruments
• sub-portfolio of 25 instruments
• sub-portfolio of 40 instruments

All instruments are picked at random from the list of 51 instruments in the original portfolio.

Each of the 3,000 runs generated a full system performance record. Below are plotted the CAGR and Max Drawdown for each instance:

Click to zoom in

The original system is also represented as the yellow dot.

Note that the “portfolio randomizer” did not account for any logic in terms of sector allocation. The original portfolio is balanced over several sectors (currencies, energies, rates, agriculturals, etc.) and there is no account for correlation between the different instruments (obviously correlation plays a big role in diversification: there is not much point in having dozens of instruments if they are all strongly correlated). However, over the large number of simulations, the main ideas of the test still come through.

Another point is that the only difference between the different runs were regarding the position sizing of each trade (fixed fractional), which were adjusted to obtain results of similar magnitude in each test (a portfolio with less instruments will require a slightly higher position size to match the return/drawdown rate of a portfolio with more instruments).

Looking at the plot chart, there are two main observations:

• We can see the gradual effect of diversification improving the system results by “moving” the cloud of performance points towards the left (less drawdown) and up (more return).
• The other observation is that the more diversification there is, the lower the deviation in the system results – therefore providing more robustness and less chance of data mining impact from portfolio selection on your back-tests.

Diversification or Why the Coffee Cup Never Jumps

That last point makes me think of an example discussed by Nassim Taleb in his Black Swan explaining the averaging of randomness:

Yet physical reality makes it possible for my coffee cup to jump – very unlikely, but possible. Particles jump around all the time. How come the coffee cup, itself composed of jumping particles, does not? The reason is, simply, that for the cup to jump would require all of the several trillion particles to jump in the same direction, and do so in lockstep several times in a row. This is not going to happen in the lifetime of this universe

Every trade/instrument can be seen as a particle composed of a (large) random element and a smaller edge that we try to extract via a mechanical system.

A portfolio composed of too few instruments would be like drinking your coffee or tea from a cup made up of only a few particles: cups would be jumping around everywhere, making coffee drinking a perilous venture. Same concept applies to trading.

This is the way I see diversification: by adding a large number of mostly random elements, you can ensure that random moves have some cancelling effect on each other so that your “trading cup” never jumps. All that is left is to collect the small edge from all the instruments via your preferred trading strategy(ies).

In effect, this is how casinos operate – and with diversification you somehow get to be the house!
 
 
Credits: The use of a portfolio randomizer and the display of results in a CAGR/MaxDD scatterplot was directly inspired from user sluggo on this Trading Blox forum thread.
 
 

OK, Profit Targets. You’ll tell me this is the anti-thesis of Trend Following, with one of its strongest mantra being:

Cut you losses short, let your winners run…

Indeed, Trend Following relies on a few positive outlier trades (from a fat-tailed distribution) to derive most of its gains. Stopping trades from running as high as they could might not sound like a good idea.

But it is also a tempting idea to try and exit the trade before the typical Trend Following reversal, if only to try and smooth the equity curve.

I know about it: I have been there before with my discretionary trading. It is so easy to fall into these two traps (let your losers run and cut your winners short!). That is one of the reasons I decided to trade in a systematic fashion.

Anyway, I digress… Despite hearing many times over that take-profit orders are detrimental to a Trend Following strategy, I wanted to test this for myself (always double-check your sources…).

Profit Target Test

I ran a quick test using a dual Moving Average cross-over system (the medium-term one from the State of Trend Following report)

The test involved stepping through two different parameters:

• Target profit level (expressed in multiples of entry ATR).
• Percentage of position to sell upon hitting the target profit.

There are three main cases driven by the position reduction percentage:

• 0% – no profit target: the position runs as per its normal course (ie. normal system exit signal)
• 100% – position is fully exited at the profit target level
• a value between 0% and 100%: position is partially exited at the profit target level

When partial take-profit orders are executed (eg cut 50% of position after price has risen/fallen by 6x ATR), the rest of the position gets liquidated when price rises/falls by another ATR-multiple.

The Results

Below are the results in graphical 2D and 3D form. Each surface chart represents the metric output (CAGR or MAR) for each stepped parameter pair.

CAGR results in 3D and 2D – click to expand the charts:

MAR results in 3D and 2D – click to expand the charts:

 
Both CAGR and MAR clearly display their maximum at the 0% level for position reduction – which means no profit target.

This is probably just one more example of the benefits of letting winners run.

Of course this is just a one-off, very simplified example and should not necessarily be used for generalisation… But it should not be a surprise that Trend Following’s performance is negatively impacted when breaking one of its fundamental principles….

There might be still some benefits in trying to identify potential reversal points – as is sometimes being advertised by some CTAs (BlueTrend comes to mind) – in order to cut positions early. But I believe the approach needs to be more specific/sophisticated than a blanket ATR-multiple Take-Profit order.

EDIT: Updated Test Results

I realised after writing the post that there was a “trend” in the results with performance seemingly improving as the profit target moved higher. I decided to complete the test by running it all the way to 40x ATR as profit target. Below are the results:


 
We can see some small pockets of over-performance on areas outside of the 0%/no profit-taking orders, but given the variability within the results, they may well be down to random variation…
 

However, this conclusion was obtained via a single observation (ie. one back-test sample). Ideally we want many samples, to be able to establish more robust conclusions – or at least be able to calculate a level of confidence in the result from our one observation.

And this is where the bootstrap test comes in handy. In this post I’ll try and illustrate how we can apply the bootstrap concept to a slightly different problem, in order to strengthen back-testing research results.

Bootstrap to compare two populations

Instead of using the bootstrap test to check if the profitability of a single back-test is statistically significant, we use it to check whether the difference between 2 sample groups is statistically significant. Here, the 2 sample groups are:

1. Trades with entry volatility at 0-90% of historical levels (lower-volatility group)
2. Trades with entry volatility at 90-100% of historical levels (high-volatility group)

The difference between the 2 groups can be measured by the difference of the means of each group. In the original samples, the difference between the R-multiple mean values was fairly high with a ratio of nearly 3-to-1::

1. Lower-volatility group: Average R-multiple = 0.27
2. High-volatility group: Average R-multiple = 0.10

However, this difference could be down to random variation. Running a statistical test such as a bootstrap will enable us to determine a level of confidence in the results.

Similarly to a two-sample t-test, we need to formulate the null hypothesis (H0):

H0: the mean of each group is identical (mean1 = mean2). This is equivalent to the difference of the means being nil (mean1 – mean2 = 0)

with the alternative hypothesis (Ha):

Ha: the mean of group 1 (lower volatility group) is higher than the mean of group 2

The goal of the bootstrap test is to generate the sampling distribution of the difference of the means and to calculate a p-value for rejecting the null hypothesis.

Here are the steps to follow:

1. Form two samples. Each sample contains all trades’ R-multiple values for one of the group.
2. For each resample, select random instances (with replacement) of R-multiples from each group (same number of instances as the group count). Calculate the mean for each group and compute their difference (bootstrapped statistic).
3. Perform a large number of resamples to generate a large number of bootstrapped statistics (difference of the means between the 2 group resamples).
4. Form the sampling distribution of the difference of means generated in the step above.
5. Derive the p-value of the difference of the means being 0 or less

For the case under study, the bootstrap gave us this sampling distribution (10,000 resamples):

We can calculate the p-value, which is 0.0276 (97.24% of the observations are positive, leaving only 2.76% zero or negative). We can therefore reject the null hypothesis.

The conclusion is that the result is statistically significant at the 5% level (p-value less than 0.05): there is only a slim chance (2.74%) that the null hypothesis is true (Type I error).

So, the filter seems to markedly improve the performance of the system. But of course, this is not a panacea… It is very well possible that a filtered trade might end up being a large runner.
With Trend Following’s dependency on infrequent large winners, missing a major trade because it has been filtered out could make a big difference to that year’s performance.
 

System Code Concept

I developed a simple Trading Blox filter, which calculates the current volatility (via the Average True Range) and its percentile rank over the past data.

The filter defines a range of accepted volatility ranking values, for which a trade can be taken. This would allow for a system to reject all trades for which volatility is too high (eg. in the top decile) or too low, or a mix of both.

The filter code is available for download at the end of this article. The parameters I used were 39 days for the (exponential) ATR and a lookback of 250 days for the percentile ranking.

The actual system tested here is a classic 20/50 Moving Average cross-over used in the State of TF report, using the same diversified portfolio.

Volatility: Good or Bad?

Trend Following is often thought of as a “long volatility” strategy, but could high volatility prior to trade entry be detrimental?

In effect, Trend Following usually generates lots of small losers balanced out by small and medium winners, the positive performance coming from some large winners (from the fat-tails). That’s one way of looking at it – which can be illustrated with this distribution of R-multiples for the 20/50 MA cross-over system:

All winning trades from 0R to 7R can be seen as balancing out the losing trades (mostly between 0R and 1R), whereas the “big hitter” trades (8R+) make up the profitability of the system.

If using a classic fixed fractional money management based on volatility (for example R = trade entry risk = 3 ATR = 1% equity), the R-multiple is akin to a volatility multiple. And with relatively high entry volatility, it is less likely that the volatility multiple can reach high values, and therefore the system is less likely to generate “big hitting” trades.

This is what we’re going to check.

System Test Results

In order to test the impact of trade entry volatility levels on trade profitability, I ran the system in steps, with the volatility filter boundaries covering each distinct decile (ie. between 0 and 10%, 10% and 20%, etc.)

The results are below:

 
There does seem to be a negative trend between trade average R-multiple and relative volatility value:

What is quite striking is that the Win % does not change much across all deciles (and if anything goes slightly up), unlike the average R-multiple. This seems to indicate that the winning trades simply fail to hit these high R-values when volatility is high at time of entry.

Below are the results when zooming in on the top decile:

 
Adding these high volatility trades might result in extra diversification (if one is to adopt the logic of “the more the merrier“) but they definitely do not seem to be the best use of your (probably precious) margin. If your available margin allows you to trade 50 instruments with no volatility filter, you might be able to trade 55 instruments at 90% max volatility instead, and get a better trade profitability.

In any case I ran a quick test by filtering the high volatility trades at 90%, 95% and 100% (no filtering) and the results show an improvement when filtering:

• 90%: CAGR = 53.65%, MAR = 0.73
• 95%: CAGR = 52.90%, MAR = 0.68
• 100%: CAGR = 46.08%, MAR = 0.5

Code Download

The Trading Blox code is composed of 2 blox:

• Auxiliary Blox, which calculates the percent rank for the ATR
• Portfolio Filter Blox, which stops trades from being entered if they breach the acceptable percent rank range.

These could be added to any standard system without modifications.

Aux_ATRPctRank
Port_ATRRankFilter

]]> http://www.automated-trading-system.com/volatility-filters/feed/ 6 http://www.automated-trading-system.com/walk-forward-trading-blox/ http://www.automated-trading-system.com/walk-forward-trading-blox/#comments Wed, 08 Sep 2010 13:58:11 +0000 Jez Liberty http://www.automated-trading-system.com/?p=2866

A few months ago, I got quite interested when Trading Blox announced that they introduced a new walk-forward functionality in their latest version. I just got round to upgrading, and giving that walk-forward testing a go. Amongst other things, some of the chart features have been improved – as can be seen in the eye candy above.

How it works

You can check this previous post for an explanation of how walk-forward works, from a general point of view. Robert Pardo (usually credited with the invention of walk-forward) has also written his bookfew books on the subject.

This new feature is a combination of small enhancements to the core application (ie. silent tests, dynamic date and starting equity settings, etc.) and a semi-custom script implementing the walk-forward testing procedure. It feels more like a (good) hack leveraging the core stepping functionality, rather than a functionality built from the ground up, but it does the job rather well (I looked at the underlying code and its ingenuity is actually pretty cool).

A minor issue is that you have to work out the number of optimization/out-of-sample cycles. If you get that wrong, the test will not cover the desired time interval and it makes the process less automated. Apart from that, the settings are pretty straight-forward, you can choose lengths of optimization and out-of-sample phases as well as other options:



In the settings above, there are 8 cycles, each having a 5-year optimization phase and 1-year out-of-sample phase

Objective Function

At the end of the optimization phase of each cycle, the system picks the best set of parameters to be used in the out-of-sample phase. However there are many ways to determine the best system. Do you use raw CAGR, MAR, Sharpe ratio, etc.?

There is a need to define the objective function (also called bliss function, as explained here, which will be used to determine the best system performance.

Trading Blox calculates standard statistics (MAR, CAGR, Drawdown, etc.) which can all be used as the objective function, but also allows the implementation of custom statistics calculations. These custom statistics can then be used as the objective function.

Output

This is where you realise the functionality is not fully integrated into the app. The system shows the individual results of each out-of-sample runs:



The stats calculated relate to each out-of-sample run (1 year in my example), whereas ideally these should all be spliced together, to form one system with its own stats. Same applies to equity curves, which are charted year by year (as individual charts). It is still possible to access the Daily Equity log file to do this yourself but it would be nice to have it better automated. The fact that it is a script and not compiled core code might make it possible to customize it to implement this automation.

Another log file also allows you to investigate the results of each optimization run and check their performance:

Starting optimization run ,225,2005-01-03, to ,2010-01-01, with starting equity ,166470686.069881530,
 
Run ,225, of ,256, Stepped Parameters: ,
Name,Step Value,Goodness Measure of ,64.497285767,
Run ( Index ),8.000000000,
Optimization Run,1.000000000,
Entry Breakout (days),20.000000000,
Exit Breakout (days),10.000000000,
Starting optimization run ,226,2005-01-03, to ,2010-01-01, with starting equity ,166470686.069881530,
 
 
----------------------------------------,
Best goodness of ,64.497285767, was on run ,1.000000000,
----------------------------------------,
 
Starting out of sample test ,241,2010-01-01, to ,2010-09-07, with starting equity ,166470686.069881530,

Test Results

For my first “test ride” of the walk-forward functionality, I used the good ol’ Donchian system.

The system parameters being tested are the Entry and Exit breakout lengths ranging from 20 to 50 days for the entry and 10 to 25 days for the exit.

The parameters for the walk-forward procedure are a 5-year optimization phase and 1-year out-of-sample phase with the plain CAGR stat being used as the objective/bliss/goodness function. The dates of the test go from 1998 to 2010 (but bear in mind that the first out-of-sample result is from 2003, ie. 1998 to 2002 are used as first optimization period).

The results (year by year) are the ones shown in the section above.

For an interesting comparison, I ran a standard stepped test using the same Donchian system and parameter values. Check below how the walk-forward system performance stacks up against the range of outputs from the stepped test:



All test results with the walk-forward result in yellow

In hindsight, it would have been better to pick other systems that performed better, but the advantage of the walk-forward process is that it adapts itself to pick a system.

In Closing

The walk-forward functionality is definitely not a fully-fledged feature of Trading Blox (yet?) – however it does allow for some automation of the process. The fact that it is “open” is probably a good thing as it can be tinkered with and improved easily. A nice addition to the product…

]]> http://www.automated-trading-system.com/walk-forward-trading-blox/feed/ 7 http://www.automated-trading-system.com/monte-carlo-permutation/ http://www.automated-trading-system.com/monte-carlo-permutation/#comments Wed, 18 Aug 2010 13:18:25 +0000 Jez Liberty http://www.automated-trading-system.com/?p=2770
 
The second method to evaluate the statistical significance of a back-test result presented by Aronson (in EBTA) is the Monte Carlo Permutation. This is an extension of the classic Monte Carlo method, applied to rule testing.

The concept behind the Monte Carlo Permutation is similar to the Bootstrap method:

• Generate multiple random outputs based on the single sample data from the back-test.
• compare the random Monte Carlo outputs to the back-test output to evaluate its statistical significance.

The difference lies in how the multiple random outputs are generated. Whereas the bootstrap generates a sampling distribution for the back-tested rule return, the Monte Carlo Permutation focuses on the pairing of the rule positions with the instrument daily return. Its resampling randomly associates the rule positions with the market returns, without replacement.

The H0 hypothesis in the Monte Carlo Permutation test asserts that the returns of the rule evaluated are a sample from a non-profitable population, or, in other words, that rule positions are randomly correlated to market returns.

Monte Carlo Illustration

Imagine the following back-test result, presented day by day:

Day	1	2	3	4	5	6	7	8
Rule Position	Long	Long	Long	No Pos	No Pos	Short	Short	Short
Market Return	0.54%	-0.32%	1.54%	0.69%	-1.02%	-0.68%	1.20%	-2.50%
Output	0.54%	-0.32%	1.54%	0.00%	0.00%	0.68%	-1.20%	2.50%
Mean Return	0.47%

 

There are effectively two input time series:

• Rule Positions
• Market Returns

The way these two time series are linked (by date) produces the daily output for the rule return – and a mean return can be calculated.

The permutation of the Monte Carlo method will reshuffle one time series to produce random links, or pairing, and produce a different rule output.

Two examples can be found below. The market return time series has been randomly reshuffled to produce two different sample outputs:

Day	1	2	3	4	5	6	7	8
Rule Position	Long	Long	Long	No Pos	No Pos	Short	Short	Short
Market Return	1.20%	0.17%	0.54%	1.54%	-0.32%	-0.68%	-0.33%	-1.02%
Output	1.20%	0.17%	0.54%	0.00%	0.00%	0.68%	0.33%	1.02%
Mean Return	0.49%

 

Day	1	2	3	4	5	6	7	8
Rule Position	Long	Long	Long	No Pos	No Pos	Short	Short	Short
Market Return	-0.68%	-0.32%	1.20%	-0.33%	0.17%	0.54%	0.69%	-2.50%
Output	-0.68%	-0.32%	1.20%	0.00%	0.00%	-0.54%	-0.69%	2.50%
Mean Return	0.18%

 

The Monte Carlo Permutation produces a large number of these random outputs. The p-value of the original back-testing sample can then be computed (it is equal to the fraction of random rule returns equal or greater to the back-tested rule return).

Note that Aronson once again recommends to run the back-test evaluated by the Monte Carlo Permutation on detrended data. It is also mentioned that Timothy Masters (who got the idea of applying the Monte Carlo method to rule testing) has performed tests showing that the bootstrap and Monte Carlo Permutation methods produce similar results when using detrended data.

Step by Step with Data Mining Bias Handling

Of course when applying this method to more than one rule, data mining bias comes into play.

The methodology for the Monte Carlo Permutation for data mining back-testing can be broken down as follows:

1. N back-tests are run on detrended data. Both rule position and market return time series are collected for the back-tested rules.
2. The market return time series is randomly reshuffled and paired with each of the N rule position time series to produce a new daily rule output time series for each rule. The same pairings must be used for all rules to ensure that the potential correlation structure present in the rules is preserved.
3. A mean daily return is calculated for each of the N rules – the best return is selected as the value for the sampling distribution in this iteration
4. Repeat steps 2 and 3 a large number of times
5. Form the sampling distribution of the best means generated in the steps above.
6. Derive the p-value of the best back-test mean return based on the sampling distribution.

Some “Criticisms”

Aronson mentions that since the Monte Carlo Permutation does not test a hypothesis about the rule’s mean return (H0 is about random correlation of positions and market returns) it is not possible to use it to derive confidence intervals – as could be done with the bootstrap sampling distribution.

The method also requires access to more information than the bootstrap (which only needs the daily rule returns). It makes it impossible to apply to “black box” systems or programs. For example, the Monte Carlo Permutation method would not enable us to check the statistical significance of a Trend following Wizard as was done in bootstrap post #2.

The same remark concerning the use of arithmetic mean return instead of geometric mean return applies here also, but that can be easily modified.

Finally, the method, as formulated, only considers extremely simple cases of money management with identical size for all positions. The method would need to be adapted to be used for rules with more complex money management strategies.

I’ll let you come to your conclusions and experimentations but it does seem like the Monte Carlo Permutation method has more weak points than the Bootstrap test.
 
 

Monte Carlo picture credits: Kristian Golding via flickr (CC)
]]> http://www.automated-trading-system.com/monte-carlo-permutation/feed/ 2