Systematic Trading research and development, with a flavour of Trend Following
Au.Tra.Sy blog – Automated trading System header image 2

Detrending for Trend Following?

March 17th, 2010 · 17 Comments · Backtest, Data, Trend Following

detrend-seven_resist-2We’re not rockers… but let’s look at detrending price series data for backtesting: Why, How and its relevance for Trend Following.

Why Detrend?

The main premise for detrending data is to remove the underlying trend effect on the strategy. This is due to the position bias that the strategy can have (eg being long more often than short).

Let’s say your strategy is long 70% of the time; and a bull market is currently taking place, there is a higher probability that you will be profitable. This could be due to the long position bias of your strategy – which might well become unprofitable during bear markets.

How to Detrend price data?

A simple way to detrend the data is to:

  1. Calculate the underlying trend in your backtest data
  2. Derive the “daily drift” from the underlying trend
  3. For each trade in the backtest, adjust the trade return by subtracting the daily drift for each day in the trade

For this it is easier to use log of price differences.

Step 1: The underlying trend is defined by the difference in price from the start to the end of the backtest:

Trend = LOG(Price_end / Price_start)

Step 2: Simply divide the underlying trend by the number of days in the backtest data:

Daily Trend Drift (DTD) =
LOG(Price_end / Price_start) / NumDays

Step3: For each trade, apply the detrending adjustment calculated above:
For long trades:

LOG(TradeReturn) =
LOG(SellPrice / BuyPrice) - DTD x NumDaysTrade

For short trades:

LOG(TradeReturn) =
LOG(SellPrice / BuyPrice) + DTD x NumDaysTrade

Note that the backtest is executed on normal, non-detrended data. It is the results that are adjusted by applying the detrending procedure.

Assuming a full reinvestment, the formula for the full detrended backtest result containing “n” trades would be as follows:

Detrending-equation2a
Detrending-equation2b
With D = 0 for long trades and D =1 for short trades (thanks to Tony for prompting the correction in the formula in the comments below).

Detrending for Trend Following

It might sound paradoxal to remove the data component that is at the heart of the strategy (the trend)… On one hand, you realise it might make sense, as the underlying trend is at a much higher timeframe than the trends identified by the strategy. On the other hand, the detrending concept assumes a dissociation of the position bias and the underlying trend.
Is it true for Trend Following?

We could argue that if the underlying trend is bullish, bullish trends will be prominent at a lower timeframe (vice-versa for bearish trends). By the nature of Trend Following, we would therefore expect the strategy to spend more time in long positions (long position bias). Because Trend Following is exactly that: it adapts and goes in the direction of the trend.

In short: Trend Following automatically generates position bias in the direction of the trend

If these two variables (position bias and underlying trend) can not seem to be dissociated for Trend Following, it might not make sense to apply detrending for Trend Following.

Other complications

The principle of applying a linear/proportional detrending adjustment assumes that the position bias is constant over time. It seems to ignore the potential for cyclical variations, possibly in line with major trend cycles (as would be expected with Trend Following).

For futures data, the detrending impact would be dependent on the method used for back-adjusted contracts. Point-based back-adjustment, for example, does not respect the proportional ratio between prices and would flaw the underlying trend calculation.

Finally, there are other factors that might need to be taken into account such as Contango and backwardation. The roll yield in futures trading, having possibly as strong an impact on the strategy’s return.

What are you thoughts on detrending for backtesting?…

Related Posts with Thumbnails

Tags:

17 Comments so far ↓

  • jkw

    Another way to correct for trends is to compare your trading system to one that has the same number, length, and direction of trades, but with random entries. So you take all of your trades and see what percentile they are in based on their duration and direction. So if you have a trade that is long for 200 days in a ten year period, you can look at all possible long 200 day trades and see how many of them your trade was better than. If your trade’s particular 200 day period wasn’t better than average (random trading), neither is the system that generated it. Of course you need multiple trades for statistical significance.

  • Motomoto

    My first thoughts – the aim of backtesting is to try and replicate reality as close as possible. So why detrend after the fact when in reality, you will not know what the future holds.
    Your comment – “The roll yield in futures trading, having possibly as strong an impact on the strategy’s return” – far more important to understand this and understand that this can win or loose on this alone. eg; contango in oil in 2009, $30 to $80 move meant the rolls cost a lot to be long. You did not just make $50
    Its always interesting looking at a continuous contract and comparing to a back adjusted contract chart.
    (ps: keep up the good work)

  • Jez

    jkw – true, but fairly harder to find/generate such system with exactly the same stats as the ones that your backtested strategy generates…

    Motomoto – thanks for dropping by here and for your comment/thoughts!
    To be honest, I am still wrestling with this concept of detrending. I can see the attractiveness of trying to remove possible biases – for example there are a few Trend Followers that believe you should not go short because historically it hasnt been (as) profitable (most possibly because the underlying long-term trend has been inflationary). A backtest of past reality can yield these sort of conclusions – however, being the devil’s advocate, how do we know that the next 10/20/100 years will not be deflationary (ie how do we know if the future holds the same long-term underlying long bias)?
    On the other hand I can see how messing with real backtest results might be perilous – especially with the possible wrong assumption of this detrending method (for Trend Following)…
    I dont think that’ll become a standard step in my backtesting process…

  • LuckyF00L

    Hi Jez,
    It’s been a long time since the above discussion took place and i was wondering whether you have managed in the meantime to better understand the concept of detrending. In my mind it seems to make perfect sense for a lot of strategies but i am struggling to see whether it is a good decision to apply this concept to a long term trend following strategy. As you have mentioned above “On one hand, you realise it might make sense, as the underlying trend is at a much higher timeframe than the trends identified by the strategy.”, this is how i make sense of it but on the other hand by nature a TF system is supposed to take advantage of this trends. I am still trying to figure out whether i should be using detrending to evaluate my strategies so any new found knowledge you might have acquired would be very helpful.

  • LuckyF00L

    My next question would have been how useful are then the two methods described in EBTA when it comes to evaluating long term trend following rules given that we are not sure whether detrending is a sensible thing to do or not.

  • Jez Liberty

    @LuckyF00L
    Since I wrote this post, I have become more of the opinion that since Trend Following has no position bias built in the strategy logic (ie it goes long when the market is bullish and short when the market is bearish), and since detrending is performed to remove position bias (ie penalize a strategy with a bullish bias when run in a bullish market), there should not be a need to remove an (inexistent) position bias in TF strats.

    Imagine we are looking at a TF strategy.
    By simplifying things we could imagine it makes 0.1% per day over 5 years during a bullish market (by being long more often than short). Assume the bullish market generates a 0.02% drift per day. The detrended strategy return could be approximated to 0.08% (0.1 – 0.02).

    If the bullish trend exactly repeats itself for the next 5 years, the same results would repeat themselves generating an overall detrended strategy return of 0.08% over the 10 years.

    If instead, we imagine that the bullish trend gives place to an exact symmetrical opposite bearish market. The TF strategy will now make 0.1% per day trading mostly on the short side (ie longs are replaced by shorts since market days up are replaced by market days down). The daily drift is now -0.2% for a total of 0% over the 10 years. And the detrended return is 0.1% per day (0.1 – 0.0) over 10 years.

    This is the exact same strategy, but detrending its results would give different return numbers based on which market the strategy is presented with, which I do not think makes sense. Not detrending the returns would give identical results for both scenarios (bullis-bullish or bullish-bearish).

  • Mike

    Hello,

    I was wondering if the underlying concept is the same if I want to calculate a detrended version of a indicator?

    Thanks

  • Jez Liberty

    Mike
    Not sure what you mean by a detrended version of an indicator?
    The method above is to remove any trend bias contained in a dataset (ie to avoid penalizing short trading stsrategies in a bull market for example).
    An indicator would not contain such bias, would it?

  • Mike

    Hi jez,

    Not sure, I was talking with frank from engineering the other day and he mentioned he uses a detrended version of the ATR…

    Hmmm….

    M

  • Emil

    Okey, I need to bounce some ideas here. The same way that detrending is useful to avoid too much influence from trend, wouldn’t adjusting the volatility of the data-set be useful as well? For example, if you’re backtesting a strategy on several different securities and a few of them happen to have had periods of violent volatility, wouldn’t it make sense to adjust the volatility of all securities so that they all contribute the same return (minus of course the difference in prediction efficiency)?

    I’m thinking of this approach because I’m in the process of training strategies on a large set of stocks and need some guidance.

    Cheers!

  • Jez Liberty

    Hi Emil,
    Typically a trading system itself would take volatility into account and would “normalize” position sizing so as to have equal risk for any position as a percentage of equity (if a stock has a volatility of say 100, you would take a position for size of 10, but if the volatility is 1000 you would take a position size of 10).
    Jez

  • Chris

    I am coming to this post late. My thought on detrending is this…
    1) Detrending is a good tool to evaluate the asset allocation component of your system.
    2) Detrending data and testing using a trend following system is counterproductive as you are testing its ability to capture trends.
    3) You should look at the return profile of your trend system and take out the outliers, (2008 is an outlier year) if your trend system gave robust results excluding 2008 that is good. 2008 in itself was a rare occurance.
    4) To come back to why detrending is good for asset allocation is because you want to remove the position bias in you asset allocation decision process. What you are looking for is the asset allocation process that has the best risk profile.
    5) To conclude, if your system is profitable and robust taking out 2008 and other outlier performance statistics, you likely have a profitable signal to achieve RETURN.
    6) If you detrend your data when constructing your asset allocation model for your system, you remove position bias (i.e. the benefit of being overweight and long treasury bonds and gold past few years) and you get a robust result in terms of managing risk. (Yes your system return will likely decrease, but detrending the data is not to evaluate the return aspect of your system but the risk aspect, overweight an asset with a strong trend can mask the risk of your system)

  • Sarfaraz Farooqui

    Hi,
    What is the base for your LOG in the detrending formula ? Natural (2.718) or base 10 ?
    Please let me know
    Thanks
    sarfaraz

  • Jez Liberty

    Sarfaraz: Log base 10 (although this should not have an impact on the concept here since it’s applying the same logarithm to both sides of the equations).

  • Tony

    Hi,
    Great article, but I have a question about the full formula.
    While it agrees with Aronson’s formula in the EBTA book for long trades, it seems to be inconsistent for short trades. In particular, it seems that in order to be correct, the DTD component would need a sign change (or simply be multiplied by -1) for short trades.
    Do you agree, or have I missed something? Or is Aronson incorrect?
    Thanks,
    Tony

  • Jez Liberty

    Hi Tony, I believe you’re right actually. I’ll double-check and update the post but it seems to make sense: assume a positive drift: you’d want to substract it from the long trades (that are “helped” by the drift) and add it to the short trades (as the drift goes against the short trades).
    Thanks for pointing this out. I’ll update the post shortly.

  • Patrick

    Subtracting trend from long performance in an up trend and adding that same performance back to short trades in an up trend makes no sense to me. You have learned nothing about the overall robustness of the system because you simply transfer profits from long trades to short trades!

    And you have no guarantees that when the market trends down that the shorts will make that much more from the trading activity. The system can still be skewed toward longs and this “detrending” mechanism will give a false sense of hope that the system is sound when in fact, it may be highly skewed toward the long side.

Leave a Comment