Au.Tra.Sy blog - Automated trading System Au.Tra.Sy blog - Automated trading System
Search
What Everybody Ought to Know About Continous Futures Contracts

And how to avoid easy common mistakes when choosing which data to use to back-test a trading strategy on Futures… This is a “long-ish” post but I believe essential for good-practice back-testing.

No long-term continuity

Futures are specific in the way they trade in series of short-lived contracts that are only active for a few months (the most active contract for Silver at the moment is SIZ09 i.e. Dec 09 Silver 5,000 oz – see this post for Futures Symbology. Around mid-December most activity will roll-over to the March contract SIH10).
This is different to stocks which offer a continuous stream of prices (generally since start of trading). In order to back-test a system on historical futures data, you need to “stitch” the futures contract prices to generate a similar continuous stream of prices.

The $64,000 Question: How do you stitch the data?

Easy, you might say: just chain all the contracts one after the other. But there are always multiple contracts trading concurrently with different expiration dates (for Silver currently December 09, March 10, May 10, etc.). One accepted rule (in testing, but also in trading) is to always consider the front-month contract (nearest expiration date contract) and roll the price series into the next front-month contract around expiration time. That selection process determines the actual contract to consider for any given date. Now, it is “just” a matter of stitching them up together.

But…

As you might know, contango, backwardation and other factors (crop seasons, etc.) generate a difference in the price of different expiration date contracts. In effect, when comes the time to move from one contract to the next, there will most likely be a gap between the old contract price and the new contract price. These gaps can be substantial and make your data appear disjointed.

Cocoa March 2005 (yellow) and June 2005 (Green) contracts - a big price gap exists at the time of transition between the 2 contracts

Cocoa March 2005 (yellow) and June 2005 (Green) contracts - a big price gap exists at the time of transition between the 2 contracts

The distorting “Panama canal” method

One of the workarounds (described by Ed Seykota in his Panama article) is to raise or lower each of the series one after the other so that each contract joins without a gap (similar to the boats going through the Panama canal). There are several problems with this “primitive” method:

  • You introduce a trend bias (by always lowering or raising prices at rollover time, the impact over time will introduce a large drift. Far past data could even become negative)
  • You are losing relative price difference (shifting all prices by an absolute amount has that effect: 10 to 11 is +10% but add 100 to both for an extreme case and 110 to 111 becomes less than +1%).
The March contract (in yellow) has been lowered to eliminate the gap with the the June contract

The March contract (in yellow) has been lowered to eliminate the gap with the the June contract. This distorts the relative price difference: the 25% move from 1480 to 1850 becomes a 27% move from 1380 to 1750.

Taking a leaf from the Equities book

The right approach seems to consider each contract roll-over in a similar fashion to a classic “stock-split” on Equities. When a stock splits, you effectively get X shares for every Y shares that you hold and the price is adjusted accordingly to reflect market capitalization. If a company performs a 10-for-1 split (meaning you get 10 new shares for every share that you hold), the price will effectively be divided by 10 after the split is effected – and historical data will normally be divided by 10 to represent the prices in the same “terms” as today (and therefore avoid showing a big price gap during the split). The historical data is “back-adjusted” by applying a proportional ratio related to the stock split (ratio=1/10 here).

Proportional back-adjustment splicing for Futures contracts

For Futures contract we can apply the same process of proportional back-adjustment at every contract roll-over. You would determine the adjustment ratio by dividing the price of the new contract by the price of the old contract. This ensures a constant relative (percentage-wise) relationship between any prices across the trading history.

The March contract has been lowered to join the newer contract with no gap. This has been done by multiplying the whole price series for that contract by 6% = 1850/1750 (prices before/after roll-over).

The March contract has been lowered to join the newer contract with no gap. This has been done by multiplying the whole price series for that contract by 6% = 1850/1750 (prices before/after roll-over). This keeps the relative price difference: The 25% move highlighted stays a 25% move. You can see that the junction price is identical to the previous chart but the intermediate low circled in red has been raised (along with the whole series) to maintain the existing price ratio.

There might still be some concerns related to actual tick size vs. calculated tick size compared to the minimum tick size (a parameter you should probably take into account in your strategy) but proportional adjustments seems to be the best compromise because of the points highlighted above

There are many other ways to aggregate historical Futures data as well as other parameters to consider (nothing is ever so simple!) which I will address in a later post.

Credits: This issue first came to my attention and described in detail by Thomas Stridsman in his two interesting Trading Systems books:
Trading Systems that work and Trading Systems and Money Management
Stridsman Trading Systems that work  Stridsman Trading Systems and Money Management

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)

Related Posts with Thumbnails

arrow28 Responses

  1. Mike
    59 mos ago

    Jez,

    Great blog so far! I’m in about the same development phase as you are with my trend following system. I have UA and am trying to find the best contract backwardization method for system testing.

    Thanks for the site!
    -Mike

  2. 59 mos ago

    Mike – Thanks I’ll try to keep it rolling.
    What back-testing software do you use? (I have Traders Studio but still need to warm up to it…)

    I’ll probably make a later post about the Unfair Advantage settings and what my set up/options would be for continuous contracts. Would be good to compare notes.
    -Jez

  3. 59 mos ago

    Jez–
    Good to see your blog and the posts are excellent.
    Keep up the great work!

    –Matt

  4. 59 mos ago

    Thanks Matt, I enjoy your blog too!

  5. 58 mos ago

    This is a good post that illustrates a variety of clever mechanisms for massaging data for easy backtesting of futures strats.

    That said, for my money a much better approach is to leave the futures data alone and smarten up your strategies so they don’t look to trade a specific contract but instead trade the current most active contract. This avoids all the data mangling issues you mention as well as providing a strategy that can be deployed directly into the market. It also enables you to backtest strategies which are trading the curve instead of just the most active or front contract.

    For me, any approach that involves mangling your source data should be looked at with skepticism as it will inevitably introduce distortions to your backtested results.

  6. 58 mos ago

    Tito,

    I agree with what you are saying.

    This is only a compromise and will not beat any “straight clean data” back-testing. Its disadvantage is potential distortions (although – as per my examples – the idea is to minimise them) but its advantage is that it produces a format much easier to process. For example I don’t know of any back-testing systems that handles individual contract files for strategy testing. So you have to do something similar if you want to use these testing platforms – which speed up the testing process from my point of view.

    The way I envisage the testing process is a 3-step one:
    - research, test, optimise and choose strats in a rapid manner in a back-testing platform (like in TradersStudio or TradeStation)
    - Validate the elected system(s) by doing a “more serious” testing on raw contracts data (i.e. applying skepticism to results found above) by coding up the strategy from scratch and not using the back-testing platform.
    - Implement/deploy the code from step 2 in live if the results are satisfactory.

    That way, the distortions would only affect initial systems testing/selection but the decision of putting a system “live” is still based on raw data back-tests.

  7. EricC
    56 mos, 3 wks ago

    Excellent blog! I’m working on these same issues. The problem with proportional ratio adjusting is that the resulting P&L will not be correct. When you apply percent based adjustments the point moves are corrupted. You would have to keep track of all the adjustments you made and apply those to the “point value” of the contract expressed as a time series. CSI does not do this.

  8. 56 mos, 3 wks ago

    Thanks Eric!
    I agree, I dont think it is easy to find an “ideal” transformation – however it makes back-testing in most software packages much easier – and I am not convinced the actual results are so different to using the raw data. Obviously you have the issues that you highlight in your comment but it shoud still allow to test some characteristics of a system – what back-testing platform do you use by the way?

    Ultimately I would like to implement my own Futures contract concatenation program so that I would have control over all the variables I need (including adjusting the point value, also storing actual rollover dates to take rollover costs+slippage into consideration, etc.).

    One thing that CSI does not seem to provide either is historical “market characteristics” (ie meta-data such as min Tick move, margin requirements, etc.). I have been unlucky so far to even locate a source for this data. I’d be glad to hear if you have? They would obviously need to be adjusted similarly to the point value.

    I would be interested to hear on what method you have settled to concatenate your Futures data (if you have elected one)?

    As of today I only use a set of continuous Futures data as it is more practical. But in the future – and before I go “live” with a trading strategy I fully intend to back-test the version on pure raw data (se my comment just above in response to tito’s comment). When I come to that stage I’ll post an update on how the results compare between raw vs concatenated data (actually that could be an idea for a future post – before I find that “killer” trading strategy ;-)

  9. EricC
    56 mos, 3 wks ago

    I suspect there is much valuable data lost and unnecessary risk taken by transforming the data. In many commodity markets the liquidity is spread out across contracts and different contracts represent different supply/demand realities. For example, rolling out of the fall crop into the spring crop due to higher open interest and/or volume makes no logical sense to me. They are two different products that can be moving in different directions and/or can have different degrees of hedger (premium payer) participation. The fact that the spring contract is trading more volume than the fall contract doesn’t convince me that I’m now best served abandoning the fall contract and NOW starting to trade the spring. All it means is that there is plenty of liquidity in the spring contract, implying that there has probably been sufficient liquidity to trade it for some time. There is no rule that says you can’t trade both at the same time. Imagine such logic applied to stocks in the software sector. I’ve been trading Oracle and ignoring Microsoft, but now that Microsoft has more volume I’m done looking at Oracle and will start trading only Microsoft? Maybe as a market maker, but as a trend follower? Same with natural gas; front month contracts can skyrocket while back months decline. The very act of rolling locks you into a mutually exclusive decision that is a function of liquidity AFTER it has transacted. People rarely question whether the decision SHOULD be mutually exclusive. A roll methodology requires one selection from the following two options (re: frontMonth/nextMonth): Yes/No; No/Yes. Not using a roll methodology offers the following four options: Yes/No; No/Yes; Yes/Yes; No/No. I understand that simple is preferable, but most people do not consider the risks and opportunity costs resulting from assumptions.

    The logic supporting abnormal returns in the futures markets argues against rolling on volume or open interest. Trend followers are supposed to be providing insurance to hedgers by purchasing rising markets and selling falling markets (trading against hedgers). But if you wait until AFTER the open interest or volume is so high in a contract that it exceeds that of any other contract you are choosing to avoid most of the hedging activity (since that’s what produced the open interest and volume). If you wait until the hedgers are already in (already hedged) to participate, all that’s left is the beta of the market. That’s close to just trading the spot market. Not saying that can’t work…but it should be considered for what it is.

    I’ve tried manually concatenating contracts using various different methods (OI, Vol, time, delta of each, etc.) As expected, the results are different.

    At this point I’m inclined to use un-adjusted data, if only to be intellectually honest. This is a problem for most backtesting software programs. But most backtesting software programs don’t produce realistic results anyway. TradingBlox, Mechanica, TradersStudio, PowerST are a few that are worthy. It is inconvenient to use individual contracts. But in this business you do not want to be a payer of the “convenience premium”…because it tends to be very high.

    I use PowerST http://www.powerst.com

  10. 56 mos, 3 wks ago

    Thanks for a great and insightful comment Eric!

    I have to agree that I tend to think of concatenating contracts for markets where it seems to make more sense (ie where there is no/less seasonality – like gold for example – although it could well exhibit serious backwardation in the future, similarly to how oil was showing serious contango at the beginning of the year). As you highlight, agriculturals with different seasonal crops is even worse (I like your comparison with trading different stocks).

    I see what you are saying with trading something close to the spot market – which is basically what I have in mind when generating a continuous contract – but as you rightly point out, it is true that we might be missing some opportunities in the other contracts.

    In any case thanks again for taking the time to post your views – which will definitely benefit my reflections on that matter (and hopefully so for other readers)

    I’ll definitely check PowerST – thanks.

  11. blaine
    44 mos ago

    what about chaining the series consecutively, taking the log return of the entire price series, including the gap… and setting the immediate period return subsequent to the roll equal to zero. the front and second contracts should have a high enough correlated period return to do it this way.

  12. 44 mos ago

    @blaine
    It seems that it would be equivalent to the point-based adjustment / “Panama canal” adjustment described above?
    I’m not sure I understand what the benefit of such approach would be…

  13. stefan
    28 mos ago

    Few points:
    1) All traders has to define specific rules to roll. The difference in results between continuous and individual contract fed into the machine is marginal (5-10%).
    2) Who cares about “relative price difference” aside from analysts and those who uses “relative price difference” models that are inferior to start with.
    3) Fundamental stories of some summer or moon crops is totally misplaced to CTAs who run money management business on whatever moves and has liquidity.
    4) Finally, the rolling method should reflect what we do in markets and how we roll. It is that simple. For trader, the only reality is the brokerage statement. Under this assumption who cares that Panama introduces price bias if this is what reflects your P&L. For analyst interested in angles or some historical level of noise, Panama or any back/forward adjusted contract can be an issue. I will there were less analysts.
    S

  14. 28 mos ago

    Stefan,
    Re. point 4) I’m with you on that one (using Panama for P&L calcs). Check this later post where I cover a similar point: http://www.automated-trading-system.com/trade-what-you-test-and-test-what-you-trade/
    Not sure I would be as definitive on point 2) (models using relative price difference are inferior).
    Jez

  15. Stefan
    28 mos ago

    Hi Jez,
    Thanks. One point that is also sometimes missed is that rolling on “Open Interest” only can push you to contracts where there is highest OI but very limited volume. Recently this just happened in Crude Oil (in February traders would roll to December contracts with great OI but limited daily volume). For me it makes sense to combine IO+Vol if I am lazy to code FND for each instrument (going with dates).
    S.

  16. Peter
    22 mos, 1 wk ago

    There is even a much more serious problem. Depending on the day I take the difference between two contracts can vary greatly.
    Example Corn Contract:
    On 1st July 2011 quotes are:
    July Contract: Close=640.75
    Sep Contract: Close=606.75
    Difference=34

    On next day (5th July 2011) quotes are:
    July Contract: Close=680.50
    Sep Contract: Close=625.50
    Difference=55

    And on 6th July the difference is again 30

    So, if I stitch the data together on 1st of July then my backtest results will be quite different than when I stitch the data together on 5th July.

    A problem I could not solve up to now

  17. 22 mos, 1 wk ago

    Peter,
    This is expected as the two contracts trade independently and the traders/market consider them so. The difference between two contracts of the same underlying at different maturity is actually a calendar spread and for some commodities it can indeed vary greatly than one might think as they actually represent a “physical reality” subject to variations (e.g different crops for agriculturals or seasonal consumptions with natural gas).
    In terms of back-testing and stitching the data to generate a continuous time-series, a usually accepted practice is to “move with the market” by following the liquidity (when the main liquidity switches from one contract to the next, you would do the same and record the difference at that time). In any case, as mentioned before in the blog, you should really aim to “test what you trade and trade what you test”: determine your stitching logic in your back-testing based on how you would do it in real life trading.
    Jez

  18. Chinmay
    21 mos, 3 wks ago

    Jez,
    Let me first say that you have mighty impressive blog here. Real great work..

    I am looking at backtesting some futures based trend following strategies as well with 20-30 years of data and have run into same dilemma.

    How to trust backtesting with unrealistic prices.

    What type of continuos contract do you use for backtesting?

    I am inclined towards Proportional Adjustment Contracts as it preserves the % changes.
    I understand that answer will depend on the algorithm.
    For. eg. if you are using 2 moving average crossover system with ATR based position sizing, then the timing of the signals should not be affected due to proportional adjustment. Since your ATR number will also get blown up / down by same factor as price , your size calculation will also “adjust” itself and hence you will reach same final PNL number for the trade. So in this case Proportional Adjustment can be safely be used for backtesting.

    Is this logic correct? Am I missing anything?

  19. 21 mos, 3 wks ago

    @Chinmay:
    I think this later post answers most of your questions:
    http://www.automated-trading-system.com/trade-what-you-test-and-test-what-you-trade/

  20. Peter
    21 mos, 2 wks ago

    Whether to backadjust or “only” splice contracts together depends on the trading style and the applied indicators. I am trend follower and therefore backadjsut contracts.
    I am not sure whether rollover on front month or by volume etc. is the best method. I did the following test:
    a) CrudeOil for 10 years, always rollover to front month (=12 contracts per year)
    b) CrudeOil for 10 years, staying in 1 contract for one year. I have chosen the march contract
    Then I compared both. The difference is huge, I mean really huge.
    I could understand that the difference is 10 or maybe 20%, but it is over 100%. This does not make sense to me.
    I was now also searching the internet for hours and could not find anything on this topic, discussions are always about rollover-methods and time-related only with front month or volue,

    Did you do similiar tests? Do you have an explanation about such huge differences?

  21. 21 mos, 2 wks ago

    Peter,
    Crude Oil is going to be one of the instruments most subject to differences in contracts you pick because of strong impact of contango/backwardation.
    The point you are making is also often made by Attain on their blog (by comparing passive commodity ETF investing vs. “1 contract for one year” futures investing and they usually show significant differences for several products. One potential problem in trading far-out contracts is the liquidity and the extra slippage you might be faced with.

    Please check that post where I explore the concept of roll yield (contango/backwardation) optimization combined with a trend following strategy:
    http://www.automated-trading-system.com/better-trend-following-improved-roll-yield/

  22. 21 mos, 2 wks ago

    Whatever the method of rolling, it is a good practice to see the scenario you trade plus other alternatives to see the sensitivity. Results are usually very correlated on a large sample of data.

  23. Chinmay
    21 mos, 2 wks ago

    @Jez, I went through that post but still have my doubts about backtesting with continuous contracts. Not testing with real fills is just unsettling for me. I will either come to terms with it or end up writing my own backtester.

    @Peter, your contract rolling methodology is bound to have huge impact on the your backtest.
    Contango and backwardation will add fair bit of “side-effect” to your normal directional/non-directional strategies, as different contracts trade like two different underlyings altogether.

  24. Peter
    21 mos, 2 wks ago

    @Jez, thanks for the link. An interesting article, as many others on your excellent website

    @Jez,@Chinmay:
    1. I was also looking at Corn-Data. Also there we have quite a big difference over 15 years. The smallest difference we have when I take one contract per year and then do a rollover.

    2.Interesting: In the wheat futures we do not have this behavior, we have exactly the opposite. From an economic point of view I do not understand why this happens, I do not see any logic in that (Wheat behaves exactly opposite as Corn does)

    3. The remaining question is: what to test (and trade)? If I would trade futures, I would choose now half-year rollover, for markets with higher volume 1-year. Since I trade CFD (a kind of mini-futures in Europe and Australia), I can trade only the front month.
    I am a swing/long term trend follower and stay with daily data. but maybe it would be better to try with 4 hours in order to be lonter in one contract and have better backtest possibilities

  25. Chinmay
    21 mos, 2 wks ago

    @Peter,
    For your point 1 and 2, you need to look into fundamental characteristics of underlying to understand why Contango or Backwardation exist in Corn or Wheat. My guess is that it will be a lot related to seasons in which each crop grows, weather variations etc. I am not expert so can’t say much further about why they exist , but can say with guarantee that they will make huge difference :)

    For point 3, what I think ideal way to test your strategies for CFD would be probably to check when does your broker changes underlying ticker of the CFD. What rule do they use. Do they do it based on open interest / volume / or both. And then create your own continuous contract using same logic.

    I am sure you will have to price in lot more slippage and wider spreads compared to futures while backtesting your strategies if your backtesting your CFD strategies on futures historical data.

  26. Peter
    21 mos ago

    I am aware that contango and backwardaton is different for each product. But that it is consequently reverse in soybean (not wheat, I wrote wrong above) does not make sense to me economically

    CFD:
    - Almost all provider simply rollover a few days before contract end. There is no choice
    - Slippage: Yes. Unless you have a portfolio of several millions USD and want still want to aply moneymanagement and portfolio management only CFD remains. Furthermore, when you trade longterm, it is not a problem

  27. 13 mos, 2 wks ago

    What about adjusting Volume and OI by the same ratio?? Or, does that give an illusion of more liquidity than there really was?

  28. 13 mos, 1 wk ago

    It’s not really an issue for volume/OI because there are no gaps/steps you cross every time you change contract like in the price (i.e. the volume of Crude Oil last year is directly comparable to the volume right now).

Leave A Comment