One of my favourite non-trading-related blogs is FlowingData, a visualization and statistics site by Nathan Yau.

I find Data visualization fascinating and inspiring. A good visualization usually “brings data to life” making it easier to understand. Most visualizations featured on FlowingData are very aesthetically pleasing, making the subject a sort of enjoyable and useful “geeky art”. The Correlations Heatmap video below is an attempt at some of this.

Trend Following Wizard Correlations Visualization: Video of Evolution through time

For this post, I got inspired by a reader comment mentioning that it would be interesting to calculate correlations between the Trend Following Wizards reported on the blog.

I agreed with the idea, and seeing this tutorial post on creating heatmaps in R on FlowingData (with sample code: thanks!) gave me the idea of a data visualization of my own: showing the evolution of correlations through a “time-lapse” video of monthly correlation heatmaps through time. Fun and quick little project dealing with trading-related data, visualization and programming. Perfect combination!

The resulting movie is embedded below (available directly on the vimeo website in slightly larger size):

With the color key:

 
Every month, a rolling 12-month correlation number is calculated (using Pearson) between each Wizard, The full matrix of pairwise correlations is then represented in a heatmap format, with the cell color, directly representing the correlation coefficient, as per the color key above.

The movie simply stitches all heatmaps in chronological order, month after month, from January 1990.

You’ll notice that some cells are grey: light grey means that I do not have any available data for that month for the corresponding Wizard. Dark grey is to hide the correlation between a Wizard and itself (irrelevant as obviously always equal to 1).

It is interesting to see how the heatmap moves from periods of strong correlation (near all-blue, check the Summer of 2003 for example) to more heterogeneous periods.

I was also intrigued by some periods showing real jumps for the whole heatmap (check July 99 to August 99). It is not so surprising though, when you know how data outliers can have a strong influence on the (Pearson) correlation numbers (this is what happens in the jump from July to August in 1999: August 1998 showed strong returns for most Wizards; when these numbers drop off the correlation calculation, the correlation coefficients sharply decrease as a result).

Software note: The movie was created using R to generate the heatmaps (as per FlowingData heatmap tutorial) and XnView + FFmpeg to edit and stitch images into a video.

More Data Visualizations

FlowingData publishes very regularly and the quality is usually very good (Nathan Yau’s book is definitely on my Wish List).
For readers interested in checking it out (watch out, you can quickly spend an hour on the site!), here are some posts or links that caught my attention:

Finance

In investing, timing is everything (we know that!): This one is directly related to investing/trading, and I believe Trend Following Wizard CTAs could use the idea to present their past results to prospective investors in a similar fashion. This is a grid diagram which shows annualized returns for the S.& P. 500 for every starting year and every ending year since 1920 – nearly 4,000 combinations in all:

Originally from the NY Times, link to full original diagram

CTAs could plot their monthly returns to get an overall view of what an investor could have made at different periods and highlight specific stats like 36-month, 60-month rolling returns. Possibly another similar chart with drawdowns to complete the picture. This could be an interesting project.

Past century of government and economy: Displays several economic indicators over 100 years, aligned with political parties in power.

Sports

For sport stats lovers:

• Pennant iPad app: Every team. Every game. Every play. 1952 to 2010. If I had an iPad and were into baseball, I’d definitely get this one: it looks really cool. Check the video out.
• Juice Analytics display NFL statistics visually, more interesting than the usual tables.

Maps

Facebook worldwide friendships mapped:

Commercial Air Traffic Seen Around the World (video):

Watching the Growth of Walmart Across America (video):

Our changing world in cartograms:

Series of videos showing the evolution of several indicators per country, morphing the size of countries proportionally to its indicator value. Some surprising world map redrawing!

Others

10 Best Data Visualization Projects of the Year – 2010:

What New Yorkers complain about:

Fancy drinks recipes infographics:

And finally, an interesting one hour documentary video on “The Joy of Stats” by Hans Rosling

Enjoy!

I have already covered the idea of using ETFs in place of Futures. Today, I wanted to run a quantitative comparison between the two instrument types

The ETF sector has been growing at an impressive rate, with new offerings popping up every month and providing a wider choice in selecting a portfolio to trade. One of the drawbacks of using ETFs for mechanical trading strategies is the relative short history of these instruments, making it hard or impossible to run to back-tests looking back far in the past – which I believe is critical to understand how a strategy works over different types of market conditions.

In A Practical Guide to ETF Trading Systems, author Anthony Garner uses futures contracts as a proxy for testing strategies on ETFs. If futures are good proxies for ETFs, the back-test results from a long run with futures can be extrapolated to the ETF world.

This post is attempting to look at one way of quantifying how well futures can act as a proxy.

Quantification by Correlation

The main “tool” used in this post, to test whether futures are good proxies to ETFs, is that of correlation between prices as well as returns. The assumption being that correlations between an ETF and a good candidate futures proxy must be as close as possible to 1.0. The closer to 1, the better the proxy relationship.

I picked a few ETFs and associated futures and calculated the correlation (using Pearson product-moment) between several of the data attributes:

• Open
• High
• Low
• Close
• t-1 Return
• t-5 Return
• t-30 Return

The data considered in each case was the list of all dates where both futures and ETF had data. The futures contract were proportionally back-adjusted (to keep price ratio at correct values, as discussed in this post).

The table below shows the number of records for each comparison, the start date and the various correlation coefficients.

Note: please make sure you view this post with your “browser zoom” at 100% for the numbers to be readable.

Most of the numbers are pretty close to 1, which seems to validate the idea of using the future contract as a proxy for ETF.

Some underlyings can be traded through different ETFs (or ETNs). Crude Oil, for example, has several ETFs aiming to capture its performance (I picked DBO, OIL and USO for this test). We can see that there are some differences between the correlation figures of these three ETFs. Not all ETFs seem to track the underlying as well….

A more obvious case of divergence is between the two Cotton ETFs: BAL and COTN (traded in London). The COTN ETF has a low correlation reading on previous-day returns (0.319). Obviously, the fact that the two instruments are trading in different timezones (London vs. NY) will cause their daily performance returns to differ as the market still moves after the London close. This would be something to also keep in mind when choosing an ETF/future pair.

Note on Correlation: Pearson vs. Spearman

I have chosen the Pearson correlation method for these calculations, as the linear relationship between the variables under comparison was what I felt needed to be looked at. However there are several “limitations” of using this correlation calculation. One of them is the assumption of the data being normally distributed, resulting in the calculation being very sensitive to outliers.

This is best illustrated with the correlation of “t-1 Returns” between the COTN ETF and its corresponding Cotton future contract. Back in September 08, the ETF price dropped by 60% before gaining 75% the next day. I seem to remember this was related to a scare around AIG-backed ETFs/ETNs, when the insurance giant was rumoured to go bankrupt.

For the purpose of a test, I simply removed these 2 extreme returns from the file and re-ran the correlation calculation. The figure came in at 0.518. A large increase, obtained by simply removing two outliers. This highlights the point about Pearson correlation not being the most robust with regards to outlier data points.

There are other methods for calculating correlation, and Spearman’s rank method does not make any assumptions on the distribution of the data. It can also detect some non-linear relationship (which we were not interested here). Nevertheless, I re-ran the whole set using this alternative method. The figures were approximately identical – except for the case of COTN.

The Spearman correlation coefficient for COTN v CT using both original and modified files were near-identical at around 0.61. No big jump by removing two outliers.

Note on ETFs

ETPs (Exchange Traded Products), be they called ETF, ETN or ETC all come in different flavours. They can be structured in different ways and track different benchmarks, even when they are associated to the same market. They also add an extra layer and therefore additional counterparty risk on the intermediaries, costs/fees and tracking error.

A good example of the differences between some products can be found in this ETFdb article: USO vs. OIL: A Better Crude Oil ETF?

Finally, it is not (yet?) possible to trade the equivalent of a fully diversified futures portfolio with ETFs, as all markets are not represented in the ETF universe.

Appendix: R Code

Once again, I used R to generate the various correlation calculations. The idea being that one it is coded up it can be applied very quickly to any data.

The code takes a parameter file containing the files to be compared (1st column = description, 2nd and 3rd columns = file names) and the correlation calculation type. See sample here.

Here is the code below. Hopefully the comments are clear enough to help you understand the logic. This might not be the most perfect way to implement this but it does the job:

First, it defines a function to perform the calculations on two files (ie ETF vs. future):

# Define correlETFFut function
correlETFFut <- function(params, i, cor.method,...) {
 
  colsFut <- c("date", "O", "H", "L", "C", "Vol", "OI", "Con", "UC")
 
  # Adding headers
  cols1 <- colsFut
  cols2 <- colsFut
 
  # Read in data sets
  data.1 <- read.csv(toString(params[i,2]), col.names=cols1)
  data.2 <- read.csv(toString(params[i,3]), col.names=cols2)
 
  # Join both data sets
  data.merged <- merge(data.1, data.2, by.x="date", by.y="date")
 
  #Calculate correlation on Open, High, Low and Close
  cor.O <- cor.test(data.merged$O.x, data.merged$O.y,method=cor.method)
  cor.H <- cor.test(data.merged$H.x, data.merged$H.y,method=cor.method)
  cor.L <- cor.test(data.merged$L.x, data.merged$L.y,method=cor.method)
  cor.C <- cor.test(data.merged$C.x, data.merged$C.y,method=cor.method)
 
  #Calculate correlation on t-1 Rtn
  log.diffs1 <- log( data.merged$C.x[2:sum ( !is.na ( data.merged$C.x ) )]/data.merged$C.x[1:(sum ( !is.na ( data.merged$C.x ) ) - 1)])
  log.diffs2 <- log( data.merged$C.y[2:sum ( !is.na ( data.merged$C.y ) )]/data.merged$C.y[1:(sum ( !is.na ( data.merged$C.y ) ) - 1)])
 
  cor.Rtn <- cor.test(log.diffs1, log.diffs2,method=cor.method)
 
  #Calculate correlation on t-5 Rtn
  log.diffs1W <- log( data.merged$C.x[6:sum ( !is.na ( data.merged$C.x ) )]/data.merged$C.x[1:(sum ( !is.na ( data.merged$C.x ) ) - 5)])
  log.diffs2W <- log( data.merged$C.y[6:sum ( !is.na ( data.merged$C.y ) )]/data.merged$C.y[1:(sum ( !is.na ( data.merged$C.y ) ) - 5)])
 
  cor.RtnW <- cor.test(log.diffs1W, log.diffs2W,method=cor.method)
 
  #Calculate correlation on t-30 Rtn
  log.diffs1M <- log( data.merged$C.x[31:sum ( !is.na ( data.merged$C.x ) )]/data.merged$C.x[1:(sum ( !is.na ( data.merged$C.x ) ) - 30)])
  log.diffs2M <- log( data.merged$C.y[31:sum ( !is.na ( data.merged$C.y ) )]/data.merged$C.y[1:(sum ( !is.na ( data.merged$C.y ) ) - 30)])
 
  cor.RtnM <- cor.test(log.diffs1M, log.diffs2M,method=cor.method)
 
  #Put it all into cor.all
  cor.all <- c(sum ( !is.na ( data.merged$C.x )), data.merged$date[1], signif(cor.O$estimate,5), signif(cor.H$estimate,5), signif(cor.L$estimate,5), signif(cor.C$estimate,5), signif(cor.Rtn$estimate,5), signif(cor.RtnW$estimate,5), signif(cor.RtnM$estimate,5))
  return(cor.all)
 
}

Second, a function reads the parameter file of all files to be compared, iterates through it and call the first function for each pair of files, building an array of all results:

# Define fullCorrel function
fullCorrel <- function(file, cor.method,...) {
 
  # read input file
  comps <- read.csv(file, header=F)
  for (i in 1:dim(comps[1])[1]){
    cor.tmp <- correlETFFut(comps,i,cor.method)
    if (i==1) cor.full <- cor.tmp else cor.full <- cbind(cor.full, cor.tmp)
  }
 
  rownames(cor.full) <- c("Record Count", " Start Date", " cor Open", " cor High", " cor Low", " cor Close", " cor Rtn", " cor RtnW", " cor RtnM")
  colnames(cor.full) <- t(comps[1])
 return(t(cor.full))
}

Finally, set your working directory (where all the files are) and call the second function with the relevant parameters (here the function is called twice with the same parameter file and two different correlation methods (make sure to update the working dir to your own path):

# Set Working Dir - Replace with your own path!
setwd("D:/ATS/Blog/Posts/ETF v Futures/Data")
 
# Run Pearson correlations
fullCorrel("ETFvFut.txt", "pearson")
 
# Run Spearman correlations
fullCorrel("ETFvFut.txt", "spearman")

Below are included some sample files to run the code above:
Sample_Data

In order to avoid bad surprises in live trading, the maxim forming this post title should apply to as much of a trading system as possible. The goal is to ensure back-testing results are as realistic as they can.

Input data is one component of a trading system and the area this post focuses on, more specifically roll-over methodologies for futures contracts.

Real Monthly Trading Also Means Monthly Roll-Over

One of the triggers for this post was a comment on the Monthly Trading post with regards to the futures data used for the test. In order to have comparable results between the daily and monthly instances of the systems tested, I simply and mechanically reused the same portfolio of instruments…

However, futures contracts in the portfolio roll on Open Interest. What this practically means, is that a trader using this methodology would have to monitor OI every day to decide whether to roll to another contract. Not much of a monthly process.

I still think that test had some value, but it is important to realise that the monthly instance in that test was not representative of a pure monthly process. Instead we were actually simulating a trading strategy with monthly signal taking but daily roll-over monitoring process.

It highlights the fact that it is important to understand what the data used for testing actually represents, and to roll futures contract data in the same way as they are rolled in real-life.

This post is a good opportunity to revisit the concepts of generating continuous futures data and some of the older posts I had written on the subject using CSI (here and here).

ETF: Outsource your Contract Rolling Process

Another reason why I think that the monthly trading test has some value is because the individual monthly trader might decide to trade ETFs instead of futures.

ETFs can be seen as simple wrappers of futures contracts: they trade continuously and their managers take care of rolling their underlying positions with a predefined methodology and timing, charging you a fee in the process.

Of course, this is not futures Trend Following any more, and using ETFs lacks the inherent leverage embedded in futures instruments. But it is possible to obtain decent performance with Trend Following on ETFs (check this ETF Trading book by Anthony Garner for example). An assumption in that book (which I will attempt to verify in a next post) is that futures can act as proxies for ETFs/ETCs.

Contract Rolling and Testing

Rolling contracts is a process necessary to generate a continuous time-series for back-testing futures instruments. There are many different ways to perform this operation (some of which are covered just below). What is important is that the continuous contract should be a realistic representation of one’s trading.

Proportional vs Point-Based Adjustments

At the time of writing these posts on continuous contracts, I was leaning towards the idea that proportional adjustment was the way to go:

“This ensures a constant relative (percentage-wise) relationship between any prices across the trading history“, as opposed to the point-based adjustment, with which “you lose relative price difference (shifting all prices by an absolute amount has that effect on percentage changes: 10 to 11 is +10% but add 100 to both for an extreme case and 110 to 111 becomes less than +1%)“.

When back-testing with Trading Blox, the standard functionality expects the prices to be point-based adjusted, to calculate trade PnL:
(ExitPrice – EntryPrice) * BigPointValue * NumberContracts
With a point-based adjustment, the number of points between entry and exit will represent the actual points the market has moved during the life of the trade, as opposed to proportional adjustment (which would give wrong PnL calculations).

But point-based adjustments prevents us from using relative price difference, such as “check if close today is 5% higher than close yesterday” as an example. This might be an issue for some systems, but there are workarounds for this in Trading Blox though.

We do not have to limit ourselves to one or the other methodology, but we have to understand what the rolling algorithm represents and which data best suit each test.

Of course, you can even go with a more fancy approach to rolling (if this represents what your trading is), like one we had covered here last year with a rolling methodology attempting to optimize roll yield.

Back-Adjusted or Forward-Adjusted?

Another switch on the rolling methodology dashboard (in CSI) is the possibility to perform back-adjustments or forward adjustments. This only determines which price is fixed and in which direction the “adjustment drift” will propagate through the time-series. Forward-adjustment picks the earliest price as the reference price and moves each following contract by the required adjustment amount/ratio to fill the “rolling gap”. Backward-adjustment is the opposite: reference price is last price and adjustment propagates backward.

In order to get recent prices in line with real price levels, most traders prefer back-adjustment. From a back-testing point of view, it should not have much impact. However, the chart time-series will look quite different.

To illustrate this, here is an example chart plotting the same underlying (Crude Oil) since 2007, with four different standard CSI rolling methodologies:

• Proportional, Forward Adjustment
• Proportional, Backward Adjustment
• Point-based, Forward Adjustment
• Point-based, Backward Adjustment

Mix of Point and Proportional, Back and Forward Adjustments

Going back to the monthly trading test, I did take my data “for granted” as it works for most cases… but not really for this one. It does emphasise the point that “Trade what you Test and Test what you Trade” should also apply to data and rolling methodologies, which need to be checked/reviewed for each new system.
 
 

I use CSI as my main source of data, including for all the research published on the blog and I have no problems recommending them.

So, when they approached me with a proposition for a “partnership”, which could also benefit the blog readers, I did not really hesitate.

I hope some of you can benefit from this offer – and unlike the January sale, it will be there all year.

Disclaimer (in the interest of transparency): any reader signing up to CSI using the coupon code would result in an affiliate commission – on top of your discount. A win-win proposition in a way. Note that I am a paying CSI customer and have been enjoying their data services for 2 years now (I use the Gold World Futures package and am about to get the Gold Stocks package).

For more info on CSI, please find below is a list of posts I ran a while ago on CSI data, their API, etc.:

• I just got myself an Unfair Advantage…
• Continuous Contract options
• Unfair Advantage API: Retrieve Back-Adjusted Contracts function
• Unfair Advantage API Code (C#): Extract Futures Continuous data
• CSI Raw Contract Data Extractor

Anscombe's quartet

Pretty different graphs, right?

Yet, you might be surprised to hear that they are all identical in the light of common summary statistics: mean, variance, (Pearson) correlation and linear regression.

Here are the exact figures:

 
Details of all dataset points are at the bottom of this post if you fancy double-checking this yourself.

Credits for this amusing and quite extraordinary illustration go to statistician Francis Anscombe, who created this Anscombe’s Quartet back in 1973 (more info on wikipedia).

I believe Anscombe’s main point was to prove that statistics can be misleading (a fact greatly abused by today’s media?) but also that outliers can have a strong impact on statistical properties. Surely two points that can impact your trading system design and testing conclusions.

Exploratory Data Analysis

Anscombe’s quartet is often used to highlight the importance of graphical exploration of the data for analysis. This concept is behind an area of statistics known as Exploratory Data Analysis. From wikipedia:

Exploratory data analysis (EDA) is an approach to analysing data for the purpose of formulating hypotheses worth testing, complementing the tools of conventional statistics for testing hypotheses. It was so named by John Tukey to contrast with Confirmatory Data Analysis, the term used for the set of ideas about hypothesis testing, p-values, confidence intervals etc. which formed the key tools in the arsenal of practising statisticians at the time.

The objectives of EDA are to:

• Suggest hypotheses about the causes of observed phenomena
• Assess assumptions on which statistical inference will be based
• Support the selection of appropriate statistical tools and techniques
• Provide a basis for further data collection through surveys or experiments

 
 
In this interview, David Harding, from Winton Capital Management, stresses the importance of research and statistical data analysis in his company. The two books he highlights cover Exploratory Data Analysis (Understanding robust and exploratory data analysis and Nonparametric Statistical Inference).

Correlation

Another topic that Anscombe’s quartet illustrates is correlation (and non-linearity).

The most common measure of correlation is Pearson product-moment correlation coefficient. Its calculation only derives the linear dependence between two variables. If a non-linear relationship exists between these variables, it will go undetected by the Pearson correlation.

To illustrate this, consider Anscombe’s top right dataset, exhibiting a perfect functional relationship between x and y. The correlation should be 1, yet the Pearson coefficient is irrelevant at 0.816.

Consider the following more extreme case (of the type y=(x-a)^2 + b), where a 100% relationship translates in a zero linear/Pearson correlation:

Taleb is probably a bit extreme when he says:

Anything that relies on correlation is charlatanism.

But using correlation on market data, typically described as non-linear, can have its pitfalls.
I was re-reading Ralph Vince’s paper on Leverage Space Model not long ago and he describes the fallacy of correlation:

Correlation fails when you are counting on it the most – at the (fat) tails of the distribution. The point is evident throughout this study; big moves in one market amplify the correlation between other markets, and vice versa

In the paper, he explains that correlation between two instruments becomes much stronger during periods of large standard-deviation moves (when potentially diversification/non-correlation would be required most).

RIP Benoît Mandelbrot (20/11/1924 – 14/10/2010)

 
Finally, I would like to pay tribute to Benoît Mandelbrot, who passed away last week.

Mandelbrot has been dubbed “The Father of Fractals” and spent a big part of his life trying to understand how markets work. Looking at market data through the prism of fractal geometry, he tried to uncover non-linear, chaotic relationships in the data.

I’ll reiterate my recommendation to read his book, The (Mis)Behavior of Markets – one of the most enjoyable rebuttals of the Efficient Market Hypothesis. Although I have not (yet) found a practical application of the principles described in his book and papers, Mandelbrot’s attempt to define a new paradigm to understand the markets is interesting – and it is a book that reinforced my belief and understanding of Trend Following (despite not covering this topic explicitely).

One of his main finding is that price changes in financial markets do not follow a normal (Gaussian) distribution, but rather a fat-tailed distribution (Levy, Paretian or also called power-law distributions). This is potentially one of the main reasons for Trend Following to work (well that’s my interpretation).

I might actually pick up that book again and read it soon (and write a summary post on it).
 
 

Appendix: Dataset point coordinates:

We have seen previously that backwardation and/or contango can induce a fairly large drift between the performance of an instrument’s spot market and its corresponding futures market.

This phenomenon can be described as roll yield of futures trading and I suggested it was one of the four components in Trend Following returns. As per that last post’s conclusion, breaking down the returns should allow to focus, study and improve a trading system’s individual component – and this is what we are doing here with the roll yield, by looking into the Commodity Yield Curve.

The roll yield had a negative impact in the Crude Oil example, however let’s explore how it could be turned into an extra source of profit…

Term Structure – or Yield Curve

Term structure best applies to interest rates analysis. If we take the example of US Treasuries; they cover a wide range of maturities, from T-Bills (one year or less maturity) , T-Notes (one year to ten years) to T-Bonds (up to thirty years). For any given date, each treasury is priced depending on its maturity and an implied yield can be derived for each maturity. Charting these yields for each maturity gives the yield curve, also called term structure of interest rates.

Below can be found a “normal” (upward sloping) yield curve for US Treasuries (from last Friday) showing that yields went slightly down in the middle of the curve whereas the short and long ends of the curve did not change much:

The yield curve, or term structures of interest rates is a major economic indicator, closely watched by many traders to gain insights in the market. You can read more about it at investopedia and wikipedia.

Application in Futures Trading

Term structure can be similarly transposed to futures trading, which deals with the same concept of trading the same instrument (i.e. where a commodity like Corn replaces money used in the case of the classic yield curve) with different maturities (all different contracts and their various expiry/delivery months).

Note that the term Commodity Yield Curve is a bit of misnomer as the concept could be applied to any instrument traded in different maturities (i.e. all futures for instance).

Commodity/Futures Yield Curve: How to Build it?

The futures yield curve is a representation of the backwardation/contango rate for the different maturities of futures contracts. The contango/backwardation rates are calculated by taking the price difference between the spot/cash market price and the futures contract price. This difference can be expressed as a percentage (for x number of days: time to expiration of the futures contract) and then annualised to calculate an annual contango or backwardation rate, representing an implied yield for the specific maturity.

For example, if the spot market trades at 70 and the October contract (expiring in 90 days) trades at 71, the yield can be calculated as follows:

1. Price difference = 71 – 70 = 1
2. Percentage price difference = 1/70 = 1.43% (for 90 days)
3. Annualised contango rate (yield) = (1.43% + 1)^(365/90) – 1 = 5.92%

Performing this calculation for each contract/maturity would allow to chart a futures yield curve.

Volatility of the Yield Curve

I calculated and charted the Crude Oil yield curve in order to analyse and visualise the evolution of the contango/backwardation rates, both across the historical timeline and the range of contract maturities. This was done as discussed above for all contracts available for a given date.

Here is a sample chart using data from December 03 to January 04 where Crude Oil was trading in backwardation:

Each “ribbon” on the chart represents the yields for all 18 different maturities (1=short, 18=long) for a given date: each ribbon represents the yield curve for that day. All yields are negative because of the backwardated aspect of Crude Oil at the time. Backwardation rates have reasonable values (-15%/+5%) and the associated yield curves vary slightly across time, albeit maintaining a similar pattern.

However this snapshot was taken at relatively quiet times in the market. Fast forward to 2009 and you can see that the market now exhibits contango, with rates spiking to much higher values:

Adding any more dates to this chart would make it hardly readable, but to give an idea of how the term structure of Crude Oil can vary over a longer period of time, I have simply plotted the shortest end of the curve (i.e. yields for the contract with closest maturity), which you can imagine as a “vertical slice” of the 3-D chart alongside the date axis, where maturity = 1. Here is the chart:

There are spikes, alternation between contango and backwardation. It clearly displays some volatility and, for lack of a better term, a character of its own.

Note that because the yield is calculated as an implied value based solely on the price difference between contracts, some discrepancies in these contract prices (for whatever reason) would impact the curve in a more volatile and “artificial” way than if it did purely represent the fundamental yield factors (cost of carry, supply/demand and shortage, convenience yield, etc.)

Some Observations

The main observations are:

• Yield curves and their associated contango and backwardation rates evolve in a fairly volatile fashion across time.
• For any given date, different contracts and their associated maturities offer different yields, possibly within a wide range of values.

This can add an extra dimension to your trading decisions. The yield curve offers other opportunities (than the traditional front-month contract) when wanting to buy or sell any instrument via futures trading. These extra opportunities can be turned into an extra source of profit.

This is really an introduction to a next post, in which we will look into how we can use this information and apply it to a trading system to improve its roll yield component and enhance its overall performance.
Stay tuned…

CSI Data Extractor: Utility tool to extract CSI data in plain text files

I have mentioned before that I use CSI as my main provider for Futures data. Overall, I judge CSI’s service as good quality – however one drawback is the closed aspect of the data access. And I decided to write a utility to make it more “open”.

To access the historical data, you have to go through their software (Unfair Advantage), which extracts and decrypts it from their proprietary binary format. You can always “build” historical files to disk, but these are usually pre-processed with a special algorithm (ie. back-adjusted, etc.).

What more, if you decide to stop your subscription to CSI’s data services, you will not be able to access any data any more, even historical data downloaded under the subscription contract.

For some of the reasons above, it might be interesting to be able to extract/store CSI’s raw data in plain text files. This would allow for:

• Browsing the data outside of the CSI’s Unfair Advantage software
• Applying back-adjustment or other manipulation algorithms not supported by CSI’s UA (such as calculating the yield curve)
• Keep the historical data independently of your CSI subscription

Note: what I consider CSI raw data is the actual daily information for each contract traded on any day. For example, in the Corn market, there are currently 15 different contracts trading (with different maturities), each with their own Open, High, Low, Close, Volume and Open Interest.

CSI API to the rescue

Now, you could easily go to Unfair Advantage and pick every single historical contract that you want to extract… but this would be a fairly tedious job

Luckily for us, the API allows for programmatic access to CSI’s data to automate the process of extracting all contracts that we want – so I decided to write a little utility to do just that. And now my data is really mine!

Below is a zip file containing the files to copy to “install” the utility:

CSI-DataExtractor.zip

Note that you need to have Unfair Advantage installed on the machine running this exe (which does not come with any guarantee by the way… but I have been using it fine so far).

The operation is very simple: just enter CSI’s number for the market you want to extract and choose which path the files need to be extracted to.

Hope you’re finding this tool useful…

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)

Data - image by Pixelsior@flickr

In my other job, at a big investment bank, one of our main focus on a daily basis is DATA. Making sure that the hundreds of feeds and millions of records get uploaded correctly and contain the right information is key to a smooth, successful day.

There is surely not as much data to interact with in automatic/systematic trading but getting data right is equally vital to ensure quality trading or backtesting results.

Even with the best data package, there can be many pitfalls to look out for when trading or backtesting a systematic trading strategy. Below is a non-exhaustive list – please let me know in the comments if you think of anything to add.

Data Errors

Despite CSI being deemed one of the most reliable End-Of-Day Data providers, they still send you bogus data (albeit rarely) and you might want to run additional checks to see if the data is valid. Some ideas of checks to perform:

• Open, Low or Close above the High
• Prices = 0
• Price/Volume/Open Interest changes above tolerance
• Gap in the dates
• etc.

Of course, this will not catch all errors, but will give a pointer on large data anomalies.

Static Data

Back in the bank job, part of the system data which changes infrequently (hence static) is subject to 4-eyes approval (basically enforcing that another pair of eyes double-checks the correctness of the data). The idea is that, as this data drives the calculations in the system, any error would give flawed calculation results, and needs double-checking.

The equivalent in the automatic trading system world is the futures contract specifications and formats information (price quote type, decimal places, native currency, point value), contained in the FuturesInfo file in TradingBlox. Imagine an error of a factor 10 on the contract point value – this could have you get in a trade at 10 times the intended position size!

CSI appears to contain several of these errors (they seem to have less focus in getting this type of data right) and even official exchange websites have been seen to quote incorrect information. Reconciling both would be a starting point to getting the right information.

A possible solution to mitigate this problem (when starting trading a new, exotic instrument) is a quick in-and-out trade to double-check the actual information on your broker statement.

Data History and Overrides

The CSI data downloading process firstly retrieves all new prices to be stored in the Unfair Advantage database. It then applies potential corrections to past prices, in case errors were present. Rollover logic is applied and the full data history file is generated (to be used in the trading/backtesting software).

The change in data resulting from error corrections might change the outcome of past trade results. For example, a data correction in the recent past history might trigger a breakout and therefore a trade, now open. How would you handle this?

Additonally, if the historical data contains errors not corrected by your data provider, you might want to apply overrides to correct them yourself. However, since the price history gets re-written everyday, you need to have some form of automated overrides upload.

You might also want to make daily backups of price files to be able to compare when price corrections have taken place (and run your system with the exact data that was available on that day).

Other Data

Data such as historical FX/interest rates or Margin data are necessary to calculate some of the system statistics (including overall performance).

The FX rates are mostly used to convert the P&L from the native contract currency to the main account currency.

Interest rates would be used to calculate the interest earned on the account, which impact on the overall performance is not negligible for a long-term Trend Following system.

The Margin data is useful to determine the total amount of margin your system requires at any one time, with statistics such as the Margin-to-Equity ratio. Unfortunately, it is fairly hard to get hold of this type of data, which changes fairly often, at the discretion of the exchanges. Most backtesting platforms I have come across use a static margin number, which is fairly inaccurate.

To get more accurate results, you could run your backtesting simulations using historical (real or simulated) margin requirements. A method suggested on various forums to estimate historical margin requirements is to run a regression analysis between margin requirements (known history) and underlying instrument volatility (using different potential measurements such ATR, High-Low, Bollinger Bands, etc.). Finding a good-fit model would allow you to extrapolate past margin requirements based on volatility.

Holidays are also useful data to collect, to be able to know which exchanges are open when (note that exchanges can have different holidays even when they are in the same country).

Backup Provider

Your main data provider might be down or unavailable (hopefully on rare occasisons only). A backup, alternative source of market data might come in very handy for those special days.

Back-Adjustments and Rollovers

There is unfortunately no ideal or universal solution to roll and back-adjust contracts. As discussed earlier, a proportional back-adjustment allows for better representations of prices, keeping historical ratio values between all prices. However, backtesting systems usually require point-based back-adjustements in order to calculate the P&L of each trade correctly. You might need to maintain several sets of back-adjusted prices to be used in your simulations.

Furthermore, different instruments might require different rolling logic. Different delivery months in agricultural futures, for example, might relate to different crops. It might not make as much sense to roll from one ot the other in the same way that could be done with equity index futures.

Do you get the rollover information on rollover day?
Let’s assume that, on Thursday, you receive the data indicating that a roll should have occurred on Tuesday (ie. roll based on Open Interest or Volume shifts). Let’s also assume that an entry signal was triggered on the Wednesday and that the high volatility of the front contract would have prevented the system from entering the trade (risk management threshold breached) – whereas the next contract’s volatility, being much lower, would have allowed the trade. In real life, Wednesday’s trade would not take place (as the system does not know yet that the contract should have rolled on Tuesday). In backtested life however, Wednesday’s trade would take place, as the system would assume the roll has taken place on Tuesday. These things do actually happen!

You might want to test using the data as you would have received it, instead of a constructed back-adjusted series, potentially containing hindsight information.

LME Data

Some instruments deal in “strange fashion” and the London Metal Exchange is a perfect example of this. Trading on the LME is done by purchasing a forward contract with a given maturity (or prompt date). For instance, you might purchase a 3-month forward contract, which will run for the next 3 months gradually maturing to converge to the Cash price (at expiration), including the effect of contango or backwardation. However, the data quoted from CSI (and probably any other data provider) is a new daily quote for a new 3-month forward, which does not reflect real trading.

You would probably need to construct a custom price series with a bespoke algorithm to be able to reflect and backtest real-life trading. This topic was further discussed on a tradingblox forum post

Need for an ETL layer?

Drawing the parallel to the bank job again, where we use specialised tools/platforms to deal with data (Extract-Transform-Load = ETL); I believe that a similar layer between your data provider(s) and your trading/backtesting system, used to perform data checks, archive, roll and back-adjustments, etc. (all poinst discussed above) is necessary. It could also be used to add extra functionality such as creating spread time series to be traded/backtested as new instruments.

The data issue might not be as much of a trading problem as an IT one, nonetheless it is an important one that needs to be addressed – or it could impact your trading and backtesting results.

Why Detrend?

The main premise for detrending data is to remove the underlying trend effect on the strategy. This is due to the position bias that the strategy can have (eg being long more often than short).

Let’s say your strategy is long 70% of the time; and a bull market is currently taking place, there is a higher probability that you will be profitable. This could be due to the long position bias of your strategy – which might well become unprofitable during bear markets.

How to Detrend price data?

A simple way to detrend the data is to:

1. Calculate the underlying trend in your backtest data
2. Derive the “daily drift” from the underlying trend
3. For each trade in the backtest, adjust the trade return by subtracting the daily drift for each day in the trade

For this it is easier to use log of price differences.

Step 1: The underlying trend is defined by the difference in price from the start to the end of the backtest:

Trend = LOG(Price_end / Price_start)

Step 2: Simply divide the underlying trend by the number of days in the backtest data:

Daily Trend Drift (DTD) =
LOG(Price_end / Price_start) / NumDays

Step3: For each trade, apply the detrending adjustment calculated above:

LOG(TradeReturn) =
LOG(SellPrice / BuyPrice) - DTD x NumDaysTrade

Note that the backtest is executed on normal, non-detrended data. It is the results that are adjusted by applying the detrending procedure.

Assuming a full reinvestment, the formula for the full detrended backtest result containing “n” trades would be as follows:

Detrending for Trend Following

It might sound paradoxal to remove the data component that is at the heart of the strategy (the trend)… On one hand, you realise it might make sense, as the underlying trend is at a much higher timeframe than the trends identified by the strategy. On the other hand, the detrending concept assumes a dissociation of the position bias and the underlying trend.
Is it true for Trend Following?

We could argue that if the underlying trend is bullish, bullish trends will be prominent at a lower timeframe (vice-versa for bearish trends). By the nature of Trend Following, we would therefore expect the strategy to spend more time in long positions (long position bias). Because Trend Following is exactly that: it adapts and goes in the direction of the trend.

In short: Trend Following automatically generates position bias in the direction of the trend

If these two variables (position bias and underlying trend) can not seem to be dissociated for Trend Following, it might not make sense to apply detrending for Trend Following.

Other complications

The principle of applying a linear/proportional detrending adjustment assumes that the position bias is constant over time. It seems to ignore the potential for cyclical variations, possibly in line with major trend cycles (as would be expected with Trend Following).

For futures data, the detrending impact would be dependent on the method used for back-adjusted contracts. Point-based back-adjustment, for example, does not respect the proportional ratio between prices and would flaw the underlying trend calculation.

Finally, there are other factors that might need to be taken into account such as Contango and backwardation. The roll yield in futures trading, having possibly as strong an impact on the strategy’s return.

What are you thoughts on detrending for backtesting?…

some Autocorrelation representation by kylemcdonald@flickr (CC)

Kurtosis only?

In an earlier post, I discussed how fat-tails are a reason for Trend Following success (or in technical terms: the excess kurtosis of price distributions).

However, there is something unsatisfying in that explanation: if the kurtosis was the sole source of Trend Following success:

• Random entries should work as well as any other entries
• Strategies such as buying Out-of-The-Money (OTM) options (think Nassim Taleb for example) should exhibit similar performance to Trend Following (with the advantage of being a rather simpler strategy)

Something Extra?

I recently came across this paper (PDF) explaining that Trend Following and OTM options buying are strategies exhibiting similar performance profiles. However, the conclusion of this paper was that Trend Following showed superior performance.

Additionally, there is definitely a measurable edge to Trend Following entries (such as this Donchian breakout e-ratio calculation shows). Random entries would not show such an edge.

So, there must be something extra to the kurtosis story explaining Trend Following success…

Autocorrelation

One hypothesis that I want to investigate further is autocorrelation (also referred to serial correlation).

One of the main principles of Trend Following entries – in the face of conventional wisdom – is:

Buy High and Sell Low

Well, it should really say “Buy High, Sell Higher and Sell Short Low, Buy Back Lower”. The point is that Trend Following entries are made at extremes, in the direction of the extremes.

If market exhibit positive autocorrelation at extremes, it can be derived that following the direction of the extreme moves should provide an edge (positive expectancy). This would explain why Trend Following entries perform better than random entries and why Trend Following is a superior strategy to buying Out-of-The-Money options.

Calculation Project

Now, this sounds all well and fine in theory but does this stack up to verification?

To check this, I am planning to run some calculations on historical prices and see if markets exhibit such autocorrelation at extremes. Another aspect that will be interesting to look into is whether this autocorrelation evolves over time and whether these autocorrelation levels are autocorrelated themselves (ie is there some degree of predictability in the autocorrelation evolution).

Now, please note that I am stepping out of my comfort zone here: my “heavy maths” days are quite far behind me and I know that using statistics can be a minefield (because it is so easy to use it in an incorrect manner). For example, the “standard” correlation calculation (Pearson’s correlation coefficient) only determines linear dependence – although market data is non-linear. Might set myself up for some hardship but as we say in French: “Qui ne risque rien n’a rien” (no pain, no gain).

I am also thinking of getting one or two Econometrics books to give me a headstart on this. But if any of you clever readers have any suggestions or tips on any of the above, please let me know.

Please bear with me and stay tuned.