CSI Data Extractor: Utility tool to extract CSI data in plain text files

I have mentioned before that I use CSI as my main provider for Futures data. Overall, I judge CSI’s service as good quality – however one drawback is the closed aspect of the data access. And I decided to write a utility to make it more “open”.

To access the historical data, you have to go through their software (Unfair Advantage), which extracts and decrypts it from their proprietary binary format. You can always “build” historical files to disk, but these are usually pre-processed with a special algorithm (ie. back-adjusted, etc.).

What more, if you decide to stop your subscription to CSI’s data services, you will not be able to access any data any more, even historical data downloaded under the subscription contract.

For some of the reasons above, it might be interesting to be able to extract/store CSI’s raw data in plain text files. This would allow for:

• Browsing the data outside of the CSI’s Unfair Advantage software
• Applying back-adjustment or other manipulation algorithms not supported by CSI’s UA (such as calculating the yield curve)
• Keep the historical data independently of your CSI subscription

Note: what I consider CSI raw data is the actual daily information for each contract traded on any day. For example, in the Corn market, there are currently 15 different contracts trading (with different maturities), each with their own Open, High, Low, Close, Volume and Open Interest.

CSI API to the rescue

Now, you could easily go to Unfair Advantage and pick every single historical contract that you want to extract… but this would be a fairly tedious job

Luckily for us, the API allows for programmatic access to CSI’s data to automate the process of extracting all contracts that we want – so I decided to write a little utility to do just that. And now my data is really mine!

Below is a zip file containing the files to copy to “install” the utility:

CSI-DataExtractor.zip

Note that you need to have Unfair Advantage installed on the machine running this exe (which does not come with any guarantee by the way… but I have been using it fine so far).

The operation is very simple: just enter CSI’s number for the market you want to extract and choose which path the files need to be extracted to.

Hope you’re finding this tool useful…

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)

I wanted to start this year with an idea I left off last year, when investigating robustness: Using the median as a more robust tool than the ubiquitous mean (or average). This will be covered in the next post via a TradersStudio simulation results. To do that, I had to re-import some data from CSI Unfair Advantage to TradersStudio. I thought a “how-to” post on this would be useful – if only as an aide-memoire for next time (note-taking and documentation are important after all…)

Extract CSI Data

The first step is to extract the data from the CSI database into text files
(by the way, CSI have a 20% discount running until the end of January – just check their homepage).
This can be done with code using the Unfair Advantage API.

Pick all the instruments to be extracted and create a “contracts.param” file containing the CSI number for those instruments in the first column (comma-separated file with other columns being ignored). The CSI numbers can be found on their website (like here for the CME Futures) or in their Unfair Advantage software UI. Running the EXE code will generate one file for each instrument (proportionally back-adjusted contract) in the following format:

19900403,199006,0,240,110,918,503,2,0,1.59,1.655,1.58,1.635,1.635,1.635,1.635
19900404,199006,0,84,127,656,721,3,0,1.6,1.65,1.599,1.62,1.62,1.62,1.62
19900405,199006,0,57,112,554,885,4,0,1.615,1.635,1.61,1.615,1.615,1.615,1.615
19900406,199006,0,62,136,436,1092,5,0,1.615,1.635,1.61,1.625,1.625,1.625,1.625

Load in TradersStudio

The second step is to load the files in TradersStudio. Copy all generated text files in a dedicated data directory (you might have to rename them as TradersStudio allowed filename length is fairly short, i.e. 10-15 characters).
Create the relevant mapping file (txtinfo.txt) in the data directory. The mapping file should read:

delimit=,
skip=0
dt=1
ti=0
op=10
hi=11
lo=12
cl=13
vol=6
oi=7
dtformat=CCYYMMDD
tiformat=

In TradersStudio, menu Data Manager/Add Data Links, point to the relevant data directory with the following options:

If all goes well, this will load the data in the system (errors I encountered were due to file names being too long).

Data Universe: Contract meta-data

The third step is to populate the meta-data (margin, min tick move, etc.) for each instrument in TradersStudio. This is done via the menu Data Manager/Data Universe.

Once done, the instruments can be used just by adding them to any session.

As mentioned earlier the next post will be a test comparing moving average and moving median indicators.

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)

Why I chose TradersStudio

I did consider a few options before buying TradersStudio (Amibroker, TradeStation, Trading Blox, WealthLab, NinjaTrader) and based on feature analysis, recommendations and actual testing, I settled on TradersStudio.
My requirements were for a standalone platform that would allow me to test any system, with money management and multiple system interaction.
I did hesitate with Trading Blox, which I enjoyed testing (demo version), but in the end the price sorted the argument (Trading Blox is $3,000 for similar functionality).
For more info on what each product offers, I would definitely recommend heading to EliteTrader where many platform comparison posts can be found. Please also feel free to ask me questions in the Comments section below.

First impressions

Not so good… There is no demo available on the website, you can not download the app from the website once purchased and the manuals you receive after waiting a few days seem quite unprofessional (despite being over 300 pages).
However, the install is quite easy and you can be up and running following examples from the manual in a few minutes.

Main Features

The app offers some interesting features which help me take the decision:

Integrated Systems testing

TradersStudio uses different hierarchical levels: Instrument, System, Session, Trade Plan. This allows you to test a complete system (i.e. more than simple entry and exit testing). A Trade Plan can contain several sessions, which in turn can contain several systems and instruments with possible interaction between them. Additionally, Money Management options can be tested as part of the Trading Plan. This makes it much more realistic than a collection of independent strategy tests.
There is also to the concept of virtual systems that allow you to run a system in “monitor” mode and only use it for decision making on other “live” systems (i.e. enter a trade in real system only after virtual system posts 3 losing trades).

Customisation

TradersStudio implements its own Macro language, which appears to be a cross between TradeStation EasyLanguage and Visual Basic. Theoritically you can implement whatever extra functionality you require (indicator, software add-in, etc.). Moreover there is an import tool that automatically converts TradeStation EasyLanguage code, which should allow to reuse available code.

Data-independent

You can basically load data from whatever source you wish in any supported format (any standard text file format will do). I personally use Unfair Advantage from CSI and getting it loaded to TradersStudio was a breeze (see Unfair Advantage posts for more information on CSI UA -> TradersStudio)

Walk Forward

Walk forward is a useful concept to use in back-testing to avoid over-optimization and curve-fitting. TradersStudio implements this functionality and runs it automatically which means less manual workaround for you to deal with.

Documentation issues

My main grudge is about the manuals. Despite being over 300 pages it is still missing an essential reference guide for programming using their language (there are a few examples but they do not cover the whole range of functionality and much is left to be guessed by the user). It is a “Learning by Example” type of manual which can sound a bit patronising at times.
For that reason I have found it hard to seriously get into TradersStudio… but I have just signed up for the yahoo group dedicated to it and will be spending many hours in front of it very soon!

I will explore TradersStudio in more details as I start using it and share in a later post if you are interested.

Getting started with the API

Fire up your favourite IDE/language that supports COM Interoperability (I am using Visual C# 2008 Express Edition), start Console Application project and add a reference to the Unfair Advantage API. For this you just need to “Add reference” and browse to the Unfair Advantage installation folder and select the main EXE (uad.exe). This gives access to the CSI UA API.

Add UA into your project and "Bob's your Uncle!"

Simple extraction code

The rest is fairly simple as long as you understand the RetrieveBackAdjustedContract2 parameters.
The code below takes 2 parameters: an input file name containing the list of markets to extract and a folder name to extract each continuous contract to. It sets specific UA session parameters and calls the RetrieveBackAdjustedContract2 with specific variables to obtain the concatenation desired. Each continuous contract is output to a folder in a .txt file.
The file containing the list of markets is of the following format (CSI number, Symbol, Market name):
…
8,CT,Cotton
…

The code

Please find the code there (very nicely formatted thanks to this great tool by manoli). You can also download the Program.cs file:

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)

CSI main API document (word doc) is accessible publicly online from their document page along with other docs including the full 300 page manual (great for getting a feel of the app).

Pivotal API function

This post will focus on is the API function to retrieve Back-Adjusted Contracts from the Unfair Advantage database.
As the last post on Continuous Contract options showed, there are multiple choices available and these all translate to parameters to the API function: RetrieveBackAdjustedContract2. This is what I consider to be the meat of the API: it queries the proprietary UA database and applies the concatenation algorithms to produce a continuous contract.

27 Parameters!

Yes… that function’s behaviour is driven by the value of 27 properties or variables. There are actually 14 “Session” properties and another 13 variables passed to the function. The aim of these parameters is to reproduce exactly all the options covered in the previous post… and then some.

My own doc

Because I found the CSI doc not ideal (you have to keep flipping back and forth for a description of all 27 parameters) I decided to re-write my own “cheat-sheet” documentation for the RetrieveBackAdjustedContract2 API function.
The rest of this post is that document – formatted into HTML by M$ Word (so please excuse any formatting mishaps). The document contains 2 parts: the UA session properties and the function variables. Hope this helps.
You can also download the document in the RTF format.

Stay tuned for the next episode, where I will share a sample code using the function above to extract a list of Futures contracts. This should illustrate how to use all these parameters.

The document:

FUNCTION: RetrieveBackAdjustedContract2

–>

CONVERSION FACTOR        CONVERSION ACTION REQUIRED

        +6             Move decimal point 6 places left

        +5             Move decimal point 5 places left

        +4             Move decimal point 4 places left

        +3             Move decimal point 3 places left

        +2             Move decimal point 2 places left

        +1             Move decimal point 1 place left

         0             Leave number as it is

        -1             Terminal digit is in eighths

        -2             Terminal 2 digits are in 16ths

        -3             Terminal 2 digits are in 32nds

        -4             Terminal 2 digits are in 64ths

        -5             Terminal 3 digits are in 128ths

        -6             Terminal 3 digits are in 256ths

        -7             Terminal 3 digits are in 32nds & 1/2 32nds.

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)

Continuous contracts allow you to chart Futures contract prices in one long time-series (here Copper ranging from 1965 to 2009)

Let’s look at the different possibilities:

Non-adjusted contracts

Take the front-month contract (nearest expiration date) and roll it over into the next one at expiration date. This generates a series of actual traded prices. But as we discussed in our previous post, it does have negative impacts when used for back-testing because of the potential big gaps at roll-over times.

Standard Back-Adjusted contracts

A solution to this “big gap” problem is to move each contract up or down by a certain amount (the gap value) so that the rollover junction is continuous. This is point-based adjustment and is usualy performed “backwards”. You would start from the current contract, move the previous contract by a set amount, then subsequently move each of the anterior contracts to eliminate each gap you come across until the first contract of the timeseries.

Proportionally Back-Adjusted contracts

But as we saw in the last post, this point-based adjustment is not ideal either, as it loses the relative price differences and can even make past prices become negative. This is where ratio-based adjustment comes to the rescue.
By applying a relative or proportional adjustment for each contract you will be able to maintain the historical price ratios and rid remove the gaps. This is equivalent to stock split adjustments in the Equities world. This is also done backwards – starting from the most recent contract going back, one by one, to the first contract in the series.
Again, please check the last post for more details on the issues discussed above.

Forward-Adjusted contracts

Both point-based adjustment and ratio-based adjustment are usually worked out backwards – this allows to keep reference of the real/actual traded price in the current contract. However the same logic can be applied forward (start from the first contract and adjust your way up to the last contract). This will result in different contract values to back-adjusted contracts (but ratios would stay identical in case of proportional adjustments).

Nth Nearest contracts

This is a slight variation to all cases discussed above. Instead of always considering the nearest contract (ie front-month), you can decide to look at the nth nearest contract instead. For example, in Copper, the 3rd nearest contract is currently March 2010: HGH10. Once the current front-month contract expires (October 2009: HGV09), you would roll into the next available 3rd nearest contract which is May 2010: HGK10.
This process should insulate you from possible more volatile price changes in the front-month contract around expiration time.

Gann contract

Gann contracts use a specific contract month and roll over only to the same contract in the next year. A March Gann contract, for example, consists of the nearest March contract for each day.

Perpetual Contracts™

This is an algorithm trademarked by CSI. To avoid the “gap” problem around rollover time, CSI have decided to smooth the transition phase out. The shift from one contract to the next one(s) spreads over time. The algorithm uses a time-weighted average of different expiration date contracts to gradually shift from the front-month to the further month as expiration date approaches. You can find more details in the UA manual.

Other parameters: when to roll-over?

You can roll-over right at expiration time. However this does not reflect actual trading activity, where physical roll-over (ie money flowing from one contract to the next) takes place earlier.
You can decide to always roll-over n days before expiration but you can also decide to track actual market activity to trigger the roll-over. You would observe Open interest or Volume and roll-over when one of them shifts from one contract to another. This is a much more realistic approach.

Here is a screenshot of the settings I have used in Unfair Advantage to extract and concatenate the data, it is proportionally forward-adjusted with a roll-over triggered by a 2 consecutive shifts in Open Interest, but less than 5 days prior to expiration:

Unfair Advantage settings to generate a Proportionally Forward-Adjusted Contract

To be continued…

There are many more options which will all be addressed in a further post detailing how to use the Unfair Advantage API to extract data directly from your own code (Exciting!).

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)

Well, I have just purchased End Of Day historical futures data from Commodity Systems Inc. and this is how they call their Software/Database!

Admittedly it is supposed to be a very good data feed that Google and Yahoo use for their respective Finance portal. They are also recommended by back-testing software companies (Trading Blox, Amibroker, etc.). Futures magazine also ran a piece listing them as the “cleanest” data around (back in 99).

Futures article - CSI historical data

Why CSI?

I chose them because they appear to have credibility, good quality data, compatibility with Trading Blox and TradersStudio and I was interested in their Futures contracts splicing/concatenation algorithm choice. I am not really interested in all the “bells and whistles” that come with the software (charting, analysis, etc.) but they might become useful. My primary interest is to feed the data to a back-testing software.

Getting up and running: in a breeze

After downloading the install (500MB!) the software installs itself very quickly and starts downloading the latest database updates. You are supposed to run updates every so often to keep the database current. Once it is done you can start playing with the data within Unfair Advantage. The docs (both online and printed) are pretty good too.

Wide choice of markets and contract types

There are about 900 futures in the database and you can pretty much splice them as you want – to recreate different TimeSeries. I’ll explore futures contract concatenation in a later post but to give you a taster of the Unfair Advantage “menu” you can “order” a CBT Corn (Floor + Electronic trading) in a March Gann Series format as a “starter” followed by CME Lean Hogs (Floor Trading only) proportionally back-adjusted with roll-overs triggered by Open Interest shifts as your “main”; and finally a Perpetual Contract of Robusta Coffee on EuroNext. Of course these are just examples (I’ll explain all these type of contracts in a follow-up post). Full functionality can be found here, all Futures coverage can be found there and finally pricing options.

Export functionality

For each of these contracts that you add to the Unfair Advantage portfolio, you can define several export formats that will generate specific files ready to be imported to your favourite back-testing tool. There is also an API that allows you to access the data to allow for some automation – this will definitely prove handy further down the line.

The User Interface – screenshots

Here is what the software looks like:

Main window – the portfolio on the left-hand side contains all contracts you would want to follow while the right-hand side displays the current contract as a chart.

Contract selection screen: a myriad of options to splice the data.

In closing

I feel pretty satisfied with what I got and eager to dig more into the software/data and play with it. The application seems intuitive enough and if CSI are as good as their reputation implies, this should increase back-test results quality. Promising…

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)