Systematic Trading research and development, with a flavour of Trend Following
Au.Tra.Sy blog – Automated trading System header image 2

CSI Raw Contract Data Extractor

June 10th, 2010 · 19 Comments · Code, Data

CSI Data Extractor: Utility tool to extract CSI data in plain text files

CSI Data Extractor: Utility tool to extract CSI data in plain text files

I have mentioned before that I use CSI as my main provider for Futures data. Overall, I judge CSI’s service as good quality – however one drawback is the closed aspect of the data access. And I decided to write a utility to make it more “open”.

To access the historical data, you have to go through their software (Unfair Advantage), which extracts and decrypts it from their proprietary binary format. You can always “build” historical files to disk, but these are usually pre-processed with a special algorithm (ie. back-adjusted, etc.).

What more, if you decide to stop your subscription to CSI’s data services, you will not be able to access any data any more, even historical data downloaded under the subscription contract.

For some of the reasons above, it might be interesting to be able to extract/store CSI’s raw data in plain text files. This would allow for:

  • Browsing the data outside of the CSI’s Unfair Advantage software
  • Applying back-adjustment or other manipulation algorithms not supported by CSI’s UA (such as calculating the yield curve)
  • Keep the historical data independently of your CSI subscription

Note: what I consider CSI raw data is the actual daily information for each contract traded on any day. For example, in the Corn market, there are currently 15 different contracts trading (with different maturities), each with their own Open, High, Low, Close, Volume and Open Interest.

CSI API to the rescue

Now, you could easily go to Unfair Advantage and pick every single historical contract that you want to extract… but this would be a fairly tedious job

Luckily for us, the API allows for programmatic access to CSI’s data to automate the process of extracting all contracts that we want – so I decided to write a little utility to do just that. And now my data is really mine!

Below is a zip file containing the files to copy to “install” the utility:

CSI-DataExtractor.zip

Note that you need to have Unfair Advantage installed on the machine running this exe (which does not come with any guarantee by the way… but I have been using it fine so far).

The operation is very simple: just enter CSI’s number for the market you want to extract and choose which path the files need to be extracted to.

Hope you’re finding this tool useful…

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)

Related Posts with Thumbnails

Tags: ··

19 Comments so far ↓

  • Motomoto

    brilliant – a great labour saving tool. Thankyou.

  • Jez

    No worries! Glad you’re finding it useful.

    I enhanced it a bit to enable multiple instrument extracts in one “go”:

    http://www.automated-trading-system.com/wp-content/uploads/2010/06/CSI-DataExtractor1.11.zip

    Just enter all the CSI instrument numbers in the (larger) textbox separated by a comma.

  • Al Fonce

    Hey Jez, is there a way to run your – very usefull – application from Excel/VBA? This would be great to systematize the extracting process that way!
    Cheers! :)

    Al

  • Jez

    Thanks Al Fonce.
    Sorry but that version cannot be run other than interactively through the GUI.
    I might update it to enable running it from the command line (which could also be triggered from VBA) – stay tuned.
    Other than that, if you cannot wait, I recommend looking at AutoHotKey, which is a nifty tool, and which can automate the running of interactive apps – I use it for several other things and really recommend it generally…

  • Cachetier

    Can i use the CSI data in R or am i limited to using the historical database only in connection with UA software ?

  • Jez Liberty

    Cachetier – I do not think that you can access the CSI DB other than through the UA software OR the API, which at least allows you to extract CSI data into plain text files that you can access from R.
    I am not sure if you could link from R to the API or if it is even possible though (my R knowlegde is not that good but if you could call some EXE that uses the CSI API, that could work)

  • Cachetier

    I simply want to extract the data from UA to a plain text file, so i can burn the data to disk or do whatever i want with the data without any limitations. I need the World Futures historical data(10 years).

  • Jez Liberty

    Well – the tool mentioned above should help you do this. If you are after continuous contracts, you should be able to extract the data to file from either the UA software (comes with the data subscription) or the API.
    Once it is extracted to file/disk, there is no limitations any more.

  • Cachetier

    Interesting ,because two days ago i asked CSI if it is possible to extract the data from their software and burn it to disk and they answered NO. Maybe you can send me the data(if you have it) cause i don’t won’t to install Windows,install UA and extract the data.

  • Jez Liberty

    mmmh – that is confusing..
    If you think about it I am sure that most CSI subscribers need to feed the data to another system (for example Trading Blox users like I use CSI data for back-testing) and there is definitely the possibility to extract a portfolio of instruments to files.

    Moreover, if the data is accessible via API, any code could just write it to a file (this is exactly what the tool above does…)

    Unfortunately, the license does not allow for redistribution of the data so I am afraid I could not send it to you.

  • Cachetier

    No license can restrict the use of historical data generated on the world exchanges,also a number of other companies offer the same historical databases so a restriction to use the data would be a contradiction in terms. No problem anyway if somebody needs my historical tick data i will give it to everybody who wants it.
    Information must be free !

  • Anthony Marsland

    Hi Jez,
    Really looking forward to using your creation. Is it working with the latest version of UA? I’m getting “unhandled exception…” when I try to use the utility.
    Many thanks in advance,
    Anthony

  • Anthony Marsland

    Hey Jez,
    I’ve answered my own query…
    CSI charge a $500 one off fee to access the API. It is only available to yearly or monthly subscriptions. I have a full history subscription so API unavailable…
    Regards,
    Anthony

  • Jez Liberty

    Anthony – thanks for the update – I was not aware that CSI had a charge for history-only API access.
    Cheers,
    Jez

  • Anthony Marsland

    Hi Jez,
    Update on the API…
    If you’re a BLOX customer CSI will turn on the API for gratis.
    Regards,
    Anthony

  • Jez Liberty

    Thanks for the update Anthony – I think I saw that discussed on the TB forum. Back when I first used the CSI API, there was nothing to activate

  • Anthony Marsland

    Is your extractor still working with latest CSI UA?
    I’ve downloaded it into Windows 8 and getting:

    ERROR
    Retrieving the COM class factory for component CLISD…

    at CSI Extract.Form1.button

  • Jez Liberty

    Anthony,

    I have not upgraded to the latest CSI UA unfortunately Anthony.
    Also, the code was developed and runs on Windows XP. It might be that it needs some converting to run on Windows 8 (but I’m a slooow upgrader and I still use XP!)

    If I ever need to use this more extensively and finally upgrade to a 21st century Windows OS ;-) I’l’ post an update of the code.
    Cheers,
    Jez

Leave a Comment