Au.Tra.Sy blog - Automated trading System » API

CSI Raw Contract Data Extractor

Jez Liberty — Thu, 10 Jun 2010 09:34:38 +0000

CSI Data Extractor: Utility tool to extract CSI data in plain text files

I have mentioned before that I use CSI as my main provider for Futures data. Overall, I judge CSI’s service as good quality – however one drawback is the closed aspect of the data access. And I decided to write a utility to make it more “open”.

To access the historical data, you have to go through their software (Unfair Advantage), which extracts and decrypts it from their proprietary binary format. You can always “build” historical files to disk, but these are usually pre-processed with a special algorithm (ie. back-adjusted, etc.).

What more, if you decide to stop your subscription to CSI’s data services, you will not be able to access any data any more, even historical data downloaded under the subscription contract.

For some of the reasons above, it might be interesting to be able to extract/store CSI’s raw data in plain text files. This would allow for:

Browsing the data outside of the CSI’s Unfair Advantage software
Applying back-adjustment or other manipulation algorithms not supported by CSI’s UA (such as calculating the yield curve)
Keep the historical data independently of your CSI subscription

Note: what I consider CSI raw data is the actual daily information for each contract traded on any day. For example, in the Corn market, there are currently 15 different contracts trading (with different maturities), each with their own Open, High, Low, Close, Volume and Open Interest.

CSI API to the rescue

Now, you could easily go to Unfair Advantage and pick every single historical contract that you want to extract… but this would be a fairly tedious job

Luckily for us, the API allows for programmatic access to CSI’s data to automate the process of extracting all contracts that we want – so I decided to write a little utility to do just that. And now my data is really mine!

Below is a zip file containing the files to copy to “install” the utility:

CSI-DataExtractor.zip

Note that you need to have Unfair Advantage installed on the machine running this exe (which does not come with any guarantee by the way… but I have been using it fine so far).

The operation is very simple: just enter CSI’s number for the market you want to extract and choose which path the files need to be extracted to.

Hope you’re finding this tool useful…

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)

How-To: CSI Data into TradersStudio

Jez Liberty — Tue, 12 Jan 2010 13:34:09 +0000

Dear readers, I hope you enjoyed the holiday break. I wish you a happy New Year – may it bring you progress and success in your automated trading endeavours!

I wanted to start this year with an idea I left off last year, when investigating robustness: Using the median as a more robust tool than the ubiquitous mean (or average). This will be covered in the next post via a TradersStudio simulation results. To do that, I had to re-import some data from CSI Unfair Advantage to TradersStudio. I thought a “how-to” post on this would be useful – if only as an aide-memoire for next time (note-taking and documentation are important after all…)

Extract CSI Data

The first step is to extract the data from the CSI database into text files
(by the way, CSI have a 20% discount running until the end of January – just check their homepage).
This can be done with code using the Unfair Advantage API.

Pick all the instruments to be extracted and create a “contracts.param” file containing the CSI number for those instruments in the first column (comma-separated file with other columns being ignored). The CSI numbers can be found on their website (like here for the CME Futures) or in their Unfair Advantage software UI. Running the EXE code will generate one file for each instrument (proportionally back-adjusted contract) in the following format:

19900403,199006,0,240,110,918,503,2,0,1.59,1.655,1.58,1.635,1.635,1.635,1.635
19900404,199006,0,84,127,656,721,3,0,1.6,1.65,1.599,1.62,1.62,1.62,1.62
19900405,199006,0,57,112,554,885,4,0,1.615,1.635,1.61,1.615,1.615,1.615,1.615
19900406,199006,0,62,136,436,1092,5,0,1.615,1.635,1.61,1.625,1.625,1.625,1.625

Load in TradersStudio

The second step is to load the files in TradersStudio. Copy all generated text files in a dedicated data directory (you might have to rename them as TradersStudio allowed filename length is fairly short, i.e. 10-15 characters).
Create the relevant mapping file (txtinfo.txt) in the data directory. The mapping file should read:

delimit=,
skip=0
dt=1
ti=0
op=10
hi=11
lo=12
cl=13
vol=6
oi=7
dtformat=CCYYMMDD
tiformat=

In TradersStudio, menu Data Manager/Add Data Links, point to the relevant data directory with the following options:

If all goes well, this will load the data in the system (errors I encountered were due to file names being too long).

Data Universe: Contract meta-data

The third step is to populate the meta-data (margin, min tick move, etc.) for each instrument in TradersStudio. This is done via the menu Data Manager/Data Universe.

Once done, the instruments can be used just by adding them to any session.

As mentioned earlier the next post will be a test comparing moving average and moving median indicators.

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)

Unfair Advantage API Code (C#): Extract Futures Continuous data

Jez Liberty — Sun, 04 Oct 2009 15:37:09 +0000

As mentioned in the previous post on retrieving Back-Adjusted Contracts using the RetrieveBackAdjustedContract2 function of the Unfair Advantage API, I have coded up this very simple project to read a list of Futures underlying instruments, retrieve a proportionally back-adjusted contract for each of the instruments and oputput it to a file.

Getting started with the API

Fire up your favourite IDE/language that supports COM Interoperability (I am using Visual C# 2008 Express Edition), start Console Application project and add a reference to the Unfair Advantage API. For this you just need to “Add reference” and browse to the Unfair Advantage installation folder and select the main EXE (uad.exe). This gives access to the CSI UA API.

Add UA into your project and "Bob's your Uncle!"

Simple extraction code

The rest is fairly simple as long as you understand the RetrieveBackAdjustedContract2 parameters.
The code below takes 2 parameters: an input file name containing the list of markets to extract and a folder name to extract each continuous contract to. It sets specific UA session parameters and calls the RetrieveBackAdjustedContract2 with specific variables to obtain the concatenation desired. Each continuous contract is output to a folder in a .txt file.
The file containing the list of markets is of the following format (CSI number, Symbol, Market name):
…
8,CT,Cotton
…

The code

Please find the code there (very nicely formatted thanks to this great tool by manoli). You can also download the Program.cs file:

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)

Unfair Advantage API: Retrieve Back-Adjusted Contracts function

Jez Liberty — Fri, 02 Oct 2009 10:41:06 +0000

Let’s have fun with the CSI UA API!

CSI main API document (word doc) is accessible publicly online from their document page along with other docs including the full 300 page manual (great for getting a feel of the app).

Pivotal API function

This post will focus on is the API function to retrieve Back-Adjusted Contracts from the Unfair Advantage database.
As the last post on Continuous Contract options showed, there are multiple choices available and these all translate to parameters to the API function: RetrieveBackAdjustedContract2. This is what I consider to be the meat of the API: it queries the proprietary UA database and applies the concatenation algorithms to produce a continuous contract.

27 Parameters!

Yes… that function’s behaviour is driven by the value of 27 properties or variables. There are actually 14 “Session” properties and another 13 variables passed to the function. The aim of these parameters is to reproduce exactly all the options covered in the previous post… and then some.

My own doc

Because I found the CSI doc not ideal (you have to keep flipping back and forth for a description of all 27 parameters) I decided to re-write my own “cheat-sheet” documentation for the RetrieveBackAdjustedContract2 API function.
The rest of this post is that document – formatted into HTML by M$ Word (so please excuse any formatting mishaps). The document contains 2 parts: the UA session properties and the function variables. Hope this helps.
You can also download the document in the RTF format.

Stay tuned for the next episode, where I will share a sample code using the function above to extract a list of Futures contracts. This should illustrate how to use all these parameters.

The document:

FUNCTION: RetrieveBackAdjustedContract2

–>

FUNCTION: RetrieveBackAdjustedContract2

PROPERTIES:

IncludeSaturdays:“True” if Saturday data is desired.

IncludeHolidays: If NOT TRUE, than all data records reflect actual trading. If TRUE, then weekdays on which there is no record are recorded with a day of week value of 8 and no price information.

OnHolidaysUsePreviousData: If 1, then the previous days data is copied with the Date and DayOfWeek changed.

OnHolidaysUseCloseOnly: If 1, then the price values are set to the previous close and the volumes are set to zero.

ShowDecimalPoint: controls whether prices are listed as integers or as strings representing the number (see 4.1.2.6 for details – ConversionFactor.)

ConversionFactor: Most CSI data formats present price values as integers, according to the CSI format found in the “WSJ Price/CSI Price” column of the CSI Commodity Data Fact Sheet. Please note the CSI price and the WSJ price, and when dealing with the CSI price values, consult the conversion factor column on the right hand side of each page. A conversion factor of -1, for example, reflects that the final digit is in eighths. A CSI value of 3746 for wheat would be equivalent to 374 and 6/8, or 374-3/4..)

CONVERSION FACTOR        CONVERSION ACTION REQUIRED

        +6             Move decimal point 6 places left

        +5             Move decimal point 5 places left

        +4             Move decimal point 4 places left

        +3             Move decimal point 3 places left

        +2             Move decimal point 2 places left

        +1             Move decimal point 1 place left

         0             Leave number as it is

        -1             Terminal digit is in eighths

        -2             Terminal 2 digits are in 16ths

        -3             Terminal 2 digits are in 32nds

        -4             Terminal 2 digits are in 64ths

        -5             Terminal 3 digits are in 128ths

        -6             Terminal 3 digits are in 256ths

        -7             Terminal 3 digits are in 32nds & 1/2 32nds.

DetrendMethod: Indicates how and whether to detrend data being retrieved. Detendingtakes two prices, the first close (of the earliest date) in the series and a pivot price. The pivot price may either be the close of the last day for the second-to-the-last contract or the close of the second-to-the-last day on file. All days within the range have an amount added so that afterward the first close in the series matches the pivot price. Here are the valid values.

Name	Value
Detrend By Second Contract	-1
Not Detrended	0
Detrend by Second Day	+1

FillInCashPrices: Determines whether to retrieve Cash Prices. The “cash” price may be an actual cash price, it may be lagged cash, which would only have been known the day after, or it may simply be the nearest future contract.

ApplyCommodityAdjustments: “True” to have adjustments made for unit of contract specification changes.

UseAlternateBackAdjuster: If 1, then the Alternate Back Adjuster is used for all series, (default is the non-portfolio chart option value in UA). Notice that the standard back adjuster is fundamentally backward looking while the alternate back adjuster is fundamentally forward-looking which is why the Proportional accumulation is different.

RaiseNegBackAdjustSeries: controls whether UA automatically adds the smallest number of thousands of points to bring the back adjuster series into the non-negative domain

NumDaysConfirmation: selects how many consecutive volume and/or open interest roll signals are required before the algorithm rolls. For the Standard Back Adjuster which views the market backward in time from the most recent to the most distant, this argument is ignored and has an effective value of 1. For the Nth Nearest Future and the Alternate Back Adjuster, a value of at least 1 is required.

CloseOutOfRangeAdjustmentMethod: When a series settles outside of the daily range some software required the data to be “adjusted”. The following are allowed:

typedef enum
{

LeaveAlone_CloseOutOfRangeAdjustmentMethod=0,

CompromiseHighLow_CloseOutOfRangeAdjustmentMethod=1,

CompromiseClose_CloseOutOfRangeAdjustmentMethod=2,

} CloseOutOfRangeAdjustmentMethodType;

RoundToTick: controls whether UA rounds values to the nearest tick

UseTradeableTick: if RoundToTick is TRUE, determines whether the settlement price should be rounded as per the exchange or the same way as the other prices.

AggregateVolAs, AggregateOiAs, and AggregateCashAs: When making weekly, monthly, quarterly, or annual Documents, how are these fields to be treated:

AggregateAsLargest,

AggregateAsSmallest,

AggregateAsAverage,

AggregateAsLast,

AggregateAsTotal

AggregateDateAsLastRecord: TRUE if weekly, monthly, quarterly, and annual Documents should end on the date of the last data available. For a weekly file generated after the close on Tuesday, the records would be Friday, Friday, …, Friday, Tuesday. If not TRUE, all records are on the same day of the week.

AggregateDayOfWeekAs: If AggregateDateAsLastRecord is not TRUE, then what day of the week should the aggregate data be assigned to:

“Sunday”,

“Monday”,

“Tuesday”,

“Wednesday”,

“Thursday”,

“Friday”,

“Saturday”

PARAMETERS

MarketNumber: integer assigned by CSI to identify a contract underlying.

rollLogicType: Identifies the logic to be used in deciding when to ignore a near contract:

RollOnOpenInterest (-1)

RollOnVolume (-2)

RollOnOpenInterestAndVolume (-4)

RollOnOpenInterestOrVolume (-3)

RollOnDayRelativeToStartOfMonth (-5)

RollOnDayRelativeToEndOfMonth (-6)

rollWhen: time delay between when the day’s prices are published and when the day’s volume and open interest are published is handled. (There is almost always a one-day delay.)

Name	Value	Nth Nearest Rolls	Alt Back Adjust	Std Back Adjust
RollWhen_Anticipated	2	Invalid	Invalid	28^th
RollWhen_Reported	0	28^th	28^th	29^th
RollWhen_Known	1	29^th	29^th	30^th
RollWhen_Lagged2	3	30^th	30^th	Invalid

DayOfMonthToRoll and MonthsPriorToRoll: used when the rollLogicType is a date roll to determine when the roll takes place.

accumulationMethod: Determines whether to adjust backward or forward.

Name	Valid	Meaning
Proportional	-1	(Deprecated)A multiplicative ratio factor is applied rather than an additive factor. Because of the backward viewing perspective, the standard back adjuster brings the data to the current day contract level whereas because of the forward viewing perspective, the alternate back adjuster brings the data to the earliest contract level. Prices cannot fall into negative territory in the past with this option.
Back	0	Adds adjustments to bring historical data to the current day contract level. This option can result in negative price readings into the past.
Forward	+1	Subtracts adjustments to bring historical data to the earliest contract level. Because of the effects of inflation, by using this option, prices are less likely to move negative into the future.

rollDeltaType: identifies which contract values are used to measure the carrying charges for which the data is being adjusted.

Name	Value	Effect
OpenToOpen	-1	Simulates rolling by an on-the-open spread order.
CloseToClose	0	Simulates rolling by an on-the-close spread order.
CloseToOpen	+1	Simulates an exit order on the close and an enter order on the next open.
CloseToOpenOldGap	+2	Mathematically equivalent to OpenToOpen.
CloseToOpenNewGap	+3	Mathematically equivalent to CloseToClose, except that the Alternate back adjuster requires one fewer days of existence for an explicit CloseToClose.

ValidMonths: Selects which delivery months to consider. The ValidMonths argument must be a string of 12 characters, one character for each month. Each character determines whether contracts deliverable in that month are included. A contract is included if there is a ‘V’ (for valid) in that position. A value of ‘I’ for invalid causes exclusion.

GenerateForward: If the UseAlternateBackAdjuster is zero then GenerateFoward will have no effect. TRUE or 1 to begin with the earliest data and step day-by-day to the current day. FLASE or 0 to begin with the current contract and prepend earlier data.

RollAtLeastNDaysBeforeExpiration: definition a bit shaky…

causes a roll to happen if necessary to avoid getting too close to expiration.

If you are not generating forward (zero), then the standard back adjuster is applied. If the RollAtLeastNDaysBeforeExpiration is greater than zero, then the rollLogicType is overridden so that rolls happen based only on expiration. (Like a roll relative to month end.).

StartDate and EndDate: These two long integers allow the user to limit the retrieval to a date range. Use the values -1, -1 or 0 ,0 if you want the full history available. Can significantly improve performance, but for forward generated Nth Nearest Future and Back-Adjusted contracts can result in different contract choices.

ProportionalAdjustment: If the ProportionalAdjustment parameter is non-zero, then the Backward or Forward adjustments are applied by a multiplicative ratio factor rather than an additive factor. When the standard back adjuster is applied, only Backward adjustments can be applied proportionally.

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)