Au.Tra.Sy blog - Automated trading System » Continuous contract

Unfair Advantage API: Retrieve Back-Adjusted Contracts function

Jez Liberty — Fri, 02 Oct 2009 10:41:06 +0000

Let’s have fun with the CSI UA API!

CSI main API document (word doc) is accessible publicly online from their document page along with other docs including the full 300 page manual (great for getting a feel of the app).

Pivotal API function

This post will focus on is the API function to retrieve Back-Adjusted Contracts from the Unfair Advantage database.
As the last post on Continuous Contract options showed, there are multiple choices available and these all translate to parameters to the API function: RetrieveBackAdjustedContract2. This is what I consider to be the meat of the API: it queries the proprietary UA database and applies the concatenation algorithms to produce a continuous contract.

27 Parameters!

Yes… that function’s behaviour is driven by the value of 27 properties or variables. There are actually 14 “Session” properties and another 13 variables passed to the function. The aim of these parameters is to reproduce exactly all the options covered in the previous post… and then some.

My own doc

Because I found the CSI doc not ideal (you have to keep flipping back and forth for a description of all 27 parameters) I decided to re-write my own “cheat-sheet” documentation for the RetrieveBackAdjustedContract2 API function.
The rest of this post is that document – formatted into HTML by M$ Word (so please excuse any formatting mishaps). The document contains 2 parts: the UA session properties and the function variables. Hope this helps.
You can also download the document in the RTF format.

Stay tuned for the next episode, where I will share a sample code using the function above to extract a list of Futures contracts. This should illustrate how to use all these parameters.

The document:

FUNCTION: RetrieveBackAdjustedContract2

–>

FUNCTION: RetrieveBackAdjustedContract2

PROPERTIES:

IncludeSaturdays:“True” if Saturday data is desired.

IncludeHolidays: If NOT TRUE, than all data records reflect actual trading. If TRUE, then weekdays on which there is no record are recorded with a day of week value of 8 and no price information.

OnHolidaysUsePreviousData: If 1, then the previous days data is copied with the Date and DayOfWeek changed.

OnHolidaysUseCloseOnly: If 1, then the price values are set to the previous close and the volumes are set to zero.

ShowDecimalPoint: controls whether prices are listed as integers or as strings representing the number (see 4.1.2.6 for details – ConversionFactor.)

ConversionFactor: Most CSI data formats present price values as integers, according to the CSI format found in the “WSJ Price/CSI Price” column of the CSI Commodity Data Fact Sheet. Please note the CSI price and the WSJ price, and when dealing with the CSI price values, consult the conversion factor column on the right hand side of each page. A conversion factor of -1, for example, reflects that the final digit is in eighths. A CSI value of 3746 for wheat would be equivalent to 374 and 6/8, or 374-3/4..)

CONVERSION FACTOR        CONVERSION ACTION REQUIRED

        +6             Move decimal point 6 places left

        +5             Move decimal point 5 places left

        +4             Move decimal point 4 places left

        +3             Move decimal point 3 places left

        +2             Move decimal point 2 places left

        +1             Move decimal point 1 place left

         0             Leave number as it is

        -1             Terminal digit is in eighths

        -2             Terminal 2 digits are in 16ths

        -3             Terminal 2 digits are in 32nds

        -4             Terminal 2 digits are in 64ths

        -5             Terminal 3 digits are in 128ths

        -6             Terminal 3 digits are in 256ths

        -7             Terminal 3 digits are in 32nds & 1/2 32nds.

DetrendMethod: Indicates how and whether to detrend data being retrieved. Detendingtakes two prices, the first close (of the earliest date) in the series and a pivot price. The pivot price may either be the close of the last day for the second-to-the-last contract or the close of the second-to-the-last day on file. All days within the range have an amount added so that afterward the first close in the series matches the pivot price. Here are the valid values.

Name	Value
Detrend By Second Contract	-1
Not Detrended	0
Detrend by Second Day	+1

FillInCashPrices: Determines whether to retrieve Cash Prices. The “cash” price may be an actual cash price, it may be lagged cash, which would only have been known the day after, or it may simply be the nearest future contract.

ApplyCommodityAdjustments: “True” to have adjustments made for unit of contract specification changes.

UseAlternateBackAdjuster: If 1, then the Alternate Back Adjuster is used for all series, (default is the non-portfolio chart option value in UA). Notice that the standard back adjuster is fundamentally backward looking while the alternate back adjuster is fundamentally forward-looking which is why the Proportional accumulation is different.

RaiseNegBackAdjustSeries: controls whether UA automatically adds the smallest number of thousands of points to bring the back adjuster series into the non-negative domain

NumDaysConfirmation: selects how many consecutive volume and/or open interest roll signals are required before the algorithm rolls. For the Standard Back Adjuster which views the market backward in time from the most recent to the most distant, this argument is ignored and has an effective value of 1. For the Nth Nearest Future and the Alternate Back Adjuster, a value of at least 1 is required.

CloseOutOfRangeAdjustmentMethod: When a series settles outside of the daily range some software required the data to be “adjusted”. The following are allowed:

typedef enum
{

LeaveAlone_CloseOutOfRangeAdjustmentMethod=0,

CompromiseHighLow_CloseOutOfRangeAdjustmentMethod=1,

CompromiseClose_CloseOutOfRangeAdjustmentMethod=2,

} CloseOutOfRangeAdjustmentMethodType;

RoundToTick: controls whether UA rounds values to the nearest tick

UseTradeableTick: if RoundToTick is TRUE, determines whether the settlement price should be rounded as per the exchange or the same way as the other prices.

AggregateVolAs, AggregateOiAs, and AggregateCashAs: When making weekly, monthly, quarterly, or annual Documents, how are these fields to be treated:

AggregateAsLargest,

AggregateAsSmallest,

AggregateAsAverage,

AggregateAsLast,

AggregateAsTotal

AggregateDateAsLastRecord: TRUE if weekly, monthly, quarterly, and annual Documents should end on the date of the last data available. For a weekly file generated after the close on Tuesday, the records would be Friday, Friday, …, Friday, Tuesday. If not TRUE, all records are on the same day of the week.

AggregateDayOfWeekAs: If AggregateDateAsLastRecord is not TRUE, then what day of the week should the aggregate data be assigned to:

“Sunday”,

“Monday”,

“Tuesday”,

“Wednesday”,

“Thursday”,

“Friday”,

“Saturday”

PARAMETERS

MarketNumber: integer assigned by CSI to identify a contract underlying.

rollLogicType: Identifies the logic to be used in deciding when to ignore a near contract:

RollOnOpenInterest (-1)

RollOnVolume (-2)

RollOnOpenInterestAndVolume (-4)

RollOnOpenInterestOrVolume (-3)

RollOnDayRelativeToStartOfMonth (-5)

RollOnDayRelativeToEndOfMonth (-6)

rollWhen: time delay between when the day’s prices are published and when the day’s volume and open interest are published is handled. (There is almost always a one-day delay.)

Name	Value	Nth Nearest Rolls	Alt Back Adjust	Std Back Adjust
RollWhen_Anticipated	2	Invalid	Invalid	28^th
RollWhen_Reported	0	28^th	28^th	29^th
RollWhen_Known	1	29^th	29^th	30^th
RollWhen_Lagged2	3	30^th	30^th	Invalid

DayOfMonthToRoll and MonthsPriorToRoll: used when the rollLogicType is a date roll to determine when the roll takes place.

accumulationMethod: Determines whether to adjust backward or forward.

Name	Valid	Meaning
Proportional	-1	(Deprecated)A multiplicative ratio factor is applied rather than an additive factor. Because of the backward viewing perspective, the standard back adjuster brings the data to the current day contract level whereas because of the forward viewing perspective, the alternate back adjuster brings the data to the earliest contract level. Prices cannot fall into negative territory in the past with this option.
Back	0	Adds adjustments to bring historical data to the current day contract level. This option can result in negative price readings into the past.
Forward	+1	Subtracts adjustments to bring historical data to the earliest contract level. Because of the effects of inflation, by using this option, prices are less likely to move negative into the future.

rollDeltaType: identifies which contract values are used to measure the carrying charges for which the data is being adjusted.

Name	Value	Effect
OpenToOpen	-1	Simulates rolling by an on-the-open spread order.
CloseToClose	0	Simulates rolling by an on-the-close spread order.
CloseToOpen	+1	Simulates an exit order on the close and an enter order on the next open.
CloseToOpenOldGap	+2	Mathematically equivalent to OpenToOpen.
CloseToOpenNewGap	+3	Mathematically equivalent to CloseToClose, except that the Alternate back adjuster requires one fewer days of existence for an explicit CloseToClose.

ValidMonths: Selects which delivery months to consider. The ValidMonths argument must be a string of 12 characters, one character for each month. Each character determines whether contracts deliverable in that month are included. A contract is included if there is a ‘V’ (for valid) in that position. A value of ‘I’ for invalid causes exclusion.

GenerateForward: If the UseAlternateBackAdjuster is zero then GenerateFoward will have no effect. TRUE or 1 to begin with the earliest data and step day-by-day to the current day. FLASE or 0 to begin with the current contract and prepend earlier data.

RollAtLeastNDaysBeforeExpiration: definition a bit shaky…

causes a roll to happen if necessary to avoid getting too close to expiration.

If you are not generating forward (zero), then the standard back adjuster is applied. If the RollAtLeastNDaysBeforeExpiration is greater than zero, then the rollLogicType is overridden so that rolls happen based only on expiration. (Like a roll relative to month end.).

StartDate and EndDate: These two long integers allow the user to limit the retrieval to a date range. Use the values -1, -1 or 0 ,0 if you want the full history available. Can significantly improve performance, but for forward generated Nth Nearest Future and Back-Adjusted contracts can result in different contract choices.

ProportionalAdjustment: If the ProportionalAdjustment parameter is non-zero, then the Backward or Forward adjustments are applied by a multiplicative ratio factor rather than an additive factor. When the standard back adjuster is applied, only Backward adjustments can be applied proportionally.

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)

Continuous Contract options

Jez Liberty — Sun, 27 Sep 2009 19:26:19 +0000

As mentioned previously in What everybody ought to know about continuous Futures contracts, there are multiple ways to concatenate Futures contracts. Unfair Advantage offers a rich choice of concatenation algorithms – I will use UA as the base for this post.
Please also refer to the Unfair Advantage online manual.

Continuous contracts allow you to chart Futures contract prices in one long time-series (here Copper ranging from 1965 to 2009)

Let’s look at the different possibilities:

Non-adjusted contracts

Take the front-month contract (nearest expiration date) and roll it over into the next one at expiration date. This generates a series of actual traded prices. But as we discussed in our previous post, it does have negative impacts when used for back-testing because of the potential big gaps at roll-over times.

Standard Back-Adjusted contracts

A solution to this “big gap” problem is to move each contract up or down by a certain amount (the gap value) so that the rollover junction is continuous. This is point-based adjustment and is usualy performed “backwards”. You would start from the current contract, move the previous contract by a set amount, then subsequently move each of the anterior contracts to eliminate each gap you come across until the first contract of the timeseries.

Proportionally Back-Adjusted contracts

But as we saw in the last post, this point-based adjustment is not ideal either, as it loses the relative price differences and can even make past prices become negative. This is where ratio-based adjustment comes to the rescue.
By applying a relative or proportional adjustment for each contract you will be able to maintain the historical price ratios and rid remove the gaps. This is equivalent to stock split adjustments in the Equities world. This is also done backwards – starting from the most recent contract going back, one by one, to the first contract in the series.
Again, please check the last post for more details on the issues discussed above.

Forward-Adjusted contracts

Both point-based adjustment and ratio-based adjustment are usually worked out backwards – this allows to keep reference of the real/actual traded price in the current contract. However the same logic can be applied forward (start from the first contract and adjust your way up to the last contract). This will result in different contract values to back-adjusted contracts (but ratios would stay identical in case of proportional adjustments).

Nth Nearest contracts

This is a slight variation to all cases discussed above. Instead of always considering the nearest contract (ie front-month), you can decide to look at the nth nearest contract instead. For example, in Copper, the 3rd nearest contract is currently March 2010: HGH10. Once the current front-month contract expires (October 2009: HGV09), you would roll into the next available 3rd nearest contract which is May 2010: HGK10.
This process should insulate you from possible more volatile price changes in the front-month contract around expiration time.

Gann contract

Gann contracts use a specific contract month and roll over only to the same contract in the next year. A March Gann contract, for example, consists of the nearest March contract for each day.

Perpetual Contracts™

This is an algorithm trademarked by CSI. To avoid the “gap” problem around rollover time, CSI have decided to smooth the transition phase out. The shift from one contract to the next one(s) spreads over time. The algorithm uses a time-weighted average of different expiration date contracts to gradually shift from the front-month to the further month as expiration date approaches. You can find more details in the UA manual.

Other parameters: when to roll-over?

You can roll-over right at expiration time. However this does not reflect actual trading activity, where physical roll-over (ie money flowing from one contract to the next) takes place earlier.
You can decide to always roll-over n days before expiration but you can also decide to track actual market activity to trigger the roll-over. You would observe Open interest or Volume and roll-over when one of them shifts from one contract to another. This is a much more realistic approach.

Here is a screenshot of the settings I have used in Unfair Advantage to extract and concatenate the data, it is proportionally forward-adjusted with a roll-over triggered by a 2 consecutive shifts in Open Interest, but less than 5 days prior to expiration:

Unfair Advantage settings to generate a Proportionally Forward-Adjusted Contract

To be continued…

There are many more options which will all be addressed in a further post detailing how to use the Unfair Advantage API to extract data directly from your own code (Exciting!).

Note: You can receive a 10% discount on CSI data subscriptions (use coupon code LIBERTY)