At the same time as Josh Ulrich was making some very good points on why to use R, over on his FOSS Trading blog, I was coming to the same realisation that it is a very neat, useful tool indeed. In this post, I’ll present the code I used for the last post on running a Walk-Forward test of the LSPM. Hopefully this can serve as an introduction to R to readers who do not use it yet.

Getting into R

The LSPM java app distributed by Ralph Vince during his seminar did not seem very robust – and in any case, I could not manage to get it to generate the Joint-Probability Table (JPT) for 6 market-systems over a 20-year history of monthly returns. It would simply hang after 4 of them.

I then decided to write my own JPT tool. Did not take me too long, an hour or two and about 200 lines of C# code. It did the job ok and I could run it in an automated/batch mode. But I had forgotten that Josh had posted some R code to do exactly that (Rats! A couple of hours wasted…). On top of that the code was about 30 lines long. Impressed by this, I decided that the study on the last post was a good opportunity to try and learn R and I attempted to automate the test using R only.

What the Code Does

At a high level, the code reads several files each containing the history of monthly returns for a market-system. The format of the files is as follows:

19900201,-0.5
19900301,-1.16
19900401,2.24
19900501,3.3

First column = date, second column = percentage return

This is a Walk-Forward test, so the optimization runs on a sliding window of past returns, and results are applied to the walk-forward window of returns. The cycle is repeated as many times as necessary.

For each cycle, the return data from the optimization window is extracted from the full set of returns, passed to the LSPM optimization, and calculated leveraged returns (using f amounts from the optimization) are stored for the corresponding walk-forward window. The full results of the test consists of a concatenation of all leveraged returns in each walk-forward window.

The code below is my first forray in R and its robustness could be “enhanced” (it contains several assumptions such as file formats, number of market-systems) but it should be fairly easy to adapt to run different scenarios or make it more generic.
R is quite different from the programming languages I am used to, so some of the syntax might seem a bit hacked – but it does the job (I appreciate any corrections/pointers on better practices from experienced R readers…).

Hopefully, it is a useful example for readers wanting to get started with R. Readers might want to check my initial post on using R to run the LSPM optimization as this code expands on some aspects of it.

The Code

The first part of the code sets the walk-forward parameters, here: 10 years for the optimization window and one year for the walk-forward window:

# Set Walk-Forward parameters (number of periods)
optim<-120 #10 years = 120 monthly returns
wf<-12 #1 year = 12 monthly returns

Open the market-system files:

# Open files
setwd("C:/MyPath")
a <- read.csv("TMA-20-50-200.csv", header = FALSE, as.is = TRUE, sep = ",", dec=".");
b <- read.csv("MA-50-200.csv", header = FALSE, as.is = TRUE, sep = ",", dec=".")
c <- read.csv("BBO-20.csv", header = FALSE, as.is = TRUE, sep = ",", dec=".")
d <- read.csv("Donchian-200.csv", header = FALSE, as.is = TRUE, sep = ",", dec=".")
e <- read.csv("Donchian-20.csv", header = FALSE, as.is = TRUE, sep = ",", dec=".")
f <- read.csv("BBO-50.csv", header = FALSE, as.is = TRUE, sep = ",", dec=".")

Calculate the number of Walk-Forward cycles:

# Calculate number of WF cycles
numCycles = ceiling((nrow(a)-optim)/wf)

Below is the code mentioned earlier, which can be found on Josh Ulrich’s blog – used to create the LSPM JPT:

# Define JPT function
jointProbTable <- function(x, n=3, FUN=median, ...) {
 
  # Load LSPM
  if(!require(LSPM,quietly=TRUE)) stop(warnings())
 
  # Function to bin data
  quantize <- function(x, n, FUN=median, ...) {
    if(is.character(FUN)) FUN <- get(FUN)
    bins <- cut(x, n, labels=FALSE)
    res <- sapply(1:NROW(x), function(i) FUN(x[bins==bins[i]], ...))
  }
 
  # Allow for different values of 'n' for each system in 'x'
  if(NROW(n)==1) {
    n <- rep(n,NCOL(x))
  } else
  if(NROW(n)!=NCOL(x)) stop("invalid 'n'")
 
  # Bin data in 'x'
  qd <- sapply(1:NCOL(x), function(i) quantize(x[,i],n=n[i],FUN=FUN,...))
 
  # Aggregate probabilities
  probs <- rep(1/NROW(x),NROW(x))
  res <- aggregate(probs, by=lapply(1:NCOL(qd), function(i) qd[,i]), sum)
 
  # Clean up output, return lsp object
  colnames(res) <- colnames(x)
  res <- lsp(res[,1:NCOL(x)],res[,NCOL(res)])
  return(res)
}

Loop through each Walk-Forward cycle:

for (i in 1:numCycles) {

Extract the relevant set of past returns for the optimization:

	# Define cycle boundaries
	start<-1+(i*wf)
	end<-optim+(i*wf)
 
	# Get returns for optimization cycle and create the JPT
	rtn <- cbind(a$V2[start:end],b$V2[start:end],c$V2[start:end],d$V2[start:end],e$V2[start:end],f$V2[start:end])

Create the JPT and run the actual LSPM optimization (note that the case above is the “simple” optimization, only taking optimal geometric growth rate into account – no consideration for probability of drawdown. This was mostly to ensure the run did not take hours and hours):

	jpt <- jointProbTable(rtn,n=c(20,20,20,20,20,20))
	outcomes<-jpt[[1]]
	probs<-jpt[[2]]
	port<-lsp(outcomes,probs)
 
	# DEoptim parameters (see ?DEoptim)
	np=60       # 10 * number of mktsys
	imax=1000       #maximum number of iterations
	crossover=0.6       #probability of crossover
 
	NR <- NROW(port$f)
	DEctrl <- list(NP=np,itermax=imax, CR=crossover, trace=TRUE)
 
	# Optimize
	res <- optimalf(port,control=DEctrl)

Retrieve the leverage amounts from the optimization as well as the set of returns from the Walk-Forward window. Apply the leverage amounts to these returns and add the data to the full set of leveraged returns:

	# Save leverage amounts
	lev<-c(100/(-jpt$maxLoss[1]/res$f[1]),  100/(-jpt$maxLoss[2]/res$f[2]),  100/(-jpt$maxLoss[3]/res$f[3]),  100/(-jpt$maxLoss[4]/res$f[4]),  100/(-jpt$maxLoss[5]/res$f[5]),  100/(-jpt$maxLoss[6]/res$f[6]))
	levmat<-c(rep(1,wf)) %o% lev #so that we can multiply with the wfrtn
 
	# Get the returns for the next Walk-Forward period
	wfrtn <- cbind(a$V2[(end+1):(end+wf)],b$V2[(end+1):(end+wf)],c$V2[(end+1):(end+wf)],d$V2[(end+1):(end+wf)],e$V2[(end+1):(end+wf)],f$V2[(end+1):(end+wf)])
	wflevrtn <- wfrtn*levmat #apply leverage to the returns
 
	if (i==0) fullrtns<-cbind(t(wflevrtn)) else fullrtns<-cbind(fullrtns,t(wflevrtn))
}

After the loop is finished, print the full set of leveraged returns

t(fullrtns)

In Closing

Despite my very limited experience, I can really see some of the potential in using R for any sort of data manipulation.

If you want to get started in R and test this code, the following links will be useful:

R homepage, which contains link to a download page. The installation is very simple and quick. Make sure you also check the docs as they are good.

The R LSPM package maintained by Josh Ulrich, the real meat of the code above – credits to him for making this!

The R code included in the post above

And a batch file to run the above code from a command prompt.

Finally, if you want to re-run the same test, here are the return files that were used in the code above:
Return Data Files (zipped)

A few months ago, I got quite interested when Trading Blox announced that they introduced a new walk-forward functionality in their latest version. I just got round to upgrading, and giving that walk-forward testing a go. Amongst other things, some of the chart features have been improved – as can be seen in the eye candy above.

How it works

You can check this previous post for an explanation of how walk-forward works, from a general point of view. Robert Pardo (usually credited with the invention of walk-forward) has also written his bookfew books on the subject.

This new feature is a combination of small enhancements to the core application (ie. silent tests, dynamic date and starting equity settings, etc.) and a semi-custom script implementing the walk-forward testing procedure. It feels more like a (good) hack leveraging the core stepping functionality, rather than a functionality built from the ground up, but it does the job rather well (I looked at the underlying code and its ingenuity is actually pretty cool).

A minor issue is that you have to work out the number of optimization/out-of-sample cycles. If you get that wrong, the test will not cover the desired time interval and it makes the process less automated. Apart from that, the settings are pretty straight-forward, you can choose lengths of optimization and out-of-sample phases as well as other options:

In the settings above, there are 8 cycles, each having a 5-year optimization phase and 1-year out-of-sample phase

Objective Function

At the end of the optimization phase of each cycle, the system picks the best set of parameters to be used in the out-of-sample phase. However there are many ways to determine the best system. Do you use raw CAGR, MAR, Sharpe ratio, etc.?

There is a need to define the objective function (also called bliss function, as explained here, which will be used to determine the best system performance.

Trading Blox calculates standard statistics (MAR, CAGR, Drawdown, etc.) which can all be used as the objective function, but also allows the implementation of custom statistics calculations. These custom statistics can then be used as the objective function.

Output

This is where you realise the functionality is not fully integrated into the app. The system shows the individual results of each out-of-sample runs:

The stats calculated relate to each out-of-sample run (1 year in my example), whereas ideally these should all be spliced together, to form one system with its own stats. Same applies to equity curves, which are charted year by year (as individual charts). It is still possible to access the Daily Equity log file to do this yourself but it would be nice to have it better automated. The fact that it is a script and not compiled core code might make it possible to customize it to implement this automation.

Another log file also allows you to investigate the results of each optimization run and check their performance:

Starting optimization run ,225,2005-01-03, to ,2010-01-01, with starting equity ,166470686.069881530,
 
Run ,225, of ,256, Stepped Parameters: ,
Name,Step Value,Goodness Measure of ,64.497285767,
Run ( Index ),8.000000000,
Optimization Run,1.000000000,
Entry Breakout (days),20.000000000,
Exit Breakout (days),10.000000000,
Starting optimization run ,226,2005-01-03, to ,2010-01-01, with starting equity ,166470686.069881530,
 
 
----------------------------------------,
Best goodness of ,64.497285767, was on run ,1.000000000,
----------------------------------------,
 
Starting out of sample test ,241,2010-01-01, to ,2010-09-07, with starting equity ,166470686.069881530,

Test Results

For my first “test ride” of the walk-forward functionality, I used the good ol’ Donchian system.

The system parameters being tested are the Entry and Exit breakout lengths ranging from 20 to 50 days for the entry and 10 to 25 days for the exit.

The parameters for the walk-forward procedure are a 5-year optimization phase and 1-year out-of-sample phase with the plain CAGR stat being used as the objective/bliss/goodness function. The dates of the test go from 1998 to 2010 (but bear in mind that the first out-of-sample result is from 2003, ie. 1998 to 2002 are used as first optimization period).

The results (year by year) are the ones shown in the section above.

For an interesting comparison, I ran a standard stepped test using the same Donchian system and parameter values. Check below how the walk-forward system performance stacks up against the range of outputs from the stepped test:

All test results with the walk-forward result in yellow

In hindsight, it would have been better to pick other systems that performed better, but the advantage of the walk-forward process is that it adapts itself to pick a system.

In Closing

The walk-forward functionality is definitely not a fully-fledged feature of Trading Blox (yet?) – however it does allow for some automation of the process. The fact that it is “open” is probably a good thing as it can be tinkered with and improved easily. A nice addition to the product…

nebulous intricacies

In the last post we looked at the Turtle Trading system and saw that its performance went from outstanding for a long period of time to flat for 20 years. This opens a can of worms:

Does Trend Following work, is it dead, do markets change, does trend following rules need to adapt to these changes?

Let’s look at the different points of view.

The EMH Crowd

The EMH crowd does not believe in anything else than the Random Walk and by definition discards any profit-generating mechanical strategy.

As much as the Efficient Market Hypothesis (EMH) is a cornerstone of most modern financial theory, it has proven to be wrong partly because of some of its assumptions (not all actors in the market are rational, market prices are not fully random and normally distributed, etc.).

One great book that discredits the EMH approach and their descendant theories (CAPM, etc.) is by Mandelbrot: The (mis)behavior of the markets. It lays the arguments against the EMH in an approachable (ie not too much maths) way.

Trend Following is dead/does not work

Curtis Faith declared that “every few years trend following traders experience a period of losses and inevitably some expert will announce the end of trend following.”

Mike Covel also has a similar quote in his Trend Following book: “every 5 years some famous trader blows up and everyone declares trend following to be dead. Then 5 years later some famous trader blows up and everyone declares trend following to be dead, etc. ”

One of the main proponents of the argument against Trend Following is infamous trader Vic Niederhoffer (who “blew up” twice). He has been highly vocal about it, declaring Trend Following as one of the top Stock Market con.
Here is another link from his website to read more about it.

Arguments like this, despite the empirical evidence against it – in the form of Trend Following Wizards success, can be taken as a motivation for healthy skepticism and push you to strengthen your statistical research.

Are markets changing? (and must Trend Following change too?)

The self-professed “Trend Following poster boy” (a.k.a. Michael Covel) authoratively declares that this is a specious argument.

Occasionally, someone trying to promote something or start a debate will argue that trend following rules must always change due to changing market conditions. This is nonsense. It is a specious argument.

This is at best ambiguous. Covel likes to cite Bill Dunn who:

proffered that his basic system rules have not changed since 1974

Now, that is seducing: it seems to sell you the idea that you can develop a system, implement it and trade it for life. However this is not strictly true. As mentioned in this interview:

Dunn annually adjusts the parameters of trading signals and each markets weighting. In February – just as the grains were about to take off – he dumped the entire grain sector. But Dunn has no regrets.

Dunn is also known to have collaborated with Robert Pardo, a strong proponent of Walk-Forward testing (see below: a constant system adjustment).

To clarify: although Trend Following principles will never change, the rules/parameters of a Trend Following system might need to be adjusted to changing market conditions.

How can a Trend Following strategy adapt to the ever-changing markets?

In an Active Trader article, Anthony Garner attempts to discuss:

Do markets change? Is it necessary to undertake continued research and development and adapt a trend-following system to maintain its profitability over the years?

The article is only available for the magazine subscribers, but the result of the equity curve can be found on the Trading Blox forum. By “tuning up” the Turtle system, Garner manages to obtain interesting stats (MAR=2.26, CAGR=35.28%). The main change to the system is the use of a longer-term timeframe.

Practically

I know I would not be happy trading the original Turtle System in the last 20 years and get a 0% return. If you started trading this system “back then”, when and how would you think it is time to switch to a revised system?

Let’s look at options:

1. Walk-Forward

Walk-Forward testing‘s principle is to keep running (in simulation) a “pool” of systems using different rules/parameters. At regular interval, you evaluate what systems are best (performance, robustness, etc.) and trade those until the next re-evaluation. The potential risk with this approach is that you might end up like a dog chasing your tail.

However, this approach would have you switched from the original Turtle system to the new one a while ago.

2. Alternative Walk-Forward

Markets exhibit some degree of inefficiency – and Trend Following is a strategy designed to profit from these inefficiencies. I am still refining my theoritical understanding and explanation of it, but I believe Trend Following’s performance is mostly the result of fat-tailed distributions (distribution kurtosis) and possibly autocorrelation.

If one can associate the evolution of these characteristics to the performance of Trend Following systems it might be possible to adapt the system rules/parameters to the values and evolution of the price distribution characteristics. This is a topic I’d like to investigate further.

3. Mixing Systems

Finally, and this seems to be a strategy adopted by many professionals: mix different systems and different timeframes. Here the rationale is that we cannot predict what systems are going to under/over perform and mixing several ones together will smooth out the equity curve.

The way it works is fairly simple. It is a combination of multiple cycles of “in-sample optimisation” with “out-of-sample verification”.

Background on optimisation and out-of-sample testing

The reason for performing out-of sample verification tests is to check whether the in-sample data optimisation resulted in curve-fitting (over-optimisation) or in a robust system parameters selection.

If the parameters derived from optimisation perform much worse in the out-of-sample verification test, it most likely means that the parameter values were (over-) optimised for the specific in-sample dataset (curve-fitted). If the system performs similarly, it should mean that the system parameters are robust and validate the approach taken for optimisation.

Walk-Forward Process: how it works

Walk-Forward testing is an on-going and dynamic process to determine whether parameters optimisation just curve fits the price and noise or produces statistically valid out-of-sample results. Here is how it works:

Let’s say we have 10 years of data from 1999 to 2009. Optimisation period is three years (in-sample data) and Verification period is one year (out-of-sample data). To begin, you start by optimising your system using only the first three years of data – in this example, 1999-2001. When the system is optimised, record the optimal parameter values and use them in the test with new data (out-of-sample) starting with 2002.

Walk Forward from 1999 to 2009

Slide the three-year window of data forward (2000-2002) and perform the same process. Once you have processed all the data available, you can collate the performance of all out-of-sample tests and compare those to in-sample optimisation runs. If the comparison shows that the system is sufficiently robust to be traded live, you simply continue the walk-forward process in real time by re-optimising every year.

In Closing

Walk-Forward is an adaptive process which re-optimises the system on a continuous basis to adapt its parameters to the most recent market conditions.

The premise of performing several optimisation/verification steps over time is that the recent past is a better environment for selecting system parameters than the distant past. This is an assumption you need to consider when choosing whether to use Walk-Forward testing or not but this is a useful tool in your Systems development arsenal.

As discussed earlier changing system parameters based on recent market conditions could result in a system chasing its tail. The next post on this topic will be an actual comparison of a basic system’s performance when optimised in a “standard way” and when optimised using “Walk-Forward”.

Thanks to marfis75 for Mr. DJ Chameleon

There are 2 approaches to systems parameters:

• Decide on a set of parameters for the strategy and stick to it
• Keep changing the system parameters based on the latest market conditions (chameleon approach)

The chameleon method will inherently be lagging and, as a result, might not give you very good results. Think about being stuck in a traffic jam on the highway: you always think the queue next to you is faster – but it seems that every time you change lanes, the one you just left picks up speed… and you end up being stuck even longer!

In terms of automated trading, you might end up with a system that chases its tail (instead of chasing the fat-tail). Of course there are “different shades of grey” and some parts of the system can be self-adapting (volatility-adjusted indicators for example) and allow for a middle ground approach. Consider the Turtle Rules as an example: they used to have static parameters (i.e. 20 and 55 days breakouts) but used ATR as a central part of their strategy to adapt to different volatility levels.

The concept of chasing its tail can also be illustrated by this story about Bill Dunn:

The World Monetary program currently trades 13 markets. Back testing 10 to 12 years (each year has equal weighting), Dunn annually adjusts the parameters of trading signals and each market’s weighting. In February – just as the grains were about to take off – he dumped the entire grain sector.

Performance will depend on how fast the chameleon can adapt and be responsive while keeping a robust approach. We will try to find out…
In a next post, we will cover Walk-Forward testing (which uses a sliding look-back period for back-test and applies the optimal parameters to the latest trading period). It can be associated with a chameleon approach. We will also run a comparison test between the 2 approaches and see how the chameleon performs!