Systematic Trading research and development, with a flavour of Trend Following
Au.Tra.Sy blog – Automated trading System header image 2

Amibroker V. TradersStudio: Speed comparison Fight

November 10th, 2009 · 4 Comments · Backtest, Software

It might not capture the imagination as much as the recent Haye v. Valuev WBA World Heavyweight Championship fight (it probably might for some of you… ;-) but I decided to organise my own “fight”: AmiBroker V. TradersStudio!
And similarly to the boxing, speed was of the essence – with one platform completely out-performing the other one. Let’s find out which one…

In the last few posts I have been exploring the e-ratio and wrote the code in TradersStudio. I also tried to implement it in AmiBroker AFL language mostly to check how it performed speed-wise – as I found TradersStudio quite slow.

While testing AmiBorker, it actually felt so fast that I decided to perform a more formal speed comparison test.

Conditions of the test

I exported the CSI data via the API for a proportionally back-adjusted Corn contract going back to 1949. The data was imported in both AmiBroker and TradersStudio via their ASCII Import.

The system tested is a simple 20-day Donchian Channel Breakout (Buy-only) and the ATR used to normalise the MAE/MFE is 20-day also.
The optimiser in both systems were used to generate the e-ratio for 50 51 different trade durations (from 10 to 50 days).

One of the condition to validate the results of the test was that the trades generated by both systems are similar (to double-check I did not make a coding mistake resulting in simpler/faster process for one of the platforms).

The computer I was running the test on is a quad-core CPU (2.4 GHz) with 3.25 GB of addressable RAM. Only one platform was running at the time it was tested.

Results are in!

And they look astoninglishly good for AmiBroker:

  • The TradersStudio test was run first and took 4 mins 15 sec to complete and produce the custom report (which needs to be manipulated in Excel to calculate the e-ratio). Furthermore, it appeared that every incremental run in the optimisation process took slightly longer than the previous one.
  • The AmiBroker test ran in (…drumrolls…) 10 sec! And the e-ratio was readily available from the results grid:
    AmiBroker optimisation results showing the e-ratio. Click to expand

    AmiBroker optimisation results showing the e-ratio. Click to expand

Both apps maxed out their allocated CPU (i.e. overall CPU usage of 25% = a quarter of the quad-core CPU available).

I then decided to run the AmiBroker code over 5 markets to get a feel of how long it would take. I added Crude Oil (going back to 1985), Cotton (1968), Gold (1975) and Yen (1972).
AmiBroker ran the same code + optimisation in 2 mins 20 sec, twice as fast as one market in TradersStudio!

Applying a simple proportional calculation to derive the time it would take in TradersStudio for the same dataset would give us a completion time of 1 hour (59 min 30 sec exactly…). Since the performance seems to degrade over the course of the optimisation in TradersStudio (i.e. the first back-test is quicker than the last one of the optimisation run) we could assume that it would actually take longer, which is what I experienced when I was testing on similar data in the last few weeks.

So here you have it: AmiBroker is 25 times faster than TradersStudio and it gives you the results in a much more friendly format.

Trade Reconciliation

As mentioned earlier, to ensure that the test was valid, the set of trades generated by both systems should be compared. I ran one back-test of the system over the Corn data for a single optimisation step (e-ratio for 20 days duration) in both systems.

I could find that 296 trades out of 340 presented a very good match (some matches had one day difference on the Entry date and rounding differences on the prices). But overall the trade comparison was good enough to give assurance that the same systems and results were being tested.

I’ll keep investigating for myself where these errors are coming from (and post a further code update if warranted) but we can assume that the differences are trivial with regards to performance comparison.

Did I mention that AmiBroker is 3 times as cheap with a much wider following (many more code samples, better docs, more forums, etc.)?…

Related Posts with Thumbnails

Tags: ·······

4 Comments so far ↓

  • Matt Busigin

    I’m surprised it AmiBroker didn’t use more than 1 CPU. My understanding is that in the past number of versions, the optimiser now runs backtests in parallel.

    I most typically run AmiBroker through a virtualised XP container from within my Mac laptop, so performance will never really be peak. It’s also important to note that if you’re doing big optimisations that take a lot of time — especially if you’re optimising parameters which calculate PositionScore! — you are likely over-fitting, and less likely to have something which works in the future. YMMV.

  • Jez

    Hey Matt,
    Nice to “see” you here too… I havent had time to check your blog more in detail but will do so soon..

    This goes slightly beyond my level of competency but I understand that the limitations come from XP which wwill not allow a process to use more than one of the 4 CPU (ie 25% of total). So the app might run hundreds of parallel threads but will never get allocated moe than 25% CPU. Anyway speed is so great that this is not a problem.

    Completely agree with the over-fitting part. But unfortunately the e-ratio does require many optimisation runs (despiet not being a proper optimisation procedure)

  • Getulio Jr

    May I ask where you get this historical data to backtest?

  • Jez Liberty

    Getulio, I use CSI data: (you can get 10% off with code LIBERTY).

Leave a Comment