A crude "API" for MetaTrader 4

In response to Ludosm's request to publish this utility, here it is.  Using this API, any other software can drive trade functions in MT4 by generating simple one line CSV files in a similar format to the original MQL4 trade commands.

So where you would write the following in an EA ....
OrderSend(symbol, cmd, volume, price, slippage, stoploss, takeprofit, comment, magic, expiration, arrow_color);

.... the CSV file looks like this
OrderSend, symbol, cmd, volume, price, slippage, stoploss, takeprofit, comment, magic, expiration, arrow_color

A price of Bid or Ask is interpreted correctly.

An important difference is that open orders are located using their Comment, meaning that comments should have a numerical sequence.

So OrderClose(ticket); becomes OrderClose,Comment

There are additional comments at the top of the script.  Hope it's useful.

1500 and counting; what's coming up

Very gratifying to see that I've passed the 1500 mark with hits to this blog - thanks everyone for the interest. So that's a very modest average of 10 hits a day, although it started very slow and has risen to about 50 per day recently. As an aside, it's fascinating to be a microscopic part of the Google machine. The blog has so far had 11 ad click-throughs to generate $18.11 revenue in five months!! Keep on clicking, guys - I'll give up my day job soon :)

In the meantime, I have lots of balls in the air with trading and coding ideas (happy as a pig in sh*t, as they say)

1. First priority - finish off an article which expands on my auto-generated documentation for MQL5 post in this blog. The Metaquotes guys have showed an interest in it and it's likely to be published on the official MQL5 site. The last thing to finish off is a comment processor which trawls through the entire MQL5 folder and subfolders and slightly modifies the comments so that they can be read in by Doxygen and appear in the comiled help documentation.
2. Finish off the MQL5 Virtual Order Manager so people can try it and comment. I've written the VOM itself, but need to polish up the EA I use to test it so it can be debugged more fully. This EA is a series of buttons in a chart window which allow real and virtual orders to be opened and closed. I'll probably release the whole lot then, but also need to follow up quickly with an indicator which reports open virtual orders in a table.
3. The Neural Network EA has been grinding along. I hit a difficult snag with the retrieval of history - other people have also complained that they can't retrieve the full history with Close[xxxxx]. My testing seems to indicate a limit of 1000 bars in the past (or is it 1024?) or only 10 days of 15 minute bars, although some others talk about a gradually increasing value of Bars but they can't work out what triggers it. The EA and associated neural network dll works well functionally, but the neural network can't be trained on a sufficiently long dataset to prove the concept. I can see two ways forward: either write a history reader which reads history directly from the MT4 *.hst files. Or, write the whole damn thing in MQL5, which I have found doesn't have the limitation. The latter will take a lot more time, but I've always had a plan to write a learning neural network in MQL5 in preparation for the trading championships in 2010, so I'm leaning that way. I wish Metaquotes would let us know when the strategy tester will become available.
4. Allow me some (OK, a lot of) misty speculation: if an MQL5 neural network EA proves to be profitable (or any MQL5 EA), then it would be quite easy to write a MetaTrader 5 to MetaTrader 4 messaging utility to enable any MQL5 EA on demo to trade live now using MetaTrader 4. Already sitting in my MT4 code archive somewhere is a crude MT4 API EA which accepts trading commands via *.csv files.

Writing a Virtual Order Manager to enable hedging in MetaTrader 5

Since I wrote about a Virtual Order Manager in August, the MetaTrader 5 public beta commenced, and it has become clear that arguably the biggest change in the transition from MetaTrader 4 to MetaTrader 5 is the management of open trades as positions. At any one time there can be one position only open for each currency pair, and the size of this position adjusts up and down each time orders are processed by the broker. Apart from anything else, this aligns with the FIFO rules recently introduced in the US which effectively outlaw hedging.

To work in this position-centric environment, each Expert Advisor written in MQL5 needs additional programming to record which trades it has open, because it cannot find this out in a simple way from the broker. The most startling example of this would be when two EAs running against the same pair, say a scalper and a trend-follower, issue orders in opposite directions. In MT4, the result would be a long and short order with zero margin used. In MT5, no position would be open at all.

An example of the programming challenge is in the following function. Most people would use something similar in their MT4 EAs to ensure that only one order is open at a time. A similar function would not work as intended in MT5.

int OpenOrders(int magic)
{
   int OrderCount = 0;
   for (int i = OpenOrdersTotal()-1 ; i >= 0 ; i--)
   {
      OrderSelect(i,SELECT_BY_POS, MODE_TRADES);
      if (OrderMagicNumber() == magic
         && (OrderType() == OP_BUY || OrderType() == OP_SELL)
         && OrderSymbol() == Symbol())
      {
         OrderCount++;
      }
   }
   return(OrderCount);
}

So ... here comes a Virtual Order Manager
To work with this new order management environment I am writing a utility which I call a Virtual Order Manager, or VOM, which maintains a local record at the MetaTrader terminal of the orders issued by each EA.

Here are the attributes of the VOM:

Behaviour

• Real orders at the broker reflect total position for that pair, but the VOM also maintains a virtual set of orders at the MT5 terminal. As an example one EA could be virtual long EURUSD 0.2 lots and another EA could be virtual short EURUSD 0.3 lots, with the real position showing in the MT5 open orders list as short 0.1 lots.
• Virtual stoplosses, takeprofits, magic numbers and comments are supported. Virtual stoplosses and takeprofits can be modified
• Real broker stoplosses are maintained at a configurable distance away from the virtual stoplosses as disaster protection against PC or internet connection failures.
• Virtual entry prices, stoplosses and takeprofits can optionally be displayed as horizontal lines on the chart.
• [for eventual development] A virtual_statement.htm can be produced similar to the normal statement.htm which lists open orders and closed order history.

Implementation

• The VOM functionality is added with an #include in each EA.
• I am modelling the API for the VOM on MT4 trading functions. These are familiar, proven functions which will make migration from MT4 EAs easier. So instead of writing OrderSend(….), one writes VirtualOrderManager.OrderSend(…..). I have simplified some of the functions so they are not necessarily a plug-in replacement, but I may also implement the original full versions to make MT4 migration even easier.
• The list of open orders is maintained in a file for all VOM-enabled EAs to access. A table listing all open orders will be viewable using a OpenVirtualOrderViewer.mq5 indicator.

Challenges

Some of the more challenging parts of writing the VOM are listed here

• The VOM must be able to track when a pending order activates. I am using the OnTrade() event in MT5 for this purpose.
• Since multiple EAs read and write to the one file listing virtual open orders, I have to use a global variable as a file locking mechanism.
• The value of the disaster protection stoploss is not easy to determine because it is obtained from a number of virtual orders.

Release

So when will version 1.0 be released? I would hope within two weeks.

Active MQL5 sites

Just thought I'd put in another plug for Metatrader 5, which is the most active Metatrader 5 blog, hosted by miranon. The latest post Indicator Update for MT5 is up to his usual standard, and he has kindly pointed to some of my work on this blog as well.

*** update 20/11/2009 *** http://www.mql5.com/ is now active!

The Russian forum  is bound to be as active as it was in mql4 and is quite readable with Google translate.

That's about it - if you google MQL5 or Metatrader 5 or MT5 you'll find lots of other sites such as Forex-TSD but although there are some interesting discussions, there's not much original MQL5 code.  If anyone reading this has another site they'd like to highlight, feel free to post a comment.

Auto-generated documentation for MQL5 code

I submitted this blog entry to Metaquotes when I wrote it, and I'm pleased to announce that they have now published it on the MQL5.com site.
The article includes a very useful MQL5 source code help file which I intend to update from time to time when new code is released.

Here's the introduction:

Most Java coders will be familiar with the auto-generated documentation that can be created with JavaDocs. The idea is to add comments into the code in a semi-structured way that can then be extracted into an easy to navigate help file.

The C++ world also has a number of documentation auto-generators, with Microsoft's SandCastle and Doxygen being two leaders. I decided to see how well Doxygen could document MQL5, which is in essence a customised subset of C++. To me this is an important step in the maturity of MQL5, because the complexity of the language is easily capable of fostering some quite large class libraries.

The experiment worked very well and I believe the help documentation that Doxygen produces from MQL5 code will add a great deal of value.

NewsTrader

I was looking through my blog posts (yeah, I know, I'm about the only one who does) and I noticed that back a while I had promised a longer description of my NewsTrader.  V2.3 has been forward testing for quite a while now (since 4th September), and has made the princely sum of $82.  This was on a ridiculously small (for a news trader) lot size of 0.1, so I've just bumped it up to a more reasonable 1.0.  With a stoploss of 10 pips this gives a nominal risk of $100 per trade, although I have seen absolutely massive slippage on news in the 30 pip range.  Many traders really get upset by things like that, but with my IT background I've no real problem with it.  Big news events would stress the forex trading network in such a fundamental way that it's not surprising that serious overloads occur.

Anyway, NewsTrader V2_3 reads the news events from the weekly XML list kindly supplied by ForexFactory, and issues a very tight straddle just before each medium or high impact event.  What's a straddle?  It's a BuyStop / SellStop pair that is able to profit from a clean move in either direction.  If you look at the forward test, you'll see a number of indications of how this trader works:-

• Large numbers of deleted stop orders.  This is when the price didn't immediately react to the news at all.
• Stopped out orders at -10 pips.  This is when a spike in one direction preceeds the real direction of the move.
• Closed orders in loss.  I have a timestop which from memory closes the orders after about 2 minutes (yes these trades are short!)
• And, wonder of wonders, a number of profitable orders which hit the TP of 10 pips

NewsTrader V2_4
When I first wrote NewsTrader I was trading from my laptop in Australia.  I've made other comments about the ping times between Australia and the US: suffice to say the theory of News Trading is greatly undermined by network latency across the Pacific.  With the use of my virtual host in New York, I've now reinstated my original code which worked so badly in practice, which was a very tight client-side trailing stoploss.  I'm running 4 pips at the moment.  You wouldn't believe how long it takes for a MetaTrader 4 terminal in Australia to deliver tick updates and do something trivial like close an order.

So that's it: NewsTrader V2_4 is NewsTrader V2_3 with a trailing stoploss of 4 pips and a takeprofit of 30 pips (which will not normally be hit).  It remains to be seen how much better it will work on the faster network.  Forward tested here.

When I get a moment I might add auto screendumps of the news events as they occur.

My first MQL5 Expert Advisor

Update 12/1/2010: this forward test now replaced with a Virtual Order Manager equivalent coexisting with the Support_Resistance EA.  See statement here and live screenshot here.  All links below are still valid.

This EA is a Moving Average Cross system using the Fractal Adaptive Moving Average.  It is almost certainly not a profitable EA - I've just used a simple system like this as an exercise in learning MT5.  Give it to me to code in MT4 and I'd have it done in literally 15 minutes, but this has taken me on an off about a week to code.

I decided to write an EA that is triggered using an indicator to really understand how things work.  The FraMa Cross indicator is here, and includes arrows showing long and short entries.  The default MA periods are 6 and 12.

There are various ways to code an always-in-the-market reversing EA such as this in MT5.  I simply reverse the position by sending an order for double the lots in the opposite direction, which illustrates one of the  largest differences between MT4 and MT5.

Files needed for this EA to compile and run:
FraMa cross EA V1_2.mq5 - the Expert Advisor itself. Store in MQL5/Experts
FraMa cross.mq5 - the indicator.  Store in MQL5/Indicators
EnumToStr.mqh - contains lookup tables for a couple of enums.  Store in MQL5/Include
SimpleLog.mqh - logging utitlity that I used to debug.  Store in MQL5/Include

Here is the statement of a live forward test, which is running on a EURUSD H1 chart with FastMA 6 and SlowMA 12.

[update] and here's a screenshot updated every 10 minutes.  You may need to use refresh on your browser to see updates (the picture to the left is an example only and doesn't update)

[update 2] I have uploaded a new version of the EA. It suddenly stopped working properly because I had made an error by not initialising a double in OpenLots().  A recent MT5 update must have stopped automatically initialising doubles.

[update 3]  The EA hasn't been working for a while - must have been a new build which broke it for some reason.  So I've recompiled after re-writing some of the code, such as OpenLots() (which doesn't need to loop through all positions as I'd orinally written it), and introducing a more versatile function called AdjustPosition().  The code is updated as Frama Cross EA V1_2.mq5

My first MQL5 coding impressions

MQL5 is in beta test, and I've been playing around with it for a few hours to see what it's like.  It's great fun, and I can see the enormous potential for MetaTrader 5.

I know MetaTrader 4 backwards, and am also reasonably experienced in C++, but there is much in MQL5 that needs to be learnt.  The trading functions are completely different, and so is the coding of indicators.  It's not particularly useful to import MQL4 code into the MQL5 editor and attempt to convert it inline because the conversion needs to touch on the entire coding structure.  The search is on for the equivalents to anything that doesn't exist, like IsOptimization() or TimeToStr().  Other functions exist but need to be used differently, like StringConcatenate.

I guess one could write a whole lot of pseudo MT4 function calls such as this
bool IsTesting()
{
  return(MQL5_TESTING);
}

This may assist in converting MQL4 code, but it's probably simpler to re-write the simpler ones.

A startling change is that all inbuilt indicators return a handle, not a value. This handle is then used to copy results to an array. The reason for this is to make it easier and more efficient to code indicators of indicators.

Here's my first indicator.  The new concept in indicators is OnCalculate(), which is called for each new tick.  For some weird reason the arrays in OnCalculate are from left to right.  This indicator I wrote shows a fast and slow Fractal Moving Average (FRAMA) and places arrows at the open of each bar following an MA cross event.

I'm now working thorugh the complexities of which bar is which to use this indicator to control an Expert Advisor.

Here's a useful MQH include file which converts error codes and OrderSend results to strings.

The public beta for MetaTrader 5 has begun

You can download MetaTrader 5 here.  I recommend following the public beta thread directly on the mql4 forum.  At this stage, mql5.com is still inactive.

I have successfully downloaded MT5 and connected to the demo server.  Two points to assist:

• The name field in the Open Account dialogue must be two words, implying first name and surname.
• The compiled help file for MetaEditor is in Russian only at the moment.  fai on the mql4 forum has kindly translated it into English using Google Translate.  Open up this zip file and copy the contents as mql5.chm into .\MetaTrader 5\help.  You may need to be a member of the forum to do this.

MetaTrader 5 public beta testing

At last - the public beta for MetaTrader 5 is due to start on 12th October!

I'm seriously thinking of starting up an MQL4 to MQL5 conversion and EA coding service.  In the last 12 months I've written MT4 EAs for free for quite a number of people.  I was hoping that the exercise would be worthwhile because I would be gaining valuable ideas in exchange, but in reality I haven't seen much new apart from a promising idea related to fixed range bars.

This new version of MetaTrader will generate an avalanche of coding requirements, so perhaps it's time to go commercial.  I should research how other service providers structure their charges, but one way would be to provide a compiled EX5 for a small cost, and charge say $100-$500 for the source code depending on complexity.  I'm looking forward to creating a set of module/objects which can easily be snapped into projects.

MetaTrader 5 Beta review

The MetaTrader 5 blog has translated a review of the closed beta. It was originally written in russian here. Hopefully the public beta is just around the corner, with one comment on the oroginal russian site suggesting 1st October.

"Practically, everything that many developers wanted from MQL 5 and Metatrader 5 was implemented and realized at the highest level. The major change in MQL5 is the introduction of OOP (Object-oriented programming). Professional programmers have now more opportunities, but those who learned to write in MQL4, can use the MQL4 programming in MQL5 without the benefit of OOP."

Assessing risk

Assessing risk - the most important aspect of any financial investment, and critical for evaluating Forex trading systems.

I've often seen comments in Forex forums something like this: "I'm not looking for a gold mine -- just a system that makes 10 pips per day". So let's take this 'modest' requirement to its conclusion with some back-of-the-envelope calculations. Come on, admit it, you've done something similar -- I have. In order to buy 1 lot of EURUSD at 200:1 leverage I have to supply $100,000/200 = $500 of margin from my account. 10 pips = $100 profit @ 1 lot. Hey, wait a minute! [scribble, scribble], that means that I can make $100 * 5 days a week * 50 weeks = $25,000 per year or 5,000%, all from a $500 investment!!!! And if I increase the investment over time to 10 lots, I could … I can …….... eyes glaze over, images of palm trees on a tropical beach drift into view.

Past performance is not necessarily indicative of future results
The hard cold reality, of course, is risk, manifested as drawdown. Focusing only on the profitability of a system ignores this critical financial and psychological element, like agreeing to a major medical procedure without knowing how often it has been successful.

Historical, backtested risk is not difficult to calculate. Conversely, obtaining a useful estimate of future risk is (or should be) the single most challenging and elusive task that faces a trading system developer. Here are some ways in which I attempt to assess future risk

• The simplest assessment of risk is a reward:risk ratio calculated from (total profit)/(maximum drawdown). The MetaTrader 4 Strategy Tester reports individually on these, and it's a real pity that it cannot optimise on the combination.
• When people visually assess the smoothness of a backtesting equity curve, they are in effect trying to assess future risk. Calculating the smoothness of equity curves is again absent from the MetaTrader 4 Strategy Tester. That's why I wrote an Extended Strategy Reporter, which among other parameters calculates the Sharpe Ratio and modified Sharpe Ratio, both of which provide a numerical indication of smoothness.
• Once an optimised set of parameters has been found, it is important to verify that these are part of a "rounded hill" of generally low risk values. A set of parameters that has no neighbors is almost certain to be curve-fitted.

Breakout Neuro EA drawdown and work in progress

Hmmm - the Breakout Neuro EA is in a big drawdown at the moment. Taking the point that I first started forward testing on 7th August to now, it had a peak equity of about +1,600 pips with a drawdown of -300 pips, but then the drawdown started in earnest and is now -900 pips. That's partly bad news, but I'm hoping that this is normal behaviour - it is after all a fixed optimisation from only three months of data which is potentially starting to lose its relevance.

The work I am doing now is

• Was the initial gain a fluke? I think it's time to manually re-optimise the fixed neural network to see if it can again show a period of profitability
• Take the concept further to a self-optimising neural network that frequently goes through a learning cycle, say every day or perhaps driven by "failure", ie a series of losing trades
• Since the optimisation relies heavily on the fitness measurement from the validation equity curve, I'm expecting to have to spend time on getting that right. Is the profit/drawdown ratio good enough, or will it benefit from a more sophisticated form of equity curve measurement such as Sharpe ratio (profit/standard deviation)?
• The EA engine I used could also do with some tweaking, because it currently issues long and short trades blindly. Instead I will change it so that new long positions will first exit any existing short positions, and vice versa.

MetaTrader 5 is in closed beta testing

At last MetaTrader 5 has been released in closed beta testing on 9th September, see here. I don't have a copy unfortunately, but am looking forward to the public beta, which I am guessing will start sometime in October. They cancelled the 2009 trading championships so they could concentrate on MT5 development, and I think many traders like me will be setting themselves a goal of entering the 2010 championships. If I ever get something going with my neural network approach then I'll find it much easier to translate the C++ DLL that I am writing into MT5.

My guess is that the earliest that brokers will start changing over to offering live MT5 trading is late in 2010.

Forex signals services

There are a number of automatic signals services out there which present a smorgasboard of systems to chose from. For a while I was using FXDD Auto until I drove an account into the ground. This is no reflection on them, because their platform is quite good, but on my poor selection of systems, and I must take the time to check out their latest offerings.

FXDD Auto has a very elegant method of charging for their services - they simply add an extra pip onto every trade. This means that you can create a balanced portfolio of systems which can be easily adjusted at any time. You can start at 0.1 lot trades and, if you're lucky enough to watch your equity increase, there is automatic money management available to gradually increase trade (and fee) size. But, vouching from experience, you can also create an absolutely amazing equity curve from historical system results, only to watch it grow … then crash. You have to really question whether the builders of some of the systems have any genuine belief that they are profitable in the long term.

Another signals service that I am actively using right now is Global Forex Signals, with the PipsRain system. This system reassuringly doesn't boast massive pips per week, or a ridiculously high profitable trade ratio, but even so I am only trading it at 0.1 lot at the moment. The system costs $150 per month, so I'm hardly grossing any profit right now. Here's a forward test at 1 lot per trade (ahh, the peaceful luxury of demo accounts). The link updates only once a day to make it pointless to copy, so I hope I'm not violating any agreements. Come to think of it, they should be pleased that I'm promoting them.

GF Signals provides a free EA which reads their signal emails to automatically drive MT4. I've never tried it because I wrote a similar EA coupled with a gmail reader dll a long time ago before they released their own version. It worked fine, but I was quickly frustrated by the slippage from slow delivery of emails - it's quite normal to see at least 5 pips of entry and exit slippage simply because of the time delay. So I wrote a web page scraper instead which polls their signals web page and is far more reliable. The email reader approach would be fine as long as the average profit per trade is reasonably high.

So here's my approach to selecting a signals system:-

• At least 6 months of history, preferably 12 months
• Good profit/drawdown ratio
• Trading only 0.1 or 0.2 lots. This allows money management to be implemented earlier
• Maximum profitable trade ratio about 80% to avoid the luck factor
• Active use of stops in the 50 to 200 pip range
• Absolute minimum average profit per trade of 20 pips
• Use very low leverage

 I should emphasise that I am just a paying customer of FXDD Auto and Global Forex Signals, nothing more.

Volatility settles down

I've been looking at a few daily charts of volatility, as measured by a 14 period Average True Range.  Here's EURUSD as an example.

Interesting to see how much the volatility exploded with the GFC.  It hasn't quite settled down yet, but at least the trend is down, and it could meet the long term gradual rise quite soon.  It makes me realise how much the forex market behaviour has changed over the last year.  Will a system backtested as profitable through that timeframe still work well in the future?  Is it possible and desirable to adjust for volatility, eg by using ATR based stops instead of simple point multiples?

Note to self:

• Treat August 2008 to February 2009 as an atypical timeframe, and expect unusal system behaviour during that time.
• Try optimising from, say, January 2004 to July 2008, and compare with August 2008 to February 2009, and March 2009 to present.

Success so far with the Neural Network EA

A very pleasing result so far for my forward test of the Breakout Neuro EA V1_0.  I first wrote about the Neural Network approach I'm using in August

A recap on how this EA works:
The base EA is a simple 4 bar breakout on a 15 minute chart with 100 pip stoploss and takeprofit. Not surprisingly this is miles from being profitable in its own right. A 19-10-1 neural network provides an entry filter which is very effective (at the moment) at entering swing trades in the long direction.

The NN was optimised using an early stopping approach: backpropogate against a factset from 6 to 3 months ago, and select the NN configuration which delivers the best profit/drawdown on data in the last 3 months.

So how come I haven't broken out the champagne? The EA is making money isn't it? The problem is that the approach of optimisation and run until failure, or regular optimisation, is not backtestable. So, I thought, why not incorporate regular optimisation into the EA itself?

So I started to laboriously code the NN backpropagation algorithm into MT4 ... and gave up. It's not impossible, but translating what was a neat bit of OOP into a C-style environment, with no debugging facility, was painful. I also realised how many loops and arrays are involved and became concerned that MT4 would run too slowly.

So, for now, the opposite approach. I'm in the middle of writing a C++ dll which will spoon-feed everything to the EA, including the entries. All the EA will need to do is send out the latest closed bar info every 15 minutes and ask if there is an entry. Stay tuned for the results - in a week or two.

My favourite sites

Forex Peace Army
Even though I believe the guys that run this site have hearts of gold, this site is not entirely immune to abuse by scam artists, but it's still the most honest forex resource on everything from broker ratings, signals service reviews and now even EA forward tests

Pip Cop
An EA forward testing site that even runs some real live testing

Global Forex Signals
One of many signals sites, which I have been using for at least two years with a web page scraping EA which works much more reliably than an email reader. I can vouch for the authenticity of their records. Only a very few of the systems are reliable money-spinners long term, though. Global Trader and Pipsrain are excellent.

MQL4.com
Regularly delights me with excellent R&D articles on their forum

Forex Factory
An excellent news calender, made even more useful because it is available in XML format

Metatrader 4 Experts and Indicators
Probably the most active forex discussion groups around. Has a huge code archive.

Support Resistance EA V6_0

Update 12/1/2010: this forward test now replaced with an MT5 Virtual Order Manager equivalent coexisting with the FrAMA Cross EA. See statement here and live screenshot here. All links below are still valid.


This EA is my most successful to date, although I still wouldn't call it a really good EA.  Its strategy is the age-old concept that price will sometimes bounce off significant support and resistance points.  When I was designing it, I had an idea that I could scalp small profits off the reversals, but instead stumbled on a large takeprofit of 360 points.  The result is an EA for which the profitable trades figure is not much more than 30%.  The combination of 360 TP and 30% profitable makes it rather hard to trade, because it frequently gives back profits early in the trade.

It has pulled about 1500 pips off the market since July 2008 when I started trading it live.  Why not 2,194 pips, which is what the backtest now says it should have done?  Simply because I manually exited a lot of trades, which is an interesting example of trading psychology at work as a result of the difficulty in trading it.  Trading system purists could scoff at my weak-kneed approach, but remember that I was (and to a lesser extent, still am) not certain that it will be profitable in the long term.

I run it at the moment with no money management at a fixed 0.1 lot.  Here are three backtests from 1/1/2004 to 28/8/2009

• Flat 0.1 lot  Quite encouraging - maximum drawdown is 832 pips
• Money management with 2% x 80 pip risk - probably too agressive, but it's nice to dream.  A toned down money management at 1% looks a lot better
• Martingale - just for fun.  Even though this never trades above 0.9 lot, I've never found an EA that I would be comfortable running live in martingale mode

Note that the open prices only model is valid for this EA because it uses shift = 1 on all entry signals, and has such a large takeprofit.

Running MT4 on a virtual host

I recently decided to move all my live and test MT4 clients to a virtual host in the US. I wanted better reliability and I'm also hoping that faster network access to the broker server will be beneficial. I was surprised at how easy it was - I paid out $30 for the first month to Commercial Network Services and got my live account up and running remotely in about two hours. The only problem I have had so far is getting a gmail reader dll to work. It feels strange to be able to switch off my laptop during the week after having it running almost continuously for two years.

This virtual host is a Windows 2003 Server x64 instance in a data centre somewhere in New York, with excellent backup and network connectivity. It shows up as a Windows Terminal Server shortcut on my laptop. When I double-clicked on this for the first time, I was greeted with a normal Windows login screen, and even the Start Windows sound when I logged in. You can elect to have everything translated back to your local environment such as disk drives and printers. Aside from rather choppy screen updates, in full screen mode it looks and feels much like a local machine. When I run Windows Explorer on the remote host I can see my local drives, and transfer files (rather slowly) between environments.

Commercial Network Services has data centres in a number of cities, and I chose New York because it had the fastest ping times from FXDD. Whereas in Australia my laptop reports 400 millisecond ping times (yes, that's 0.4 second) from FXDD, this virtual host is 30 milliseconds away. Hopefully this will reduce the number of requotes and invalid price errors. My most demanding EA from a processing and network point of view is a news trader, and I'll be tweaking it in demo over the next few weeks to try to take advantage of the faster network times.

A discussion about the random walk hypothesis

Since I have "random walk" in the title of my blog, I suppose I should post some comments about it. The random walk hypothesis proposes that movements of any financial instrument such as forex or stock prices are completely random and therefore cannot be predicted. I'm in good company when I disagree, since it is quite clear to me that sentiments such as fear and greed, and the herd mentality of the market can and do cause non-random behaviour. My search is for the trading opportunities that these "non efficient" occurrences present.

But now, a trick question - do you agree with my analysis of the following charts?

Is my analysis OK? Perhaps you noted some triangle patterns that I missed, maybe a breakout?

The only problem is that they are selected from a random walk of 2,000 price ticks. That's right - using the Rand() function in Microsoft Excel, I created this spreadsheet today and just pressed F9 (recalculate) until I obtained a pattern I liked!  So what does this tell us? It means that the price movement we see can be close to random but can easily lure us into thinking that it is not.

After years of developing systems, I have come to the conclusion that there is a sweet spot somewhere between the following:-

1. Looking for large numbers of trades, which yields a statistically more significant backtest. More trades come from shorter timeframe charts and small profits per trade. It's very difficult to obtain enough trades to be comfortable with a backtest on a daily chart.
2. Looking for nuggets of sentiment amongst the noise. These seem to become more prominent in longer timeframes and larger price movements of at least 100 pips.

So this leads me to zero in on systems with TPs and SLs in the 50 to 500 pip range, on 15 minute to 1 hour chart timeframes. The smaller timeframes and stop values apply better to breakout and support/resistance type systems. The longer timeframes and stop values apply better to trend following systems.

I do think that scalping systems (ie very short timeframe, small profits) are possible, but I have never had any success in finding one. My theory is that some price behaviours and indicator patterns are so universally interpreted the same way that they become a self-fulfilling prophesy. A price might drop briefly, not because it is overbought, but because enough people think that it is overbought.

News management

Any observer of the Forex market quickly becomes acquainted with the peculiar phenomenon of news spikes. Some EAs are vulnerable to news because the spikes tend to crash through normal trend or swing trade strategies. Because the news movement is just as likely to go in a favourable direction as not, the long term profitability of most EAs is either not greatly impacted, or even enhanced by the increased volatility. But avoiding the news does have the potential to reduce short term drawdown.

Some EAs go out of their way to profit from the news events themselves, either by having a bet each way and straddling the news event with buy/sell stops, or by exploiting the high volatility just after news.

The challenge of handling news is twofold.

1. Backtesting
It is almost impossible to backtest news management or avoidance accurately because there is no historical database of news events. A worthwhile rough check is to be aware that there are a small number of timeslots during the week in which news events are frequently scheduled. For instance, US and Canadian news often comes out at 12:30pm GST. Allowing for daylight savings periods, blocking trades between 12pm and 2pm will avoid most major news from these countries and give a crude indication in a backtest of whether more comprehensive news avoidance may be worthwhile. News management can also be implemented by tightening stops or exiting trades before news.

2. Downloading news events automatically
Thanks to a series of contributions from a number of forex enthusiasts including Derk Wheler who wrote the FFCal news indicator, the Forex Factory site provides its news data downloadable in XML format . I wrote some code some time ago which processes this, which I added to a NewsTrader EA to enable it to operate automatically. I recently dusted this off and started a forward test on a virtual host, which will hopefully perform better due to low ping times to the broker. But that's another post.

Differences between live and demo

Forex forums are peppered with many comments along the lines that since most EAs lose money when trading live even though they looked good when backtested, then there must be something different about live accounts versus demo accounts. This often moves on to accusations of stop-hunting and other bad or even illegal behaviour by the broker.

The reality is that the behaviour of live and demo accounts is very similar -- it is unusual to see more than a couple of pips of difference between them. Extremely sensitive EAs can be affected by the difference, but the far more likely explanation for the live/demo differences is that the backtest was invalid in some way.

Here are some of the gotchas which will produce an invalid backtest, none of which are caused by differences in demo vs live

• Over-optimisation is the single most common cause. Put simply, over-optimisation means that the backtest is curve-fitting -- cherry-picking good trades and discarding bad ones with insufficient generalisation. Because forex price series are very close to random, the number of variables that can be optimised (the "degrees of freedom", in statistics parlance) must be carefully limited.
• Use of tight stops or take-profits (less than 20 or 30 pips) with the "every tick" option in the strategy tester. The MT4 strategy tester generates ticks from 1 minute data using "fractal interpolation", which is another way of saying that they are completely fictitious. This is why they nominate the quality of the backtest as "90%".
• Great care in the design of an EA is needed when determining trade entries using Bid, Close[0], or indicators with shift = 0
• Use of time-based entries, eg attempting to time entries for the beginning of trading sessions. This is a very difficult type of entry to backtest because of broker timezones and daylight saving changes. The forex price history downloaded by MT4 may not have the same timezone as the broker.

And finally, if I were a scam EA seller who wanted to convince more people to buy my dud EA, here's how I would do it:

1. Design an EA which shows impressive runs of 100% profitability over several weeks, then dies spectacularly with one massive loss. Many EAs will perform like this with a large stoploss of 500 pips or more.
2. Run the EA on a new demo account, wait for it to accumulate a week or two of profit, then market the EA by giving out the demo account number to prospective buyers.  Voila!  A valid forward test, indisputably making money!  This gold-mine is for sale for only $175!
3. Whenever the test EA crashes, goto 2 ...

Virtual order manager

Time to develop a virtual order manager.  This will potentially be useful now for the FIFO rules in the US, and also will help me to go through the thought processes needed to build one eventually in MQL5.

Although I've seen quite a few VOMs, the reason they have usually been developed is to avoid stop-hunting.  Many have the incredibly risky approach of eliminating broker stops entirely, which means you have no risk control if your internet connection or PC goes down.  Nonetheless, I should do a search for anything out there that might be useful first.

The VOM that I would like to see, or develop, goes like this:

• Real orders at the broker reflect total position for that pair, but the VOM also maintains a virtual set of orders on the MT4/MT5 platform.  So you could be virtual long EURUSD 0.2 lots and virtual short EURUSD 0.3 lots, and the real position showing in the open orders display would be short 0.1 lots.
• Virtual stops and virtual takeprofits are maintained as pending virtual orders in the opposite direction.
• Maintain all the usual virtual Magic numbers and comments etc
• Option to display virtual open prices, SLs & TPs on the chart
• Lastly, the most important and the thing missing from most VOMs I've seen: maintain a real broker stoploss.  This will ensure that disaster protection is there, but under normal conditions should never be hit, ie shouldn't get in the way of the virtual stops.

[update 20/11/2009] a new post on Virtual Order Manager development

MQL5

I'm keenly watching for news on the new Metatrader 5 platform, which will introduce the MQL5 programming language.  MQL5 is supposed to have a number of very welcome features, such as OO, structures and increased speed.
http://www.mql5.com/ still just says "MQL5.COM is coming soon", and the main information comes from an interview with the developer Stanislav Starikov.  There's an excellent compilation of information on http://metatrader5.blogspot.com/.

The MT5 programming language apparently will take on some of the C++ OO attributes, but that alone wouldn't necessarily make MQL4 code incompatible with MT5, since C code is forwards compatible with C++.
The main reason for incompatibility is that position management is going to be completely different. As I understand it, open positions maintained by MT5 will reflect the total position for that pair, and it will not be possible to have a simultaneous long and short trade with the broker, such what often happens now with a long term EA running with a scalper.
This change in position management approach may need to happen now anyway even in MT4, with the FIFO rules against hedging introduced in the US.
The MT5 approach doesn't stop the practice of maintaining opposite positions, but it means that position management code will need to maintain "virtual positions" in the EA.

Neural Networks

Years ago, I developed a neural network in C++ for a share trading system. It never succeeded because it either wouldn't learn anything, or when it did I couldn't get it to stop curve fitting. Recently after re-reading about early stopping in neural networks (and also remembering the success of Better in the 2007 MT4 championships) I decided to give it a go on forex, and have been immediately encouraged.

Behind the science of neural networks is craft: by trial and error the neural network environment needs to be massaged until it yields some promise. Here's what I have come up with so far.

• Start with a simple non-NN entry strategy which is not profitable and yields large numbers of trades, then use the NN to filter these in much the same way as a more traditional long term moving average would be used. This idea came from http://articles.mql4.com/777. Right now I am using a cross 4 bar high for long and cross 4 bar low for short, and both Stoploss and Takeprofit of 100 pips.
• Use a series of inputs to the NN which are the close price differences from the bars with shift 1,2,3,4,9,16,25,36,49,64,81,100,121,144,169,196,225,256,289,324. This is simply a gradually lengthening time between each close. Divide the difference by the square root of the time difference. This idea comes from Mark Jurik
• Normalise the input vector
• Use a standard simply connected 19-10-1 neural network
• Here's the key new idea: early stopping -- run the learning backpropagation against factset 1 (say three months of 15 minute data), and select the NN which delivers the best equity curve on the following 3 months of data. Typically, and in agreement with many NN texts, the NN will continue to train past the point that it generalises

So here's what has happened so far in a short forward test from 7th August to 21st August - the equity has risen 700 pips with a max drawdown of 300 pips

Update on 26 August - I'm now running a forward test on my new virtual host

Why blog?

• Because I'm interested to see what happens when a new blog starts out of thin air. Will it attract some interest and grow in value because of it? Who knows
• Because I'm hoping that the diary structure of the blog will help me to organise my thoughts