My favourite sites

Forex Peace Army
Even though I believe the guys that run this site have hearts of gold, this site is not entirely immune to abuse by scam artists, but it's still the most honest forex resource on everything from broker ratings, signals service reviews and now even EA forward tests

Pip Cop
An EA forward testing site that even runs some real live testing

Global Forex Signals
One of many signals sites, which I have been using for at least two years with a web page scraping EA which works much more reliably than an email reader. I can vouch for the authenticity of their records. Only a very few of the systems are reliable money-spinners long term, though. Global Trader and Pipsrain are excellent.

MQL4.com
Regularly delights me with excellent R&D articles on their forum

Forex Factory
An excellent news calender, made even more useful because it is available in XML format

Metatrader 4 Experts and Indicators
Probably the most active forex discussion groups around. Has a huge code archive.

Support Resistance EA V6_0

Update 12/1/2010: this forward test now replaced with an MT5 Virtual Order Manager equivalent coexisting with the FrAMA Cross EA. See statement here and live screenshot here. All links below are still valid.


This EA is my most successful to date, although I still wouldn't call it a really good EA.  Its strategy is the age-old concept that price will sometimes bounce off significant support and resistance points.  When I was designing it, I had an idea that I could scalp small profits off the reversals, but instead stumbled on a large takeprofit of 360 points.  The result is an EA for which the profitable trades figure is not much more than 30%.  The combination of 360 TP and 30% profitable makes it rather hard to trade, because it frequently gives back profits early in the trade.

It has pulled about 1500 pips off the market since July 2008 when I started trading it live.  Why not 2,194 pips, which is what the backtest now says it should have done?  Simply because I manually exited a lot of trades, which is an interesting example of trading psychology at work as a result of the difficulty in trading it.  Trading system purists could scoff at my weak-kneed approach, but remember that I was (and to a lesser extent, still am) not certain that it will be profitable in the long term.

I run it at the moment with no money management at a fixed 0.1 lot.  Here are three backtests from 1/1/2004 to 28/8/2009

• Flat 0.1 lot  Quite encouraging - maximum drawdown is 832 pips
• Money management with 2% x 80 pip risk - probably too agressive, but it's nice to dream.  A toned down money management at 1% looks a lot better
• Martingale - just for fun.  Even though this never trades above 0.9 lot, I've never found an EA that I would be comfortable running live in martingale mode

Note that the open prices only model is valid for this EA because it uses shift = 1 on all entry signals, and has such a large takeprofit.

Running MT4 on a virtual host

I recently decided to move all my live and test MT4 clients to a virtual host in the US. I wanted better reliability and I'm also hoping that faster network access to the broker server will be beneficial. I was surprised at how easy it was - I paid out $30 for the first month to Commercial Network Services and got my live account up and running remotely in about two hours. The only problem I have had so far is getting a gmail reader dll to work. It feels strange to be able to switch off my laptop during the week after having it running almost continuously for two years.

This virtual host is a Windows 2003 Server x64 instance in a data centre somewhere in New York, with excellent backup and network connectivity. It shows up as a Windows Terminal Server shortcut on my laptop. When I double-clicked on this for the first time, I was greeted with a normal Windows login screen, and even the Start Windows sound when I logged in. You can elect to have everything translated back to your local environment such as disk drives and printers. Aside from rather choppy screen updates, in full screen mode it looks and feels much like a local machine. When I run Windows Explorer on the remote host I can see my local drives, and transfer files (rather slowly) between environments.

Commercial Network Services has data centres in a number of cities, and I chose New York because it had the fastest ping times from FXDD. Whereas in Australia my laptop reports 400 millisecond ping times (yes, that's 0.4 second) from FXDD, this virtual host is 30 milliseconds away. Hopefully this will reduce the number of requotes and invalid price errors. My most demanding EA from a processing and network point of view is a news trader, and I'll be tweaking it in demo over the next few weeks to try to take advantage of the faster network times.

A discussion about the random walk hypothesis

Since I have "random walk" in the title of my blog, I suppose I should post some comments about it. The random walk hypothesis proposes that movements of any financial instrument such as forex or stock prices are completely random and therefore cannot be predicted. I'm in good company when I disagree, since it is quite clear to me that sentiments such as fear and greed, and the herd mentality of the market can and do cause non-random behaviour. My search is for the trading opportunities that these "non efficient" occurrences present.

But now, a trick question - do you agree with my analysis of the following charts?

Is my analysis OK? Perhaps you noted some triangle patterns that I missed, maybe a breakout?

The only problem is that they are selected from a random walk of 2,000 price ticks. That's right - using the Rand() function in Microsoft Excel, I created this spreadsheet today and just pressed F9 (recalculate) until I obtained a pattern I liked!  So what does this tell us? It means that the price movement we see can be close to random but can easily lure us into thinking that it is not.

After years of developing systems, I have come to the conclusion that there is a sweet spot somewhere between the following:-

1. Looking for large numbers of trades, which yields a statistically more significant backtest. More trades come from shorter timeframe charts and small profits per trade. It's very difficult to obtain enough trades to be comfortable with a backtest on a daily chart.
2. Looking for nuggets of sentiment amongst the noise. These seem to become more prominent in longer timeframes and larger price movements of at least 100 pips.

So this leads me to zero in on systems with TPs and SLs in the 50 to 500 pip range, on 15 minute to 1 hour chart timeframes. The smaller timeframes and stop values apply better to breakout and support/resistance type systems. The longer timeframes and stop values apply better to trend following systems.

I do think that scalping systems (ie very short timeframe, small profits) are possible, but I have never had any success in finding one. My theory is that some price behaviours and indicator patterns are so universally interpreted the same way that they become a self-fulfilling prophesy. A price might drop briefly, not because it is overbought, but because enough people think that it is overbought.

News management

Any observer of the Forex market quickly becomes acquainted with the peculiar phenomenon of news spikes. Some EAs are vulnerable to news because the spikes tend to crash through normal trend or swing trade strategies. Because the news movement is just as likely to go in a favourable direction as not, the long term profitability of most EAs is either not greatly impacted, or even enhanced by the increased volatility. But avoiding the news does have the potential to reduce short term drawdown.

Some EAs go out of their way to profit from the news events themselves, either by having a bet each way and straddling the news event with buy/sell stops, or by exploiting the high volatility just after news.

The challenge of handling news is twofold.

1. Backtesting
It is almost impossible to backtest news management or avoidance accurately because there is no historical database of news events. A worthwhile rough check is to be aware that there are a small number of timeslots during the week in which news events are frequently scheduled. For instance, US and Canadian news often comes out at 12:30pm GST. Allowing for daylight savings periods, blocking trades between 12pm and 2pm will avoid most major news from these countries and give a crude indication in a backtest of whether more comprehensive news avoidance may be worthwhile. News management can also be implemented by tightening stops or exiting trades before news.

2. Downloading news events automatically
Thanks to a series of contributions from a number of forex enthusiasts including Derk Wheler who wrote the FFCal news indicator, the Forex Factory site provides its news data downloadable in XML format . I wrote some code some time ago which processes this, which I added to a NewsTrader EA to enable it to operate automatically. I recently dusted this off and started a forward test on a virtual host, which will hopefully perform better due to low ping times to the broker. But that's another post.

Differences between live and demo

Forex forums are peppered with many comments along the lines that since most EAs lose money when trading live even though they looked good when backtested, then there must be something different about live accounts versus demo accounts. This often moves on to accusations of stop-hunting and other bad or even illegal behaviour by the broker.

The reality is that the behaviour of live and demo accounts is very similar -- it is unusual to see more than a couple of pips of difference between them. Extremely sensitive EAs can be affected by the difference, but the far more likely explanation for the live/demo differences is that the backtest was invalid in some way.

Here are some of the gotchas which will produce an invalid backtest, none of which are caused by differences in demo vs live

• Over-optimisation is the single most common cause. Put simply, over-optimisation means that the backtest is curve-fitting -- cherry-picking good trades and discarding bad ones with insufficient generalisation. Because forex price series are very close to random, the number of variables that can be optimised (the "degrees of freedom", in statistics parlance) must be carefully limited.
• Use of tight stops or take-profits (less than 20 or 30 pips) with the "every tick" option in the strategy tester. The MT4 strategy tester generates ticks from 1 minute data using "fractal interpolation", which is another way of saying that they are completely fictitious. This is why they nominate the quality of the backtest as "90%".
• Great care in the design of an EA is needed when determining trade entries using Bid, Close[0], or indicators with shift = 0
• Use of time-based entries, eg attempting to time entries for the beginning of trading sessions. This is a very difficult type of entry to backtest because of broker timezones and daylight saving changes. The forex price history downloaded by MT4 may not have the same timezone as the broker.

And finally, if I were a scam EA seller who wanted to convince more people to buy my dud EA, here's how I would do it:

1. Design an EA which shows impressive runs of 100% profitability over several weeks, then dies spectacularly with one massive loss. Many EAs will perform like this with a large stoploss of 500 pips or more.
2. Run the EA on a new demo account, wait for it to accumulate a week or two of profit, then market the EA by giving out the demo account number to prospective buyers.  Voila!  A valid forward test, indisputably making money!  This gold-mine is for sale for only $175!
3. Whenever the test EA crashes, goto 2 ...

Virtual order manager

Time to develop a virtual order manager.  This will potentially be useful now for the FIFO rules in the US, and also will help me to go through the thought processes needed to build one eventually in MQL5.

Although I've seen quite a few VOMs, the reason they have usually been developed is to avoid stop-hunting.  Many have the incredibly risky approach of eliminating broker stops entirely, which means you have no risk control if your internet connection or PC goes down.  Nonetheless, I should do a search for anything out there that might be useful first.

The VOM that I would like to see, or develop, goes like this:

• Real orders at the broker reflect total position for that pair, but the VOM also maintains a virtual set of orders on the MT4/MT5 platform.  So you could be virtual long EURUSD 0.2 lots and virtual short EURUSD 0.3 lots, and the real position showing in the open orders display would be short 0.1 lots.
• Virtual stops and virtual takeprofits are maintained as pending virtual orders in the opposite direction.
• Maintain all the usual virtual Magic numbers and comments etc
• Option to display virtual open prices, SLs & TPs on the chart
• Lastly, the most important and the thing missing from most VOMs I've seen: maintain a real broker stoploss.  This will ensure that disaster protection is there, but under normal conditions should never be hit, ie shouldn't get in the way of the virtual stops.

[update 20/11/2009] a new post on Virtual Order Manager development

MQL5

I'm keenly watching for news on the new Metatrader 5 platform, which will introduce the MQL5 programming language.  MQL5 is supposed to have a number of very welcome features, such as OO, structures and increased speed.
http://www.mql5.com/ still just says "MQL5.COM is coming soon", and the main information comes from an interview with the developer Stanislav Starikov.  There's an excellent compilation of information on http://metatrader5.blogspot.com/.

The MT5 programming language apparently will take on some of the C++ OO attributes, but that alone wouldn't necessarily make MQL4 code incompatible with MT5, since C code is forwards compatible with C++.
The main reason for incompatibility is that position management is going to be completely different. As I understand it, open positions maintained by MT5 will reflect the total position for that pair, and it will not be possible to have a simultaneous long and short trade with the broker, such what often happens now with a long term EA running with a scalper.
This change in position management approach may need to happen now anyway even in MT4, with the FIFO rules against hedging introduced in the US.
The MT5 approach doesn't stop the practice of maintaining opposite positions, but it means that position management code will need to maintain "virtual positions" in the EA.

Neural Networks

Years ago, I developed a neural network in C++ for a share trading system. It never succeeded because it either wouldn't learn anything, or when it did I couldn't get it to stop curve fitting. Recently after re-reading about early stopping in neural networks (and also remembering the success of Better in the 2007 MT4 championships) I decided to give it a go on forex, and have been immediately encouraged.

Behind the science of neural networks is craft: by trial and error the neural network environment needs to be massaged until it yields some promise. Here's what I have come up with so far.

• Start with a simple non-NN entry strategy which is not profitable and yields large numbers of trades, then use the NN to filter these in much the same way as a more traditional long term moving average would be used. This idea came from http://articles.mql4.com/777. Right now I am using a cross 4 bar high for long and cross 4 bar low for short, and both Stoploss and Takeprofit of 100 pips.
• Use a series of inputs to the NN which are the close price differences from the bars with shift 1,2,3,4,9,16,25,36,49,64,81,100,121,144,169,196,225,256,289,324. This is simply a gradually lengthening time between each close. Divide the difference by the square root of the time difference. This idea comes from Mark Jurik
• Normalise the input vector
• Use a standard simply connected 19-10-1 neural network
• Here's the key new idea: early stopping -- run the learning backpropagation against factset 1 (say three months of 15 minute data), and select the NN which delivers the best equity curve on the following 3 months of data. Typically, and in agreement with many NN texts, the NN will continue to train past the point that it generalises

So here's what has happened so far in a short forward test from 7th August to 21st August - the equity has risen 700 pips with a max drawdown of 300 pips

Update on 26 August - I'm now running a forward test on my new virtual host

Why blog?

• Because I'm interested to see what happens when a new blog starts out of thin air. Will it attract some interest and grow in value because of it? Who knows
• Because I'm hoping that the diary structure of the blog will help me to organise my thoughts