News management

Any observer of the Forex market quickly becomes acquainted with the peculiar phenomenon of news spikes. Some EAs are vulnerable to news because the spikes tend to crash through normal trend or swing trade strategies. Because the news movement is just as likely to go in a favourable direction as not, the long term profitability of most EAs is either not greatly impacted, or even enhanced by the increased volatility. But avoiding the news does have the potential to reduce short term drawdown.

Some EAs go out of their way to profit from the news events themselves, either by having a bet each way and straddling the news event with buy/sell stops, or by exploiting the high volatility just after news.

The challenge of handling news is twofold.

1. Backtesting
It is almost impossible to backtest news management or avoidance accurately because there is no historical database of news events. A worthwhile rough check is to be aware that there are a small number of timeslots during the week in which news events are frequently scheduled. For instance, US and Canadian news often comes out at 12:30pm GST. Allowing for daylight savings periods, blocking trades between 12pm and 2pm will avoid most major news from these countries and give a crude indication in a backtest of whether more comprehensive news avoidance may be worthwhile. News management can also be implemented by tightening stops or exiting trades before news.

2. Downloading news events automatically
Thanks to a series of contributions from a number of forex enthusiasts including Derk Wheler who wrote the FFCal news indicator, the Forex Factory site provides its news data downloadable in XML format . I wrote some code some time ago which processes this, which I added to a NewsTrader EA to enable it to operate automatically. I recently dusted this off and started a forward test on a virtual host, which will hopefully perform better due to low ping times to the broker. But that's another post.

Differences between live and demo

Forex forums are peppered with many comments along the lines that since most EAs lose money when trading live even though they looked good when backtested, then there must be something different about live accounts versus demo accounts. This often moves on to accusations of stop-hunting and other bad or even illegal behaviour by the broker.

The reality is that the behaviour of live and demo accounts is very similar -- it is unusual to see more than a couple of pips of difference between them. Extremely sensitive EAs can be affected by the difference, but the far more likely explanation for the live/demo differences is that the backtest was invalid in some way.

Here are some of the gotchas which will produce an invalid backtest, none of which are caused by differences in demo vs live

• Over-optimisation is the single most common cause. Put simply, over-optimisation means that the backtest is curve-fitting -- cherry-picking good trades and discarding bad ones with insufficient generalisation. Because forex price series are very close to random, the number of variables that can be optimised (the "degrees of freedom", in statistics parlance) must be carefully limited.
• Use of tight stops or take-profits (less than 20 or 30 pips) with the "every tick" option in the strategy tester. The MT4 strategy tester generates ticks from 1 minute data using "fractal interpolation", which is another way of saying that they are completely fictitious. This is why they nominate the quality of the backtest as "90%".
• Great care in the design of an EA is needed when determining trade entries using Bid, Close[0], or indicators with shift = 0
• Use of time-based entries, eg attempting to time entries for the beginning of trading sessions. This is a very difficult type of entry to backtest because of broker timezones and daylight saving changes. The forex price history downloaded by MT4 may not have the same timezone as the broker.

And finally, if I were a scam EA seller who wanted to convince more people to buy my dud EA, here's how I would do it:

1. Design an EA which shows impressive runs of 100% profitability over several weeks, then dies spectacularly with one massive loss. Many EAs will perform like this with a large stoploss of 500 pips or more.
2. Run the EA on a new demo account, wait for it to accumulate a week or two of profit, then market the EA by giving out the demo account number to prospective buyers.  Voila!  A valid forward test, indisputably making money!  This gold-mine is for sale for only $175!
3. Whenever the test EA crashes, goto 2 ...

Why blog?

• Because I'm interested to see what happens when a new blog starts out of thin air. Will it attract some interest and grow in value because of it? Who knows
• Because I'm hoping that the diary structure of the blog will help me to organise my thoughts