Having Confidence or Having Faith
The forums that discuss trading systems and their development often ask about the value of walk forward testing. The question is usually accompanied by comments about how hard it is to get good results from the out-of-sample tests from the walk forward runs, whereas it is relatively easy to get good results from optimization and backtesting.
My first reaction is the obvious one — it is hard to get good out-of-sample results because the markets are nearly efficient and it is hard to write a set of rules that detect an inefficiency in advance.
But the first question leads to a deeper consideration about trading systems and trading. Having confidence in a system.
It is my view that the universe of trading system application divides into two — having confidence and having faith.
If you want quantifiable confidence — the kind that tells you whether to hit soft 17 at blackjack, or to hit the blot in your inner table in backgammon, or to buy a recent low, or to buy a new high breakout — follow the methodology I recommend.
Everything else is faith.
The problem is harder than it looks at first blush. The characteristics of a trading system determine to a large extent whether it is even possible to have confidence. In order to be useful, there must be enough data points — usually closed trades — to compute useful statistical metrics. Examples of useful are:
To put a low p-value on a set of system results, such as “we can reject the hypothesis that the expectancy is less than 0.0 with a p-value of 0.05.”
To put limits on estimates, such as “with 90% confidence, the worst maximum drawdown for the next year for an account with an initial balance of $100,000 trading at a fraction of 0.40 is 20%.”
Statistical metrics such as these can be computed for any data set — real or hypothetical. If future trades will be made based on these statistics, the data set used to compute the test statistics must be as unbiased as possible.
Using the walk forward technique with trading systems that trade frequently and have short holding periods gives the trading system developer a reasonable chance of producing a set of trades that is both large enough and unbiased enough. Even at that, it is all too easy to introduce bias — bias that will cause reward to be overestimated and risk underestimated — into even the walk forward out-of-sample results.
Compare with backtesting with little or no out-of-sample testing, which is the all-too-common method (as in 80% or more of the articles in Active Trader or Technical Analysis of Stocks and Commodities), or with systems that have such long holding periods or infrequent trading that an unbiased data set cannot, for practical purposes, be produced.
The walk forward method provides the opportunity for developing confidence — whether the developer uses it or not is his or her choice.
In all other cases, the developer and trader must rely on faith.
When in doubt, test it! Do not accept traditional wisdom, such as that the 200 day moving average is a good trend indicator, or that trend following is the best system to grow a trading account with low drawdown, blindly. Those rules may be good ones, and they may lead to trading systems that are appropriate for use.
Beware of Lewis Carroll’s White Queen in “Through the Looking Glass:” “Why, sometimes I’ve believed as many as six impossible things before breakfast.”
Test everything yourself. Your logic, your data, your execution, your estimation of system health. If those tests give you confidence, act accordingly.
If you must act on faith, ask yourself how the casino can build such a fine facility. Stand next to the roulette wheel and listen to the young man tell his partner “There have been six reds in a row. Black is due.”
Thanks for listening,
Howard
Reprinted with permission Dr. Howard B. Bandy. (http://www.blueowlpress.com)