Main | One in the Hand is Worth Two in the Bush »

September 28, 2004

If You Torture the Data Long Enough, it Will Confess to Anything

Is it possible to time the market?

DALLAS, TX - I was teaching a SYGMATM Workshop last weekend, during the course of which, I began explaining some of the assumptions which underlie the SYGMA Method. One of those assumptions is that market timing and stock picking don't work.

"This is based on the data", I told them. "I have not seen or read of a verified case of someone who has been able to time the market successfully, and beat the market's returns on a risk- adjusted basis over long periods of time", I said. "There are an infinite number of documented cases of people who failed - and lost a lot of investor's money in the process."

Market timing is attempting to predict the future direction of the market, typically through the use of technical indicators or economic data, and to buy and sell accordingly.

Many investors, especially academics, believe it is impossible to time the market. Other investors, notably active traders and those who make their living selling actively managed financial products believe strongly in market timing. So, whether market timing is possible is really a matter of some debate.

During the break, one of the students came up to me and said, "I know you follow Mark Hulbert."

Well, that is not exactly true. Mark Hulbert has been producing the well-respected Hulbert's Financial Digest since 1980. His newsletter tracks the performance of other newsletters, which is generally pretty dismal.

I don't follow Hulbert because I don't subscribe to newsletters or believe in the stock picking or market- timing abilities of the writers. But it is fair to say that I use the data from Hulbert's database, which is widely quoted, to support my assertion that no one can time the market or beat the market by picking stocks.

In fact, Hulbert's data says that in any given year, 80% of newsletter writers have performance that is worse than the market and the one's that outperform vary from year to year. Just because you beat the market one year doesn't give you any better chance of beating it the next.

In other words, there is none of what academics call persistence. Their results are simply a matter of chance, not some special skill at divining the future. This finding is consistent with studies of brokerage analysts, mutual fund managers, and individual investors. So, no, I don't follow Hulbert, but I know his data pretty well.

"He has shown newsletter writers who were able to time the market", my student said.

"Not over long periods of time", I said. "It's not possible."

"Yes. Over twenty years", he said.

I told him I didn't think so and asked him to show me the Hulbert data he was referring to. And he did.

My student sent me a copy of the April 2004 Financial Digest and asked for my comments. Well, here they are.

This issue purports to show five newsletters which were able to beat the market over the last twenty years. In spite of the fact this was only five newsletters out of the 500 portfolios his web site says he covers, I was of course interested and began reading further.

Turns out the five examples he has cited were not cases where the buy and sell recommendations of the newsletter writer actually beat the market. These were hypothetical portfolios created using a method called back-testing.

Backtesting is a technique that has many critics, including me. It's a hypothetical analysis of how an investment strategy would have done if you'd used it over a certain period of years.

Less politely put, "it's a theory on something that worked in the past that, by the time they get assets in it, doesn't work anymore," says Don Phillips, president of Morningstar Inc., the mutual fund research service.

And, as a rule, it's a technique that sends regulators into a cold sweat. "Historical data is easily manipulated," says Barry Barbash, director of the investment management division of the Securities and Exchange Commission. "It's only [used] when it's good, and there is a possibility for it to be cherry-picked or misused."

What HFD has done is to substitute the hypothetical purchase of the Wilshire 5000 index for each stock pick that the writer actually made. When the writer recommends a sell, HFD substitutes 90 day T-Bills for cash.

So in this hypothetical portfolio, the only two choices are the Wilshire 5000 and 90 day T-Bills. The idea here is that by eliminating the stock picks and substituting an index, any gains or losses are limited to the writer's market-timing ability rather than good stock picking.

Just for the record, if you had followed the stock picks the writer's actually recommended, you would have done much worse than the market itself. Using Hulbert's own example, following the advice given by the Cabbot Market Letter would have gotten you an 8.8% annualized return over the twenty year period, while the market's return was over 12%.

But by using this hypothetical portfolio substituting the Wilshire 5000 for Cabbot's stock picks, the hypothetical portfolio would have returned 13.3% - better than the market returns.

The fallacy in the logic should be pretty obvious. This is a shining example of the old saying, "If you torture the data long enough, it will confess to anything!"

The conclusion that these writers could have beaten the market if they had wanted to - but didn't - is preposterous on its face. A little bit of logic reveals that the conclusion is the result of two common flaws of logic that plague back-tested models: they are known as "back-testing brilliance" and "data mining".

These refer to errors in logic that occur by throwing a computer at a set of data. Eventually, you will find a pattern that fits the hypothesis you are trying to prove, but that does not make it valid as a predictor of future success.

Some of the best known examples of data mining are the Hemline Index and the idea that the future direction of the stock market is predicted by whether the NFC or AFC wins the Super Bowl.

It is easy to conclude that because historical data shows a correlation between the length of the hemlines on ladies skirts and the stock markets, that future hemline length will predict future stock market direction, but it just isn't the case. It is an example of finding the survivor within a set of rules that could possibly work. This is also called data fitting and it is merely a coincidence.

Lest you be too amazed at how big a coincidence is, let me give you an example from Nassim Taleb's excellent book, Fooled By Randomness. If you meet someone randomly, there is a 1 in 365 chance that you share the same birthday. If you put twenty-three people in a room, the chances that someone in that room will share a birthday suddenly becomes 50%.

In other words, our perception of coincidences is that they are rare, but in reality, they are easy to turn up, especially when you begin fitting a set of rules to historical data.

According to Dean LeBaron, "What passes for investment research is usually back-testing: if I know now what I should have known then, how good the results would be." Is it any surprise that looking at historical data always produces wonderful results, but that application of the same concept in real time almost always fails?

Case in point is Value Line. Many people point to Value Line's stock picking methodology as evidence stock picking can be done successfully. When back-tested, Value Line produces results that consistently beat the market by a wide margin - even on a risk adjusted basis.

And yet, when applied in real-time by the fund manager's at Value Line's own mutual funds, their results have been exactly what we would expect. Over ten years, Value Line funds which follow their own stock picking scheme have had a return of 11.1% while the market averaged 12.9%. Over a fifteen year period, the discrepancy was even bigger.

With regard to the hypothetical portfolios in Hulbert's Financial Digest, I would submit that Mr. Hulbert, who I must confess I hold in pretty high regard for his generally forthright analyses, has fallen pray to the cognitive biases that afflict all of us if we are not careful.

To those of you who still believe that stock picking and market timing can be done: show me a complete track record with real money that stands up to careful scrutiny and then - and only then - will I be impressed.

 

Kim Snider, Kim Snider Financial Communications, Chronim Investments and/or Snider Advisors make no representation that the information and opinions expressed are accurate, complete or current. The opinions expressed should not be construed as financial, legal, tax, or other advice and are provided for informational purposes only. Call 866-952-0100 to request the Snider Investment Method™ Owner's Manual, which includes a description of the Snider Investment Method, investment objectives, risks, suitability and other information. Please read and consider carefully before investing. All investments are subject to risk including possible loss of principal.

Posted by Kim Snider on September 28, 2004 | Permalink

Comments

The comments to this entry are closed.