What is evidence-based investing?

If you’ve read this blog before, you’re probably familiar with the phrase “evidence-based investing”. But what exactly does it mean? After all, most investment professionals would claim to base their philosophy on some form of evidence. But that doesn’t mean they’re worth paying any attention to.

In fact it’s remarkably easy to come up with an idea and then to claim it’s backed by rigorous evidence when, in fact, it’s nothing of the sort.

In 2015, for example, a journalist named John Bohannon and two German TV producers set out to demonstrate how easy it is to turn bad science into the big headlines behind diet fads. Their challenge was to persuade as many media outlets as possible to publicise the results of “research” which “proved” that eating chocolate every day helps you lose weight.

So they spent a few thousand euros recruiting research subjects and a doctor to run the study. They intentionally used a tiny data set of 15 participants and just three weeks of data. Over a beer-fuelled weekend with a statistician friend, they tortured the data sufficiently heavily to extract a technically accurate if essentially meaningless conclusion that chocolate consumption had indeed contributed to weight loss.

They submitted their paper to 20 different journals, and a few fake ones too, and within 24 hours the story was accepted for publication by dozens of publications around the globe. The study even made television news in the US and Australia.

Financial newsdesks are bombarded with “research” all the time, and very often it’s about as scientific as John Bohannon’s study on chocolate and weight loss. Almost invariably it has been produced or commissioned by a company with a commercial conflict of interest.

But the problem with bogus evidence in the investment industry runs far deeper. That’s because the research that fund managers use to base their decisions on which stocks to buy and sell is, to a large extent, itself conflicted. Companies struggling for attention have been known to pay for the production of what looks like research by their broker. Those brokers often take payment for their work in warrants, or the right to buy the stock they have promoted at a future date at a substantial discount.

For fund managers, like journalists, it can be very hard to distinguish between research that is properly independent and research that isn’t. It’s often only at the end of a report that the truth is revealed, with a disclaimer along the lines of, “This material is not investment research”.

So, then, what sort of evidence can investors rely on? In short, it’s academic evidence — but no, not all academic evidence. Generally speaking, reliable evidence has four main characteristics.

1. It’s independent

New studies are being published all the time which appear to support a particular investment strategy or course of action. But all too often these reports are produced, or else commissioned and paid for, by companies with a commercial interest in publicising the outcome. Most academics, by contrast, are independent. They don’t have an agenda or a point to prove; instead they leave it to financial practitioners to act on their findings or not.

2. It’s based on robust data analysis

We all know the old adage about lies, damned lies and statistics. It’s true, abuse of data can be very misleading. Often findings are based on too short a time period or a sample that’s too small. Sometimes the fund industry ignores survivorship bias; in other words, it overlooks those funds which performed so poorly that they no longer exist. Other times it compares returns to the wrong benchmark, or it quotes performance figures before the full impact of fees and charges. Sometimes it simply gets its maths wrong.

3. It’s been peer-reviewed

To test whether their findings are reliable, academics publish their research in credible academic journals. This gives other academics the chance to agree or disagree on whether the results are sound. Again, caution is required — there are journals that are less credible than others — but evidence that has been properly peer-reviewed should carry far more weight with investors than evidence that hasn’t.

4. The results been reproduced

The fourth and final characteristic of findings you can depend on is that they’ve been tested across multiple environments and timeframes. There is some disagreement on the extent to which academic finance is properly scientific. Asset prices, for example, never move in exactly the same way as they have done in the past. However, there needs to be a strong element of repeatability to demonstrate that the findings of a particular study weren’t just down to random luck or else reached through “data mining”.