Jonathan Falk points us to this article by Kewei Hou, Chen Xue, and Lu Zhang, who write:
In retrospect, the anomalies literature is a prime target for p-hacking. First, for decades, the literature is purely empirical in nature, with little theoretical guidance. Second, with trillions of dollars invested in anomalies-based strategies in the U.S.market alone, the financial interest is overwhelming. Third, more significant results make a bigger splash, and are more likely to lead to publications as well as promotion, tenure, and prestige in academia. As a result, armies of academics and practitioners engage in searching for anomalies, and the anomalies literature is most likely one of the biggest areas in finance and accounting. Finally, as we explain later, empiricists have much flexibility in sample criteria, variable definitions, and empirical methods, which are all tools of p-hacking in chasing statistical significance.
A weakness in this study is that the use of a common data period obscures the fact that financial anomalies are contingent on being unknown: known (true) anomalies will be arbitraged away so that they no longer exist. Their methodology continues to estimate many of these anomalies after the results of the studies were public knowledge and heavily scrutinized. This should attenuate the results. (It would be interesting to see if the results weakened the earlier the study was published. On a low-hanging fruit theory, it should be just the opposite.) It’s as if Power Pose worked until Amy Cuddy wrote about it, at which point everyone wised up and the effect went away. Effects like that are really hard to replicate.
Falk’s comment, about financial anomalies being contingent on being unknown, reminds me of something: In finance (so I’m told), when someone has a great idea, they keep it secret and try to milk all the advantage out of it that they can. This also happens in some fields of science: we’ve all heard of bio labs that refuse to share their data or their experimental techniques because they want to squeeze out a couple more papers in Nature and Cell. Given all the government funding involved, that’s not cool, but it’s how it goes. But in statistics, when think we have a good idea, we put it out there for free, we scream about it and get angry that other people aren’t using our wonderful methods and our amazing free software. Funny, that.
P.S. For an image, I went and googled *cat anomaly*. I recommend you don’t do that. The pictures were really disturbing to me.
You visited SAC (a big hedge fund) about a decade ago for a talk. Did you ever blog about it? You should it!
The rest of this post is wrong in several ways:
1) “In finance (so I’m told), when someone has a great idea, they keep it secret and try to milk all the advantage out of it that they can.”
Whoever is telling you this is clueless. Many funds make it their entire business model to publish their great ideas as a fund-raising strategy. Major examples of this include AQR (which has published 20+ academic papers) and Bridgewater (which even makes videos about its (genius?) insights into how markets work). Many smaller funds do the same. Of course, as in almost every field that needs to sell something, there are those who keep secrets, either because they really have secrets to keep (Renaissance) or because (pretending) to have important secrets is part of their sales pitch. But the blare-it-from-the-rooftops strategy — or at least the tell-any-potential-client-anything-they-want-to-know strategy — is much more common.
2) “known (true) anomalies will be arbitraged away so that they no longer exist.” Hah! Falk is an idiot. Even if you, personally, knew for certain that, say, cheap stocks were going to outperform expensive stocks by 5% a year on average for the next 20 years, you would still have huge problems taking advantage of this knowledge. First, you would need to raise money. That is extremely hard to do. Just because you know something for certain does not mean that you can convince other people that you knowledge is real. Second, you would need to survive the ups-and-downs of the strategy. 5% a year on average does not mean 5% every year. What happens when you have a drawdown of 20% in a bad six month period? All your clients fire you, even though you have a massive edge.
3) The Hou et al study is, in fact, excellent. It highlights how untrustworthy the finance literature is.
Indeed, perhaps the major lesson of this blog is that we should all have much less faith in all portions of the academic literature . . .
— Regular Commentator who needs to be anonymous for this topic
As I recall, when I spoke at SAC I talked about general statistical issues. I have no expertise regarding investing, that’s for sure!
I’m not crazy about Regular Commenter’s tone, but criticism #2 seems plainly right, even to a casual observer of the investing literature. The equity premium is probably the most famous enduring anomaly, and no one knows how to arbitrage it away. (It’s implausible that someone has figured out, but is hiding the secret — that person would already be the world’s first trillionaire.)
> either because they really have secrets to keep (Renaissance)
Are these secrets known unknowns, is so what are they about, or unknown unknowns (“Renaissance so successful, they must be knowing something”)?
I always assumed that better and earlier information, rather than groundbreaking and novel ways to analyse existing information, are behind those are successful without being lucky
An idiot I may be, but you misread my remark. Are you saying that no anomaly ever published in the literature was ever arbitraged away? Many of the measured anomalies are just wrong… that’s the result of the paper and I agree with it. My only point is that some subset of the anomalies might have been true when published but not longer are because they were published. That will make their results look more null-like than they should. It’s a weakness of the study (and the authors were kind enough to write me back that they agreed with me, without publicly asserting my idiocy.) But sure lots of anomalies are resistant to arbitrage for a host of reasons, and those anomalies would have shown up in the Hou et al study as evidence that the literature was performing well.
Renaissance exists, and some of their investing strategies have persisted for years, or so I’m told. But the investments they make aren’t known (true) anomalies, since they guard that knowledge. Renaissance clients, far from firing them, aren’t allowed to invest any more money, which is by itself a statement that Renaissance has arbitraged away a lot of their own ideas, since otherwise they’d be eliciting more funds.
Idiots we all are, in that we’re posting our brilliant ideas in blog comments rather than monetizing them. Although, as Greg Mankiw might say, we might as well give away our ideas for free, what with the marginal tax rate being 93% and all.
“otherwise they’d be eliciting more funds” – not necessarily. Another problem is that anomalies may not scale. The stock market is very top heavy. The SP500 is about 75% of the total market cap of the stock market. If your anomaly doesn’t apply to the SP500, the most closely watched companies, you are playing in a very small sandbox. The ONLY way to have a really large hedge fund is to deal in large companies, and presumably that is the toughest place to find anomalies. If most anomalies revolve around smaller companies, that receive less investor/analyst attention, then there is a limit on how large any fund trying to exploit those anomalies can get.
“blare it from rooftops”
This is probably true of investing methodologies as well as active positions. Although far from always being the case, quite often if you’ve taken a position at a favourable price, then you would want others to know about it shortly thereafter in hopes they see what you see. In order to increase (or decrease) the price you need other investors to move the needle and waiting can be painful.
I don’t read Falk’s comments in the same light as you. However, your point is valid and can simply be put as there are natural limits to arbitrage (https://en.wikipedia.org/wiki/Limits_to_arbitrage).
Here’s a safe picture of a (sort of) anomalous cat; polydactyl cats, although a rare mutation at first, are now bread. Maybe a good metaphor for this post then?
Cats that are bread is indeed disturbing! ;~)
I kinda scared to google cat bread
Your response gave me a good laugh, and that emboldened me to google cat bread. And that resulted in several more good laughs. Try it!
Cats do loaf around a lot.
“It’s as if Power Pose worked until Amy Cuddy wrote about it, at which point everyone wised up and the effect went away”
As a interesting aside – this basic idea applied to all of social psychology briefly freaked out the entire field when Kenneth Gergen wrote about in his 1973 “Social Psychology as history” paper. Then researchers apparently realized that teaching people about all of the ways in which their behavior and judgment is screwed up has very little impact on either their behavior or judgment.
This paper is relevant to the discussion. Note in particular the last line of the abstract.
Abstract: We study the out-of-sample and post-publication return-predictability of 97 variables that academic studies show to predict cross-sectional stock returns. Portfolio returns are 26% lower out-of-sample and 58% lower post-publication. The out-of-sample decline is an upper bound estimate of data mining effects. We estimate a 32% (58% – 26%) lower return from publication-informed trading. Post-publication declines are greater for predictors with higher in-sample returns, and returns are higher for portfolios concentrated in stocks with high idiosyncratic risk and low liquidity. Predictor portfolios exhibit post-publication increases in correlations with other published-predictor portfolios. Our findings suggest investors learn about mispricing from academic publications.
If the market was known for a fact to be 100% efficient (with investors having reasonable access to information including academic literature) and published anomalies didn’t disappear (accounting for tax considerations and transaction costs) then one would have to conclude the anomalies never existed and the academics got it wrong (e.g. p-hacking, poor study design). If the market is known not to be efficient all of the time, and published anomalies didn’t disappear (with the same proviso as before) then what the hell is everyone in finance paid for (assuming the academics get it right, at least sometimes)? Note that the second conclusions follows only if there are assumed to be no behavioral or institutional reasons for why the published anomalies can’t be exploited.
The fact that published anomalies decline in their ability to provide excess risk-adjusted returns post-publication just makes sense.
The fact that this makes it more likely post-publication to look like that the original papers were chasing noise, is, well annoying. But to be fair, a lot of papers are adrift in the sea of spurious correlations.
Thank you very much Hal. It’s sad that the blogosphere picks up on polemic (Hou et al) instead of the Smith Breeden Prize Winner that you bring up.