Google’s search engine has replaced the encyclopedias that I grew up with in the 1970’s. Nowadays, one goes straight to Google for answers on just about anything, including of course news about stock market and what other investors are thinking about the stock market. You can probably see where I’m heading – Google search is literally ‘big data’ at your finger-tips, perhaps the ultimate place to become insanely rich by finding exploitable connections between financial variables. So, how good is Goggle as a source of information for predicting stock prices? This is what I want to talk about today. I happened to have done some serious research on this topic which I will share with you in a moment. Let let me first say a few words about data mining to set the stage.
A couple of years ago, two researchers at the University of Bristol combed a database containing 133 different variables, looking for pairs of variables that are ‘statistically significant’. Being scientists, they set the standards for significance quite high, so that the chances of getting flukes were just 1 in 100.
What they found stunned them. Out of almost 8,800 possible pairings, more than 3,000 were deemed ‘significant’. This is like catching 3,000 tunas with one fling of the fishing net from waters with a random assortment of 8,800 different types of fishes. A fisherman would be ecstatic with this result. But to the statisticians, it was too good to be true. Indeed, when they looked more closely at the 3,000 ‘significant’ variable pairs, most were junk.
Here’s another story of such ‘voodo correlations’. A few years ago, psychology Professor Edward Vul at the UC San Diego stumbled on a bizarre study that claimed to show a link between brain activity and the speed at which people walk. Curious, he and his colleagues investigated. What they found was shocking. The authors of the study had simply fished out from a random set of data, patterns that happen to fit their pet theory and then claimed that the results were ‘statistically significant’. This is a classic case of cherry picking – keeping what you like to see by throwing away what you don’t like to see! Thankfully, the majority of scientific studies uphold much higher standards. But there is broader lesson about pattern seeking and it is that our brains are incredibly clever in inventing stories to ‘fit the facts’, often downplaying the role of chance in the unfolding of events. The desire for coherence in noisy situations can be a problem for example, when you put your hard-earned money in the stock market based on skimpy evidence. I’m the first to admit that I have zero ability to predict the gyrations of stock market, which is why I choose to be long-term, buy-and-hold investor.
Before signing off, I promised to share with you my own research on predicting the stock market with Google search. I have two versions of the paper, one for an academic audience and the other, written for the general public. You can read the latter paper here.