Blog #58 The Shape of Stock Returns II

I can’t tell you for sure where the stock market is heading next week, but I can show you a formula on what return you can expect will earn if you follow a buy and hold strategy over the long term. There two key phrases in my sentence: “buy-and-hold” and “long-term”. The formula estimates the average return on your investment assuming you are a patient investor expecting a long-term reward from the stock market. It is not a speculator’s formula.

Here is the formula:

g ≅ µ – (1/2)σ²

On the left-hand side, g is the average compound rate of return on a buy-and-hold investment (say a stock portfolio). g is also known as the geometric mean rate of return.

The word “compound” implies that one simply allow short-term (say, monthly) returns to snowball into something bigger. The “snowball effect” is aided by your investment’s simple average return, also known as the arithmetic mean. The arithmetic mean is denoted by mu (µ). All things equal, a high µ means a high g. Unfortunately, the second term (1/2)σ² spoils the party. σ² is the variance of the stock portfolio. It is defined as the square of standard deviation; both are measures of risk. More volatile stocks have higher variance while stable stocks have lower variance.

The wavy symbol ≅ means that g is approximately equal to µ – (1/2)σ². Thus, the formula gives an approximation for g, not the exact solution. Still, approximations are useful because it saves us a lot of manual calculations! More importantly, the formula says that there is “tug-of-war” between µ and (1/2)σ². Everybody likes a stock that has high µ and low σ² but in general, this isn’t possible because risk and return go hand in hand (I have more to say about this interesting topic but let’s save it for another day).

Now let’s answer two important questions: what use is the formula and how to check whether it “works”?

If the formula is accurate, it should give us a close approximation of the geometric mean return earned on a buy-and-hold investment. To see if this is so, I collected a long time series of monthly stock returns to calculate (a) the exact g and (b) the approximate g given by the formula. If the formula is good, (a) should be close to (b).

My monthly returns pertain to two portfolios consisting of stocks traded in the US from 1962 to 2013. The portfolios are “Low-Vol” and “High-Vol”.  Low-Vol primarily consists of low volatility stocks. These are stocks whose prices don’y jump around too much from month to month.  High-Vol on the other hand comprises mainly high-volatility stocks. This portfolio has a disproportionate share of firms in sectors like semiconductors, computer software, digital equipment, aerospace, and precision manufacturing among others. You can download the spreadsheet here.

For each portfolio, I compute and present the following statistics: arithmetic mean, variance, exact g and approximate g. Here are the results:GM_result.jpg

Focus on the last two rows. You see that for each portfolio, the approximate g is pretty close to the exact g, which confirms that the formula does indeed work well. Best of all, it achieved this feat using just two variables, µ and σ². In physics, it is often said that the most powerful equations are the simplest (think of Einstein’s E=mc²). We don’t have many simple yet powerful equations in finance, but I’m proud to say that g ≅ µ – (1/2)σ² is one of them.

This is well and good, but can we use the formula to predict future returns?  My qualified answer is yes, provided you get your µ and σ² forecasts right. How then does one get good forecasts for µ and σ²? It doesn’t hurt to start with historical estimates of µ and σ² covering a long sample period (to prevent cheery picking particular stretches of stock market history that one fancies). You can then shade these historical estimates higher or lower if you think that past performance is unlikely to faithfully copy the past (this is admittedly a tricky exercise that calls for some expert knowledge in economics). To give you some broad perspective, historically, stock markets have rewarded investors with inflation-adjusted or real arithmetic mean returns of between 4 and 6 percent annually. If you add back the historical global inflation of 4% a year, this implies a historical range of 8% to 10% for nominal average returns (see Blog #30 Why I Love History). Some experts believe that going forward, a globally diversified equity portfolio will probably deliver average returns in the range of 5 to 7% as the low-hanging fruits have mostly been picked, and the world faces challenges due to aging populations, declining fertility, and slower productivity growth. Using a middle rate of 6% and assuming an annual standard deviation of 15%, our little formula implies that you can expect an annual compounded rate g of 4.88% in nominal terms. Even at this modest rate of return, every $100,000 invested over 30 years will compound to a median wealth of nearly $4.2 million. Such is the power of compounding over long horizons.


Blog #57 The Shape of Stock Returns I

If you have been following this blog lately, I have basically been saying that (a) investors try too hard to beat the market by attempting to predict it (b) despite their best efforts, most are doomed to fail.

Some of you may think that I’m too negative about active management (I am). Also, since I used he words “noisy” and “unpredictable” on a number of occasions to describe the zigs and zags of the stock market, you may also have the impression that I view the stock market as nothing more than a casino. That would be incorrect though. In this and the next blog, I will show why I have a high opinion of the stock market as a long-term investment tool. Instead of writing a three-page piece to bring my point across, I will show simply you a picture (below). The picture has an important message. You “read” the story by starting in the top row, left to right, then the bottom row, left to right.


What is this all about?  Why is it important?

The picture shows how wealth invested in a stock is created if you compound its returns over time. It’s a bit like how you roll over your time deposit after it matures because you want to earn “interest on interest”. Except your time deposit interest rate is fixed but stock returns bump around randomly.

How to model this bumping around random nature of stock prices? Here’s where we turn to statisticians for help. Statisticians use probability distributions, including some clever complicated ones to fit or model stock returns. Fear not. I can tell you what you really need to know using the simplest model.

This is the elegant binomial model. “Bi” means two. So, I will ask you to imagine a stock whose price evolves each period with just two outcomes: either its price goes up or it comes down compared to the price in the previous period. Since up (u) is one outcome and down (d) is another, we have just modelled a stock using the binomial model! A useful way to think about this model is to imagine the stock price as an expanding tree. At each node of the tree, the stock price splits into an up branch with a certain probability (say p) and a down branch with probability 1-p (these probabilities must add up to one if you recall from high school :))

The picture that you saw simulates a binomial stock price going forward from 2 nodes, to 5 nodes, to 50 nodes and finally 500 nodes. Using the tree analogy, it “grows” the tree from a skinny one with just 2 nodes to a bushy one with 500 nodes. More importantly, the 2-node tree represents the short-term (no compounding yet) while the 500-node tree represents the long-term after many periods of compounding. Instead of plotting all the branches sticking out (ugly!), the picture tallies all the end-period wealth and presents their distribution as a nice, smooth curve. Notice this curve has a long right tail. So it is not quite the normal or bell-shaped curve that most people are familiar with, but its long right-tailed cousin (the exact name for it is the log-normal distribution).


What’s the big deal about the log-normal distribution? Three things. First, it shows your mean and median wealth after many periods of compounding (they are indicated by the vertical lines). Half of all your possible wealth will be below the median and half above it. So, the median wealth gives you a “feel” of where you are likely to end up after years of investing. That’s good to know.

Second, the median wealth is lower than the mean. This is a bit unfortunate but is always true of right-tailed distributions.

Third, making up for the fact that the median is less than the mean is the long tail. This feature implies that a long-term investor can look forward to “bonus” lucky draws of very handsome returns. This is good news. Despite all the talk about “black swans”, the reality is that long-term investors are more likely to meet “good” outliers or “white eagles”!

Studies using actual data shows that the log-normal distribution closely model the distribution of long-term wealth derived from the stock market. Hence, we can be confident about its predictions regarding the shape of long-term returns. Now you can see why I do not think that the stock market is merely a casino for speculators. All gambling is a negative sum game (i.e., the average net payoff is negative).  True investment is different. In the long run, stocks generally pace the economy and trend upwards. It is most assuredly, a positive-sum game!. See “The Shape of Stock Returns II” coming up next..


Blog #56 Why We Persist in the Prediction Illusion?

The last few blogs sports a central theme: stock market prediction is a loser’s game for 99% of humanity (the other 1% are the odd investment geniuses we now know but wished we had known much earlier).

If market timing, momentum trading, contrarian trading, technical analysis etc. are detrimental to our wealth, then why do so many still swear by these activities? I believe that the reasons lie beyond finance and has a lot to do with psychology.  In particular, understanding how our beliefs inform or misinform us is the first crucial step in improving the way we manage our personal finance and investments.

At the risk of over-simplifying a huge literature on this subject, here are three main reasons why we often become victims of the loser’s game:

#1 We are pattern-seekers.
The “we” here refers to practically all of us. This is because the normal human brain is hard-wired to look for patterns, whether they are cloud patterns or patterns in stock prices. But as I’ve mentioned many times, stock prices are actually pretty random (which is why they are often described as ‘random walks’). We don’t see them as random because we look with the benefit of hindsight. But hindsight is a poor guide to the future when the underlying object of prediction has a high noise-to-signal ratio as is the case with the prices of financial securities.

#2 We have a poor understanding of probability.
You throw a fair coin six times and you get HHHHHT. We have trouble making the correct sense out of such situations. Some of us will think that the next throw will have a more than 50% chance of landing on a head (after all, there seems to be ‘momentum’ in the coin tosses so far). Others will disagree and bet that the next throw will land on a tail (reasoning that a tail is long ‘overdue’). Both reasoning would be wrong in the current context; heads and tails both have an equal chance of appearing in any given coin toss. We laugh at the folks who fail this simple probability test, yet we fall into a similar trap when we try to second guess future stock price changes based solely on how they have moved in the past.

#3 We regularly get our timing of thoughts and events mixed up.
Have you ever felt as though you predicted exactly when the traffic light is going to turn green, or sensed that the doorbell is about to ring? Did you think you had a sixth sense or a god-given clairvoyance for getting these predictions right? Or did you attribute them to pure coincidence?

Recent psychology research [1] shows that people who think they are “special” are also more likely to think that they have some sort of clairvoyance. The research was designed as follows. First, the subjects of the experiment were asked questions to assess whether they think they have special abilities to tell the future. Next, these subjects were asked to play a game in which they were asked to quickly predict which of 5 white squares was about to turn red. Importantly, the square that turned red from one trial to another was chosen randomly. Therefore, it was impossible to correctly predict the red square with a higher probability than 1 in 5 or 20% (this random aspect of the experiment was known only to the researchers).

Subjects could either indicate that they didn’t have time to finish making a prediction before the red square shows up, or they can state that they did make their prediction before this event and predicted either correctly or incorrectly. If subjects got the timing of their thoughts and the event mixed up, they might think they made the right prediction before the red square turns up more often than chance would explain. Let’s call them the mis-timers. As it turns out, there were indeed many mis-timers. More interestingly, mis-timers were more likely to have delusional beliefs in their “special abilities”.  The study therefore hints at a basic deficit in the brain machinery that plays to folks with the flawed belief that somehow, they are blessed with the superhuman ability to tell the future.

[1] See Adam Bear, Rebecca Fortgang, and Michael Bronstein (2017), “Mistiming of thought and perception predicts delutionality”, Proceedings of the National Academy of Sciences.  Here is the article link.


Blog #55 Let’s Go Fishing

Unless you’ve been living on another planet, you would have come across experts musing about the potential of technology to revolutionize finance.  To be sure, some of the new technologies are indeed game changers – stuff like e-payments, block-chain and cyber-security. These advances are important, but they are kind of mundane. Investors are more excited about the prospect of software so powerful that can comb through tons of financial data, pick up interesting patterns and make accurate predictions about future asset returns. If and when that happens, we will surely have arrived at the gilded age of robo-assisted predictions.

But this is a big “if” and this blog explains why. To do that, we’ll go for a “fishing trip”, where the “fishes” we’re after are a handful of influential predictors of stock returns, out of a sea of possible predictors.

Suppose you want to predict tomorrow or next month’s stock price using a bunch of predictors. There are a multitude of possible predictors for the stock market including things like interest rates, bond yield spreads, inflation expectations, industrial output and GDP numbers and so forth.

So you can see that the first challenge of making good predictions is how to separate good predictors from bad ones.  This turns out to be a daunting challenge even with modern technology.

Let’s say you want to fish out 13 of the best predictors. By “best”, I mean having the best linear fit (something like Y = aX1 + bX2 + cX3 + etc), which is called a linear regression model. Here the X’s are the set of predictors, and the “a’s”, “b’s” and “c’s” are sensitivity coefficients that measure how a change in a particular X changes Y. For stock market predictions, Y is naturally the rate of return on a stock or a stock index.

I need to emphasize that your model must show that it has a good in-sample fit (i.e., based on past data) before we can even talk about prediction. Statisticians measure a model’s goodness-of-fit using a number called adjusted R-square. Never mind how adjusted R-square is computed (Excel can easily do it). All you need to know is that th adjusted R-square ranges from zero (the model doesn’t explain stock returns at all ) to one (a perfect fit).  From my research experience using annual stock returns, models with adjusted R-squares of more than 0.3 are rare, which is a reminder than stock returns are pretty random or not systematically explained by observed fundamentals.

Going back to fitting problem, here’s the key question: how many regression models must you run to find the best 13 predictors out of 100?

Hold your breath.

Answer: 7,110,542,499,799,200

A one followed by 15 zeroes is called 7 quadrillion. So this number is about 7 quadrillion.  How big is that? For comparison, there 100 billion stars in our galaxy. Hence, if this number represents another galaxy, that galaxy has 71,105 times more stars than ours. The number gets even bigger if we start out with 1,000 potential predictors instead of just 100, so large in fact that it takes a computer which can perform 10 million calculations a second more than 22 years to finish the task! For all practical purposes, this type of search problem is intractable. Computer scientist call such problems NP-complete (where NP stands for non-deterministic polynomial time). NP-complete problems are those that are too hard for today’s computers and possibly those in the foreseeable future.

It gets worse. So far, we’ve only been concerned with finding the best k out of m predictors using linear regression based on past data. Even if you manage somehow to find the best k predictors, it doesn’t mean that you’ve found a perfect crystal ball for prediction. Why? Because the world keeps changing. As they say, “past results do not guarantee good future performance”. Hence, your best prediction model using past future may churn out lousy predictions for next month’s stock returns. You need keep learning and updating your model to cope with new market conditions. That is a tough call.  It is hard enough to find the best fit model using past data. To have the best forward-looking model at all times is pure fantasy even for the most sophisticated machine learning software available today. The best we can do with large problems like these is to use heuristics or shortcuts but the problem with shortcuts is that you can never be sure whether the resulting ad-hoc model is robust and accurate.

So, the next time you hear someone claiming to have a state-of-the art robo-advisor that can accurately predict stock returns, you may want to ask if he knows what a quadrillion is 🙂


Blog #54 Is Machine Learning the Holy Grail of Financial Prediction?

Since we are in the era of ‘big data’ and fast computers, you may wonder whether machine learning has conquered finance, more specifically whether the computer can now replace humans in making accurate predictions of asset prices. If so, more power to robo-advisers!  If not, we’ve got to downplay the hype.

There has indeed been big leaps in in machine learning in specific domains (by the way, I will use machine learning interchangeably with artificial intelligence or AI). In 2017, the world recently witnessed one particularly impressive feat when Deepmind’s AlphaGo program beat the world’s champion Lee Se-Dol in the complex game, Go. Deepmind went on to develop an even smarter version of the program called Zero that can learn on its own, given the basic rules. After three days of self-play, Zero was smart enough to defeat the version of itself that beat Lee Se-dol, winning handily 100 games to nil. We are told that there is more to come!

Despite this impressive feat, it is strange that we don’t hear much about the success of AI in finance. I can think of two reasons why this is. One, the tech geniuses are keeping their secrets close to their chest for obvious reasons. Two, maybe there isn’t much success to brag about in the first place. It’s possible that even today’s clever machine learning technologies are not clever enough to be able to predict stock and bond prices accurately. To understand why this might be the case, let’s consider the limits of AI according to some of the world’s top AI experts.

Dr. Dave Ferrucci, co-founder and chief executive of Elemental Cognition and a former AI expert at IBM stresses that machine learning is simply a statistical technique for finding patterns in large amounts of data. He adds that “having a computer spew out an answer is not sufficient in the long term. You want to say, ‘here’s why‘ “.  But understanding the “whys” of financial asset prices is much harder than understanding the rules of a board game like Go. Despite its complexity, Go, like all board games, is actually quite easy for computers to figure out because the rules are finite – there is no hidden information and importantly, no element of luck (unlike finance). This means that researchers applying AI to Go can have access to a perfect simulation of the game. They can program their software to run millions of tests and be sure it’s not missing anything. Not many fields meet these criteria. Key examples that do include language translation, speech recognition and image recognition. Finance isn’t one of them.

It is not difficult to see why.  As Stanford University’s Andrew Ng, one of the founders of Google Brain, Google’s deep learning project, emphasizes, AI works only for problems where clear inputs can be mapped or linked to a clear output. This criterion limits the applicability of AI to problems involving categorizations such as the fields mentioned above.  Finance is orders of magnitude more complex than these problems. Take stock market predictions for example. It is incredibly hard to pin down the most important inputs or causal variables that drive stock prices. First, there are too many of variables to consider (GDP, inflation, interest rates, oil prices, exchange rates, and most slippery of all, market sentiments). As Warren Buffet once said: “investment is a game of a million inferences.”

Secondly, the financial environment is a dynamic one. Variables that are highly influential for stock prices during high inflation periods may be less so during low inflation periods and vice versa. Trying to “catch” the best set of predictors is like trying to catch a butterfly; the more you chase it, the more it will elude you.

Third, even if you think you can nail down have the relevant predictors variables, how do you put them together in a model to generate your predictions. A model is an equation that relates what you want to predict (say, stock returns) to a set of predictor variables. Coming up with an accurate prediction model is far from trivial. You can ask any finance professor; I guarantee you will walk away unsatisfied with the answer (I know; I was a finance professor myself).

What about computer scientists and mathematicians, the people most closely associated with pushing the limits of AI? Swetava Ganguli and Jared Dunnmon are two ‘quants’ at Stanford University who has done research in using AI to predict bond prices. They ask: how good different machine-learning techniques are in predicting the future price of corporate bonds? They use standard “shallow learning” techniques as well as more exotic neural-network techniques to answer this question.

Before I summarize their findings, some background on bonds. Corporate bonds are IOUs issued by firms or to raise money for the firm’s business. In return, bond investors receive interest and at the bond’s maturity, the bond’s principal value. As every bond investor knows, bond prices are interest sensitive. Moreover, they have to contend with default risk. The higher the default risk, the lower is the bond price. Yet, compared to stocks, bonds are relatively simple instruments and bond prices are generally less volatile.

Given these features of bonds, you would think that predicting bond prices is a piece of cake for machine learning programs. This is not what the researchers found! I won’t go into the details of their study (see the reference below).  They found that while the best predictions came from the neural networks, these fancy techniques took several hours to work their magic – too slow to be useful for actual bond traders. Interestingly, simpler such as linear regressions aren’t so bad in terms of both accuracy and speed. This is a slap in the face for AI. At the same time, it also affirms that simpler methods may be more robust (read: less bad) for highly complex problems, a fact that data scientists have known for some time. That may be the reason why there are so few stories about the success of AI in financial predictions.

Swetava Ganguli and Jared Dunnmon (2017), “Machine learning for better models for predicting bond prices“, arXiv:1705.01143 (link)

Blog #53 Let’s Talk About Momentum Trading

I tested a popular contrarian trading strategy in blog #50 using Shiller’s PE ratio and found it wanting. The flip side of contrarian is momentum trading, which involves buying stocks after they have risen and selling stocks after they have fallen. So, momentum is about “going with the flow”, instead of against it. Since the contrarian strategy doesn’t seem to deliver the goods, you may wonder whether momentum trading will do the trick. This blog will attempt to answer this question with data.

As before, my data is from Robert Shiller’s website and consists of (a) the price levels of the S&P 500 index from Jan 1881 through Dec 2017 and (b) the index’s cyclical-adjusted PE ratio (CAPE) computed from its constituent stocks.

I will present two versions of the momentum strategy: a basic version and a ‘crash proof’ version. I will call these versions, MOM1 and MOM2 respectively.

MOM1 works as follows: starting with a capital of $1, I buy the index if the CAPE in each of the previous twelve months is above 15  (call this trading signal CAPE >15). Otherwise, I do nothing. If the index is bought, it is held until a sell signal emerges i.e., when CAPE < 15. To keep things simple, I assume that trading cost is zero and that cash earns nothing. I repeat this strategy each month to the end of the sample period (Dec 2017). I then compute the terminal value (TV) by compounding the monthly returns and compare it with the terminal value of the buy-and-hold (BH) strategy. The strategy with the higher TV is the winning strategy.

The performance of MOM1 is shown in the first line of the table below.

MOM1’s terminal value is shockingly low – just $2.58, compared with $430.43 for the buy-and-hold strategy. In fact, it is worse than the contrarian strategy discussed in blog #50. The BH strategy is always invested, while MOM1 is out of the market about 56% of the time. This can be seen from the following graph where “1” indicates being is in the market and “0” indicates being out of the market. The high outage of MOM1 is what causes its terminal value to be so embarrasingly low.

Can we salvage the momentum strategy by tweaking it? The answer of course, you can!  After all, all trading rules are arbitrary. As a researcher, I am trained to be skeptical of data mining as this is called. Data mining is useless in data where there are no repeatable patterns to aid predictions, as is the case with stock prices which behave close to random walks. Nevertheless, to satisfy skeptics, I will relent and subject another version of the momentum strategy to the test. I will call this version, MOM2.

MOM2 works like MOM1 except that it suspends buy trades following major stock market crashes. MOM2 draws on behavioral finance research. This research shows that investors tend to freak out after a big drop in stock prices. Daniel Kahneman (of Thinking Fast and Slow fame) calls this fear, loss aversion. The bigger the drop, the more it sticks in the mind, and the more loss-averse investors become. Hence, MOM2 stops buying after major crashes even if the trading signal screams ‘buy’.

It is tedious to split hairs over the definition of a crash. So, in the interest of simplicity, I suspended buy trades in just three crashes in the sample period. They are: the October 1929 crash which led to the Great Depression, the Internet bubble crash in 2001 and most recently, the 2008 crash triggered by the US sub-prime crisis. For each of these crashes, no buy trades were allowed until the S&P 500 index regained its pre-crash level. After that, trading follows MOM1.

As expected, MOM2 improves on MOM1 (see second row of table above). Alas, this is small comfort as MOM2’s terminal value of $4.59 is barely one-tenth that of the buy-and-hold strategy!

The above results come as a surprise to me, as I am sure it is to you.  But they seem robust. They are also in agreement with other evidence presented earlier, notably the tendency for investors to chase ‘hot’ markets to their detriment (see blog #46). So, unless your brilliance leads you to the discovery of a wonder trading rule, “neither a contrarian nor momentum trader be” seems to be a sensible (if boring) strategy to follow in your quest for investment wealth.

Blog #52 Does Goggle Hold Your Ticket to Riches?

Google’s search engine has replaced the encyclopedias that I grew up with in the 1970’s. Nowadays, one goes straight to Google for answers on just about anything, including of course news about stock market and what other investors are thinking about the stock market.  You can probably see where I’m heading – Google search is literally ‘big data’ at your finger-tips, perhaps the ultimate place to become insanely rich by finding exploitable connections between financial variables. So, how good is Goggle as a source of information for predicting stock prices?  This is what I want to talk about today.  I happened to have done some serious research on this topic which I will share with you in a moment. Let let me first say a few words about data mining to set the stage.

A couple of years ago, two researchers at the University of Bristol combed a database containing 133 different variables, looking for pairs of variables that are ‘statistically significant’. Being scientists, they set the standards for significance quite high, so that the chances of getting flukes were just 1 in 100.

What they found stunned them. Out of almost 8,800 possible pairings, more than 3,000 were deemed ‘significant’. This is like catching 3,000 tunas with one fling of the fishing net from waters with a random assortment of 8,800 different types of fishes. A fisherman would be ecstatic with this result. But to the statisticians, it was too good to be true. Indeed, when they looked more closely at the 3,000 ‘significant’ variable pairs, most were junk.


Here’s another story of such ‘voodo correlations’. A few years ago, psychology Professor Edward Vul at the UC San Diego stumbled on a bizarre study that claimed to show a link between brain activity and the speed at which people walk. Curious, he and his colleagues investigated. What they found was shocking. The authors of the study had simply fished out from a random set of data, patterns that happen to fit their pet theory and then claimed that the results were ‘statistically significant’. This is a classic case of cherry picking – keeping what you like to see by throwing away what you don’t like to see! Thankfully, the majority of scientific studies uphold much higher standards. But there is broader lesson about pattern seeking and it is that our brains are incredibly clever in inventing stories to ‘fit the facts’, often downplaying the role of chance in the unfolding of events. The desire for coherence in noisy situations can be a problem for example, when you put your hard-earned money in the stock market based on skimpy evidence. I’m the first to admit that I have zero ability to predict the gyrations of stock market, which is why I choose to be long-term, buy-and-hold investor.

Before signing off, I promised to share with you my own research on predicting the stock market with Google search. I have two versions of the paper, one for an academic audience and the other, written for the general public. You can read the latter paper here.