When Diversification Fails

The Myth of Diversification

Based on a precrisis data sample ending in February 2008, Chua, Kritzman, and Page (2009) documented significant “undesirable correlation asymmetries” for a broad range of asset classes. Not only did correlations increase on the downside, but they also significantly decreased on the upside. This asymmetry is the opposite of what investors want. Indeed, who wants diversification on the upside? Upside unification (or antidiversification) would be preferable. During good times, we should seek to reduce the return drag from diversifiers.

Despite the wide body of published research, we believe many investors still do not fully appreciate the impact of correlation asymmetries on portfolio efficiency—in particular, on exposure to loss. During left-tail events, diversified portfolios may have greater exposure to loss than more concentrated portfolios. Leibowitz and Bova (2009) showed that during the 2008 global financial crisis, a portfolio diversified across US stocks, US bonds, international stocks, emerging market stocks, and REITs saw its equity beta rise from 0.65 to 0.95, and the portfolio unexpectedly underperformed a simple 60% US stocks/40% US bonds portfolio by 9 percentage points.

In this article, we expand the analysis of Chua et al. (2009) in several ways. We include post-2008 data, we cover a broader set of markets, and we take an in-depth look at what drives correlations in numerous markets. As for methodology, we introduce a data-augmentation technique to improve the robustness of tail correlation estimates, and we analyze the impact of return data frequency on private asset correlations.

Measuring Tail Correlations

How correlations change during extreme markets can be estimated in several ways. For example, Longin and Solnik (2001) and Chua et al. (2009) used “double conditioning.” They isolated months during which both assets moved (up or down) by at least a given percentage. We used a similar approach, but we conditioned on a single asset, as follows: $\rho \left(\theta \right)=\{\begin{array}{l}\text{corr}\left(x,y|x>\theta \right)\text{ if }\theta >0\\ \text{corr}\left(x,y|x<\theta \right)\text{ if }\theta <0\end{array}$ (1) where x and y represent the two assets, $\theta$ is the return threshold below or above which we partitioned the data, and $\rho \left(\theta \right)$ is the conditional correlation.

Unlike Longin and Solnik’s (2001) approach, “single conditioning” measures differences in tail correlations based on which market drove the selloff. For some correlations, such as the stock–bond correlation, this difference can be substantial, and it adds information on the correlation structure. For example, we wanted to evaluate the effectiveness of bond diversification during US stock market selloffs (the flight-to-safety effect). First, we isolated months in our data sample during which US stocks, x, were down by, say, 5% or more (we calibrated thresholds, θ, to correspond to percentiles). Next, we calculated a correlation between stocks and bonds in this subsample, denoted $\text{corr}\left(x,y|x<-5\%\right).$

We also calculated the correlation between stocks and bonds when bonds, y, were down by 5% or more, denoted $\text{corr}\left(x,y|y<-5\%\right).$ As we will show, in this case, we found that bonds diversify stocks during stock selloffs but stocks do not diversify bonds during bond selloffs. Double conditioning would fail to reveal this lack of symmetry in the diversification between the two assets.

Potential Biases

Irrespective of how we partitioned the data, we expected subsample correlations to differ from full-sample estimates, even for a joint normal distribution. To measure this “conditioning bias,” we first simulated how correlations change when movoing toward the left and right tails of a bivariate normal distribution. For each asset pair, we simulated two normal distributions with the same full-sample correlations, means, and volatilities as those we observed empirically. Then, we compared the empirical subsample correlations with their simulated normal counterparts. Differences indicate departures from normality. Also, under normality, downside and upside correlation profiles should be identical. Therefore, when left-tail and right-tail correlations are compared, the conditioning bias does not matter much because it “washes out.” Any asymmetry we found indicates a departure from normality.

Another possible bias arises because extreme correlations rely on few data points. The further one goes into the tails, the smaller the sample. At the top or bottom 1% or 5% of the distribution, a single outlier may significantly bias correlations up or down. To increase robustness in our estimates, therefore, we augmented subsamples with data from the rest of the distribution. To do so, we used an exponentially weighted approach, as illustrated and derived in Appendix A. To our knowledge, this approach, although simple and intuitive, has not been used in prior studies; hence, perhaps we are making a modest methodological contribution to the measurement of conditional correlations.

We calibrated the model in such a way that observations further into the tails receive exponentially larger weights, and we fixed the half-life at the percentile under consideration. For comparison, we also report unadjusted conditional correlations. We found that the data-augmentation methodology generates estimates similar to those calculated conventionally, in terms of magnitude and directionality. Our estimates tend to be less noisy, however, and are generally less sensitive to outliers.

An important point regarding the conditioning bias is that we applied the same exponential adjustment to the corresponding simulated normal data. Hence, in all cases, comparisons between empirical and normal correlations are apples-to-apples.

The Failure of Diversification in International Equity Portfolios

The material in this section on US equity correlation with international equity will illustrate our approach. First, based on monthly data from January 1970 to June 2017,Footnote² we calculated conditional correlations between US stocks (MSCI US Total Return Index) and non-US stocks (MSCI EAFE Total Return Index).Footnote³ We conditioned correlations by percentile, based on the returns of US stocks. In Figure 1, we show how correlations changed from the worst selloffs in US stocks (1st percentile) to their strongest rallies (99th percentile). For comparison, the dotted line shows the correlation profile that we would expect if both markets were normally distributed. In the normal case, we would expect perfect symmetry between upside and downside correlations and conditional correlations would gradually decrease as we moved toward the tails.

Figure 1. Conditional Correlation Profile for US vs. Non-US Stocks, January 1970–June 2017

Notes: US stocks are represented by the MSCI US Total Return Index, and non-US stocks are represented by the MSCI EAFE Total Return Index in local currency. Empirical conditional correlations were adjusted by the data-augmentation methodology.

Display full size

As Figure 1 demonstrates, empirical correlation profiles differ substantially from their normally distributed counterparts. When US stocks were rallying (in their 99th percentile), their correlation with non-US stocks dropped all the way to –17%. During the worst 1% selloffs in US stocks, however, their correlation with non-US stocks rose to +87%. This asymmetry reveals that international diversification works only on the upside. Longin and Solnik (2001), focusing on the correlations between the United States, France, Germany, the United Kingdom, and Japan, reported similar results for stocks at the country level.

The Failure of Diversification across Risk Assets

We found similar results across risk assets. Figure 2 provides a comparison of left-tail and right-tail correlations for key asset classes.Footnote⁴ The focus is on US stocks versus other risk assets because the equity risk factor dominates the volatility factor (and exposure to loss) in most portfolios (see, for example, Page 2013). Note that we used bond returns net of duration-matched US Treasuries (i.e., “excess returns”) to isolate credit risk factors. We also show results for style and size diversification within stocks. Most investors select equity funds—and thereby seek to diversify their portfolios—based on style/size characteristics. Across the board, left-tail correlations in Figure 2 are much higher than right-tail correlations.

Figure 2. Left-Tail vs. Right-Tail Correlations for Key Risk Assets, June 2017

Notes: EM is emerging market. Monthly data, with start dates based on availability (see Appendix B, available online at www.cfapubs.org/doi/suppl/10.2469/faj.v74.n3.3), for start dates and data sources). Left-tail and right-tail correlations are at the 1st and 99th percentiles but were adjusted by the data-augmentation methodology. Full correlation profiles (adjusted, unadjusted, and normal) are shown in Appendix B.

Display full size

Studies on tail dependence corroborate these findings. Garcia-Feijóo, Jensen, and Johnson (2012) showed that when US equity returns are in their bottom 5%, non-US equities, commodities, and REITs also experience significantly negative returns—beyond what would be expected from full-sample correlations. Hartmann et al. (2010) showed that currencies co-crash more often than would be predicted by a bivariate normal distribution. Hartmann, Straetmans, and de Vries (2004) estimated that stock markets in G–5 countries were two times more likely to co-crash than were bond markets. Van Oordt and Zhou (2012) extended pairwise analysis to joint tail dependence across multiple markets and reached similar conclusions. They suggested a related approach to measure the systemic importance of financial institutions. These studies ignored asymmetries, however, between the left and right tails. They either focused on the left tail or used symmetrical measures of tail dependence, such as the joint t-distributions.

Regarding credit asset classes, the Merton (1974) model explains why credit and equity returns become more correlated in the left tail. Merton defined a corporate bond as a combination of

• a risk-free bond—in normal times, the bondholders’ upside risk is limited to the regular coupon payments and return of principal—and

• a short put position on the company’s assets. If the company’s asset value depreciates below its debt, bondholders become long the company’s assets and receive what’s left through bankruptcy proceedings. (Meanwhile, as the stock price goes to zero, stockholders are wiped out.)

Hence, as a company approaches default, the market starts to expect that bondholders will be left holding the company’s remaining assets. Merton explained that “as the probability of eventual default becomes large, . . . the risk characteristics of the debt approach that of (unlevered) equity” (463). In this context, Naik, Devarajan, Nowobilski, Page, and Pedersen (2016) argued it was not a surprise that during the 2008 crisis, credit and equity returns became highly correlated.

Diversification fails across styles, sizes, geographies, and alternative assets. Essentially, all the return-seeking building blocks that asset allocators typically use for portfolio construction are affected. The asymmetry for the stock–MBS (mortgage-backed securities) correlation is notable. Chua et al. (2009) used precrisis data, and at the time of their study, MBS were one of the few asset classes that seemed to decouple from stocks in down markets. During the fourth quarter of 2008, however, which is included in our data sample, MBS clearly joined the ranks of “risk-on” assets.

Hedge Fund Styles and Diversification

Beyond traditional asset classes, investors have increasingly looked to alternatives for new or specialized sources of diversification. For Figure 2, we used a broad hedge fund index, but one could argue that hedge fund styles are so different from each other that they should be treated as separate asset classes. So, in Figure 3, we show a comparison of left-tail and right-tail correlations of seven hedge fund styles versus US stocks. Unfortunately, all the styles, including the market-neutral funds, exhibit significantly higher left-tail than right-tail correlations

Figure 3. Left-Tail vs. Right-Tail Correlations for Hedge Fund Styles, June 2017

Notes: Left-tail and right-tail correlations are at the 1st and 99th percentiles but were adjusted by the data-augmentation methodology. See also the notes to Figure 2.

Display full size

A simple explanation could be that most hedge fund strategies are short volatility. Some are also short liquidity risk, which is akin to selling an option (Bhansali 2010). Agarwal and Naik (2004) explained jumps in hedge fund left-tail equity betas through the Merton (1974) lens. They observed that “a wide range of hedge fund strategies exhibit returns similar to those from writing a put option on the equity index” (92). In a related study, Billio, Getmansky, and Pelizzon (2012) used a regime-switching model to measure hedge fund correlations and market betas over time. They showed that the average jump in correlations for hedge fund strategies in financial crises was +33%.

What about Private Assets?

Although many investors have become skeptical of the diversification benefits of hedge funds, the belief in the benefits of direct real estate and private equity diversification has been persistent. Over the past few years, institutional investors have significantly increased their allocations to private assets. The advisory firm Willis Towers Watson reports that as of the end of 2016, pension funds, wealth managers, and sovereign wealth funds held more than $2 trillion in direct real estate and private equity investments (Flood 2017). Money has flowed into these asset classes partly because of their perceived diversification benefits. Consultants have used mean–variance optimization in asset allocation or asset/liability studies to make a strong case for increased allocations. Alternative assets are often sold as free lunches because they seem to offer high returns with low volatility and great diversification properties.

Most investors know, however, that there is more to these statistics than meets the eye. Private assets’ reported returns suffer from the smoothing bias. In fact, Pedersen, Page, and He (2014) showed that the private assets’ diversification advantage is almost entirely illusory. On a marked-to-market basis, these asset classes are exposed to many of the same factors that drive stock and bond returns.

Not only is the true equity risk exposure of private assets higher than is implied by their reported returns on average, but their left-tail exposures are much higher. In Figure 4, we show a comparison of quarterly to rolling annual (four-quarter) left-tail correlations with equity for direct real estate and private equity. Full correlation profiles are reported in Appendix B (available online at www.cfapubs.org/doi/suppl/10.2469/faj.v74.n3.3).

Figure 4. Left-Tail Correlations with Equity for Direct Real Estate and Private Assets, June 2017 (quarterly vs. rolling annual data)

Notes: Left-tail correlations are at the 1st (bottom) percentile but were adjusted by the data-augmentation methodology. See also the notes to Figure 2.

Display full size

Rolling annual correlations are less sensitive to the smoothing bias than those calculated on quarterly returns. As explained in Pedersen et al. (2014), reported quarterly returns for private assets represent a moving average of the true (unobserved) marked-to-market returns. Also, the authors showed that private assets, after their smoothing bias is removed, have exposure to credit risk, which does not, as we have shown, truly diversify equity risk in times of market stress. Moreover, liquidity risk contributes to the asymmetry of private asset returns even more than to the asymmetry of hedge fund returns. Page, Simonian, and He (2011) explained that systemic liquidity risk tends to manifest itself during stock market crashes. A systemic liquidity crisis can be compared with a burning building, in which everyone is rushing for the door, with one exception: In financial markets, to get out (sell), investors must find someone to take their place in the building (a buyer).

Risk Factors and the Diversification Benefits of Short Positions

The failure of diversification across public and private return-seeking asset classes has led, in part, to the popularity of risk factors. Bender, Briand, Nielsen, and Stefek (2010); Page and Taborsky (2011); Ilmanen and Kizer (2012); and many others have argued that risk factor diversification is more effective than asset class diversification. For Figure 5, we thus applied the methodology we used in Figures 2 and 3 to risk factors. Again, we focused on diversification versus US stocks. Our results show that several risk factors do indeed appear to be more immune to the failure of diversification than are asset classes.

Figure 5. Left-Tail vs. Right-Tail Correlations for Risk Factors, June 2017

Notes: Monthly data with start dates based on availability (see Appendix B, available online at www.cfapubs.org/doi/suppl/10.2469/faj.v74.n3.3, for start dates and data sources). The value (security selection) and momentum (security selection) factors are long–short, rank-weighted models of US individual stocks. The cross-asset value and momentum factors allocate to equity indexes, currencies, rates, and commodities, also with a long–short, rank-weighted methodology. Left-tail and right-tail correlations are at the 1st and 99th percentiles but were adjusted by the data-augmentation methodology. Full correlation profiles (adjusted, unadjusted, and normal) are shown in Appendix B. For value and momentum, we used data from Asness, Moskowitz, and Pedersen (2013; data available at www.aqr.com). The size factor is from Fama and French (1993). All three currency factors are long–short, as calculated by Deutsche Bank.

Display full size

Idzorek and Kowara (2013) and Cocoma, Czasonis, Kritzman, and Turkington (2017) pointed out, however, that risk factors are not inherently superior building blocks. They deliver better diversification than traditional asset classes simply because they allow short positions and often encompass a broader universe of assets. For example, the size and value factors in equities are often defined as long–short security-level portfolios. But if factor definitions are restricted to linear combinations of asset classes and short positions are allowed for all asset classes as well as risk factors, then risk factors do not deliver any efficiency gains over asset classes. In a sense, the argument in favor of risk factor diversification is more about the removal of the long-only constraint and the expansion of the investment universe than anything else.

In addition, momentum strategies that sell risk assets in down markets provide left-tail diversification. Portfolio insurance strategies, for example, can explicitly replicate a put option (minus the gap-risk protection). Hence, as expected, in Figure 5, currency and cross-asset momentum have much lower left-tail than right-tail correlations with US stocks.

Our results also show, however, that risk-on factors, such as size (i.e., small minus big stocks) and currency carry, may fail to diversify stocks when needed. Small-cap stocks tend to have higher equity betas than large-cap stocks, and this difference in market beta exposure is often expressed during stock market drawdowns. Similarly, the currency carry trade has an indirect equity beta exposure that remains dormant until risk assets sell off. The strategy goes long high-interest-rate currencies (the Australian dollar, emerging market currencies, etc.) and funds these positions by shorting low-interest-rate currencies (e.g., the Japanese yen). In normal markets, the investor earns a risk premium because forward rates typically do not appreciate or depreciate enough to offset profits (the “carry”) from the interest rate differential embedded in currency forward contracts. But when risk assets sell off, the carry trade unwinds as investors sell the higher-risk currencies and buy the safe havens. In a sense, many carry strategies behave like the credit risk premium. These strategies are like being short an option, and investors sometimes refer to the tired adage “picking up pennies in front of a steamroller” to describe them.

Regime Shifts and Investor Sentiment

The example of the currency carry trade illustrates the impact of regime shifts on correlations, which may explain the widespread risk-on/risk-off characteristic of return-seeking asset classes and risk factors. Financial markets tend to fluctuate between a low-volatility state and a panic-driven, high-volatility state (see, e.g., Kritzman, Page, and Turkington 2012). In fact, Ang and Bekaert (2015) directly linked the concept of regime shifts to rising left-tail correlations. But what causes regime shifts? A partial answer is that macroeconomic fundamentals themselves exhibit regime shifts, as documented for inflation and growth data.Footnote⁵

Also, we surmise that investor sentiment plays a large role. In normal markets, differences in fundamentals drive diversification of risk asset returns. During panics, however, investors often “sell risk” irrespective of differences in fundamentals. Huang, Rossi, and Wang (2015), for example, showed that sentiment is a common factor that drives both equity and credit-spread returns—beyond the effects of default risk, liquidity, and macro variables—and suggested that sentiment often spills over from equities to the credit markets.

Apparently, in financial markets, fear is more contagious than optimism. Related studies in the field of psychology suggest that to react more strongly to bad news than good news is human nature. In a paper titled “Bad Is Stronger Than Good,” Baumeister, Bratslavsky, Finkenauer, and Vohs (2001, 1) explained,

Is the Stock–Bond Correlation the Only True Source of Diversification?

When market sentiment suddenly turns negative and fear grips markets, government bonds almost always rally because of the flight-to-safety effect (Gulko 2002). In a sense, duration risk may be the only true source of diversification in multi-asset portfolios. Therefore, the expected stock–bond correlation is one of the most important inputs to the asset allocation decision.

In Figure 6, we show the empirical stock–bond conditional correlation profile and its normally distributed benchmark. Unlike results for other correlations, this profile is highly desirable: Bonds decouple from stocks in bad times and become positively correlated with stocks in good times.

Figure 6. Conditional Correlation Profile for US Stocks vs. Treasuries, January 1973–June 2017

Notes: Monthly data. For Treasuries, we used the Barclays Capital Long U.S. Treasury Index. Results were similar for the Barclays U.S. Treasury Bond and the Barclays Capital Intermediate U.S. Treasury Indexes (not reported).

Display full size

The stock–bond correlation is difficult to estimate, however, and can change drastically with macroeconomic conditions.Footnote⁶ Johnson, Naik, Page, Pedersen, and Sapra (2014) explained that when inflation and interest rates drive market volatility more than business cycles and risk appetites do, the stock–bond correlation often turns positive. For example, the authors showed that the 12-month stock–bond correlation during the 1970s and 1980s was mostly positive. Since 2008, central bank stimulus and declining rates have artificially pushed valuations higher in both the stock and bond markets. This type of “sugar high” can unwind quickly if policy normalizes unexpectedly. The “taper tantrum” of 2013, when Ben Bernanke first mentioned the idea of reducing or “tapering” the Fed’s stimulus, provides a good example. It affected stocks and bonds negatively at the same time. Starting valuations can compound the effect. The higher the valuations in both stocks and bonds, the more fragile their correlation.

To illustrate how bond selloffs can lead to a positive stock–bond correlation, Figure 7 is based on the same data as in Figure 6, but the conditional correlations are reversed. We estimated the stock–bond correlation as a function of percentiles in bond returns instead of stock returns.

Figure 7. Conditional Correlation Profile for Treasuries vs. US Stocks, January 1973–June 2017

Notes: Monthly data. See the notes to Figure 6.

Display full size

The correlation profile is not as desirable as when we conditioned on stock returns. Although the correlations are generally low, when bonds sell off, stocks can sell off at the same time. Ultimately, investors should remember that stocks and bonds both represent discounted cash flows. Unexpected changes to the discount rate or inflation expectations can push the stock–bond correlation into positive territory—especially when other conditions remain constant.

Caveats on Measuring and Forecasting Diversification

We have shown that during crises, diversification across risk assets almost always fails, and even the stock–bond correlation may fail in certain market environments. As a caveat, we advise the reader that conditional correlations represent only one way to measure diversification. Conditional betas, for example, take into account changes in relative volatilities as well as correlations. In theory, it is possible for the correlation between two assets to increase while the volatility of the diversifier decreases relative to the main engine of growth in the portfolio. In this case, a spike in correlation may be offset by decreasing relative volatilities, which could lead to a lowered stress beta and, perhaps, lower exposure to loss than expected. However, prior studies based on betas (e.g., Leibowitz and Bova 2009), on co-crash probabilities (Hartmann et al. 2004, 2010), and on tail dependence (Garcia-Feijóo et al. 2012) have shown such outcomes to be highly improbable. Ultimately, we chose to study correlations as they measure diversification directly, and correlations have been used widely in prior studies.

Another caveat is that we did not forecast left-tail events; hence, although we know that correlations are likely to increase if markets sell off, we do not necessarily know when this shift will take place. Equity selloffs are, almost by definition, unexpected. Investors can prepare for the failure of diversification, however, without the need to time markets. Consider as an analogy that although it is almost impossible for aircraft pilots to predict when they will encounter air turbulence, passengers can take comfort in the fact that airplanes are built to withstand it.

Recommendations for Asset Allocation

We recommend that investors avoid the use of full-sample correlations in portfolio construction—or, at least, that they stress-test their correlation assumptions. Scenario analysis, either historical or forward looking, should take a bigger role in asset allocation than it does. A wide range of portfolio optimization methodologies directly address nonnormal left-tail risk and, ipso facto, the failure of diversification. The most flexible is full-scale optimization (see, e.g., Cremers, Kritzman, and Page 2005; Sharpe 2007; Adler and Kritzman 2007).

These analytics are widely available, but they are often used on a “post-trade” basis—that is, after portfolio construction has taken place. Investors should use such tail-aware tools as part of “pre-trade” decisions. To do so will reveal that equity regions, styles, sizes, and sectors—as well as credit, alternative assets, and risk factors—do not diversify broad equity risk as much as average correlations suggest. To be clear, we are not arguing against diversification across traditional asset classes, but investors should be aware that traditional measures of diversification may belie exposure to loss in times of stress. Investors should calibrate their risk tolerance (against return opportunities) accordingly.

In addition, significant emphasis should be put on the stock–bond correlation and consideration of whether it will continue to be negative in the future. Shocks to interest rates or inflation can turn this correlation positive. In such situations, strategies that use leverage to increase the contribution to the risk of bonds—risk parity, for example—may experience unexpected drawdowns.

Finally, investors should look beyond diversification to manage portfolio risk. Tail-risk hedging (with equity put options or proxies), risk factors that embed short positions or defensive momentum strategies, and dynamic risk-based strategies all provide better left-tail protection than traditional diversification. The strategy of managed volatility is a particularly effective and low-cost approach to overcome the failure of diversification. Based on the empirical observation that risk is more predictable than return, this strategy adjusts the asset mix over time to stabilize a portfolio’s volatility. It is portable and can easily be applied as an overlay to smooth the ride for almost any portfolio. Importantly, because managed volatility scales down risk assets when volatility is high, it often offsets left-tail correlation spikes and thereby reduces exposure to large losses without sacrificing returns on the upside.Footnote⁷

A Final Word

In an apocryphal story, a statistician who had his head in the oven and his feet in the freezer exclaimed, “On average, I feel great!” Similarly, as a measure of diversification, the full-sample correlation is an average of extremes. Conditional correlations reveal that during market crises, diversification across risk assets almost completely disappears. Moreover, diversification seems to work remarkably well when investors do not need it—during market rallies. This undesirable asymmetry is pervasive across markets.

Our findings are not new, but we proposed a robust approach to measure left- and right-tail correlations, and we documented the extent of the failure of diversification on a large dataset of asset classes and risk factors. The good news is that tail risk–aware analytics, as well as hedging and dynamic strategies, are now widely available to help investors manage the failure of diversification.