Evaluating the effects of shelter-in-place policies during the COVID-19 pandemic

The COVID-19 pandemic has attracted immense scholarly interest and spawned a rapidly growing empirical and theoretical literature. We focus specifically on empirical studies of the effects of state and local policies in the United States. Prior studies have focused on the effects of SIP orders on three related types of outcomes: social distancing, economic outcomes, and disease prevalence.

Studies of social distancing have relied heavily on cell phone mobility data. While all studies document a large reduction in mobility associated with the spread of COVID-19, most find the trends were occurring even in states without SIP policies, suggesting that the behavioral changes were largely voluntary. Some have found that state and local policies have been more effective in reducing mobility in wealthier places (3) and more Democratic places (4). On the whole, however, the estimated effects of state and local policies on social distancing appear to be small and to dissipate quickly (5, 6).

Studies of the effects of SIP policies on economic outcomes have also found that, while there have been large economic disruptions associated with the pandemic, SIP policies account for a relatively small share of observed declines in economic activity, again suggesting that behavioral changes were largely voluntary (6–9).

Despite the reportedly small effects of SIP policies on mobility and social distancing and economic activity, several studies have found that SIP orders lead to large reductions in COVID-19 cases (1, 2, 10). These findings pose a puzzle for the literature. How can SIP policies be so effective in reducing the spread of COVID-19 if they don’t meaningfully reduce social distancing? One possibility is that we do not have the right measures of social distancing. Perhaps SIP orders do not meaningfully reduce mobility (as measured by the GPS locations of smartphones), but they do reduce the extent to which people come into close contact with one another. Another possibility is that previous studies have overestimated the effects of SIP orders on COVID-19 cases, and these policies have not dramatically mitigated the spread of the disease. Our reanalysis of the prior studies shows that the latter possibility is indeed the most likely one.

We study two outcomes that we take to be the main target of SIP policies: the number of COVID cases and the number of COVID-related deaths. We normalize each of these variables by state population to measure new cases and deaths per million residents.* Our period of study runs from February 24, 2020 through May 30, 2020, allowing us to examine trends well before the introduction of SIP policies and well after their removal.

In addition, we study the primary mechanism through which SIP orders are thought to influence cases and deaths: social distancing. No single metric fully encapsulates social distancing. Acknowledging this limitation, we utilize anonymized, aggregated smartphone data from the Norwegian location data company UNACAST that measures individual movement at the county level. Specifically, we use a daily measure of the change in average distance traveled compared to a pre−COVID-19 baseline period as a proxy for social distancing. The baseline is represented as the average distance traveled for the same county on the same weekday day over the 4 wk prior to March 8, 2020. The data assign an individual to a county based on where a device is present for the longest amount of time each day, and the data follow 15 million to 17 million devices. This study uses data collected from February through May 2020.

Our final outcome is unemployment, which is meant to capture some of the unintended disruptive impacts of SIP policies. We utilize data from the Department of Labor on the insured unemployment rate in each state, which is computed from unemployment insurance claims. This measure surely understates total unemployment, but it has the advantage, for our purposes, of being updated weekly at the state level.

Utilizing data on the timing of state and county SIP orders, we compute the proportion of state residents who are under a SIP order on a particular day. In cases where the state has implemented a SIP order, this variable takes a value of one, and it can take values between zero and one if there is no state order but one or more counties have implemented SIP orders. SIP orders include “stay-at-home,” “safer-at-home,” and related orders, as well as policies ordering the closure of nonessential businesses. Additional details regarding data sources are provided in SI Appendix.

While estimating policy effects is not easy even in the best of times, there are additional challenges in this setting. First, there were very large changes in behavior taking place both before SIP policies were adopted in states that ultimately adopted them and, at the same time, in states that never adopted SIP policies during our study period. Fig. 1 illustrates this point by showing trends in mobility for every state during our study period. Dashed lines indicate the days on which various states implemented SIP orders, and the dotted lines indicate days on which states withdrew SIP orders. We see, in the figure, that there were dramatic nationwide trends in mobility that are unrelated to state SIP policies, and, in many states, mobility was already declining before the policies were implemented. Analogous figures for our other outcomes are shown in SI Appendix, Figs. S1–S3. These figures show that there were preexisting trends in all outcomes prior to the adoption of SIP policies, meaning that it will be especially important to control for pretreatment trends when attempting to isolate changes in outcomes due to policy.

Second, SIP policies likely have both direct and indirect effects. Direct effects arise, for example, from business and school closures, because of which individuals no longer have reason to travel to those locations. Indirect effects arise from the information conveyed to the public by the enactment of the policy. The announcement of a SIP order might be an informative signal about the severity of disease conditions that could, for instance, cause some people to work from home even in the absence of a workplace closure. Such indirect effects could spill over across state borders, if one state’s policy announcement induces voluntary changes in behavior in other states. SI Appendix presents several additional analyses intended to address concerns about spillovers across states.

To account for differences across states and nationwide trends, we implement differences-in-differences designs with state and day fixed effects. State fixed effects account for any unchanging differences between states, and day fixed effects account for secular trends, allowing us to estimate the effect of SIP policies by testing for differential trends when those policies change. To further address the possibility that states might implement SIP policies precisely as COVID-19 is trending poorly in that state (even relative to the rest of the country), we also control for lagged values of the dependent variable in the prior 14 d. This approach flexibly controls for potentially nonlinear prior trends in the outcome of interest. We show, in SI Appendix, that the inclusion or exclusion of the lagged dependent variable does not affect the sign of our estimates but, in some cases, affects the magnitude, and we explain why we put more faith in the estimates that account for these lags.

This specification is depicted by the following equation:

where $D{V}_{it}$ is the outcome of interest in state i on day t, $SI{P}_{it}$ is our measure of the share of a state covered by SIP policies in state i on day t, ${\gamma }_i$ represents state fixed effects, ${\delta }_t$ represents day fixed effects, and the quantity of interest is $\beta$, a local average treatment effect of SIP policies.

To complement this analysis, we also test for lagged or decaying effects of SIP orders over time by including lagged values of the treatment variable from the prior 14 d. This allows us to estimate and visualize how the effects of the policy change over time,

In Eq. 2, ${\beta }_n$ corresponds to the effect of SIP policies n days ago. So, to estimate the effect of having SIP policies in place for two continuous days, for example, we would add ${\beta }_0$, ${\beta }_1$, and ${\beta }_2$. In SI Appendix, we show more detailed results where we include or exclude lagged values of the treatment, lagged values of the outcome, and leading values of the outcome.

Table 1 presents our results for our four main outcomes based on Eq. 1. We look at COVID-19 cases, COVID-19 deaths, mobility, and unemployment. Cases and deaths are coded as new cases or new deaths per million residents. Mobility is measured as the proportional change in distance traveled. Unemployment is the insured unemployment rate.

We find no evidence that SIP policies led to reductions in new COVID cases or deaths; indeed, the point estimates for both outcomes are positive but insignificant. We find that SIP policies did decrease mobility. The estimate implies that SIP orders decreased mobility, on average, by 0.7%. This effect is very small relative to the nationwide trend. Between late February and mid-April, mobility nationwide decreased by about 50%, so SIP orders explain only a tiny fraction of this general trend. The estimated effects of SIP orders on unemployment is positive but not statistically significant (although it is significant in some alternative specifications—see SI Appendix).

Our null results on the effects of SIP orders on COVID-19 cases are not easily attributable to imprecision. The lower bound of our 95% CIs from columns 1 and 2 in Table 1 suggest that we can statistically reject the possibility that SIP policies prevented 1.8 new cases per million residents per day or that they prevented 0.02 new deaths per million residents per day. If we multiply these lower-bound estimates with the population of each state and the number of days the policy was in effect in each state, they correspond with 32,112 cases and 372 deaths prevented nationwide during our period of study. Therefore, we can statistically reject the estimates from previous studies concluding that SIP policies saved many thousands of lives.

Fig. 2 graphically represents the results of our event study analyses based on the model in Eq. 2. For both cases and deaths, point estimates are generally positive but insignificant. While we would not expect a SIP order to affect either of these outcomes instantaneously, the results indicate that there are no declines even after the policy has been in place for 14 or more days.

Fig. 2 does show statistically significant evidence of small, negative effects of SIP orders on mobility in the days immediately after the policy is enacted. By the end of the first week, however, changes in mobility appear to have returned to trend. Finally, Fig. 2 provides some evidence that SIP orders lead to increases in unemployment when they have been in place for 10 or more continuous days.

Substantively, Fig. 2 suggests that having two continuous weeks of a SIP policy increases the insured unemployment rate by one percentage point. The average insured unemployment rate was 1.3% at the beginning of our study period in late February and early March, so a one percentage point effect represents a notable increase over that baseline. However, by early May, the average insured unemployment rate reached 13.9%, meaning the one percentage point effect we estimate represents only a small share of the nationwide increase in unemployment during the pandemic, consistent with other studies showing that SIP policies accounted for a relatively small share of COVID-related economic declines (6, 7).

SI Appendix, Tables S1–S4 show more detailed results for each of our four outcomes. Some specifications include leading values of the treatment in order to test whether SIP orders tended to be implemented at times when the outcomes were already trending differently. These coefficients are generally close to zero and statistically insignificant, and their inclusion does not meaningfully change the other coefficients, suggesting that preexisting trends do not meaningfully bias our estimates. Some specifications include or exclude lagged values of the outcome, which also do not meaningfully affect the other estimates. And some specifications include lagged values of the outcome variable, allowing us to test how the effects of SIP policies vary as more days pass. The detailed results bolster the findings in Table 1 and Fig. 2. SIP orders have no detectable effect on COVID-19 cases or deaths; a small, negative, short-lived effect on mobility; and a positive lagged effect on unemployment.

In SI Appendix, we also test additional hypotheses and explore the robustness of our results in various ways. We separately investigate the effects of expanding and withdrawing SIP orders, by focusing on different time periods. Interestingly, the introduction of SIP orders has bigger effects on mobility and unemployment than their withdrawal.

One concern with our empirical approach is that the effects of a SIP policy in one state could spill over to other states, biasing our estimates. Suppose, for example, that California’s SIP order affected behavior in Oregon. This could make the effect of California’s policy appear smaller than it really is, because, although we treat Oregon as a control state with respect to California, it was indirectly treated. In SI Appendix, we present several additional tests designed to address this concern. First, we implement our test only for smaller states, assuming that, to the extent that the effects of policies spill over to other states, this phenomenon will be more relevant for the policies of large states. And, to the extent that large states affected smaller states, this will be accounted for by the day fixed effects. Second, we implement our test only for states that implemented SIP orders later. Here, the idea is similarly that, to the extent there were spillover effects, they were probably more relevant for the early-acting states. Lastly, we explicitly estimate the extent to which having a SIP order in at least one neighboring state (as defined by a land border) may have affected behavior in that state over and above the state’s own policy. Although we do find some evidence that SIP policies affected mobility in neighboring states, the estimated effects on COVID-19 cases, mobility, and unemployment in that state are largely unchanged when we account for the possibility of spillovers between states.

SI Appendix also shows county-level estimates. Most of the variation in SIP orders comes from state policies, although some counties implemented their own orders before their states did so, and we obtain slightly more statistical precision by estimating effects at the county level. The estimates are extremely similar to those from our state-level analysis, although we do not examine unemployment, since weekly unemployment claims are only available at the state level.

We also account for changes in the extent of testing in various states. Specifically, we utilize information from Hsiang et al. (1) on the dates on which various states changed their testing regime, and, instead of including state fixed effects, we include state-by-testing-regime fixed effects. When we do this, the estimates are virtually unchanged.

SI Appendix also contains replications and extensions of two related studies attempting to estimate the effects of government policies on COVID-19 cases (1, 2). In both cases, we show that the estimated benefits of SIP policies and other similar government policies are highly sensitive to specification. Reassessing those designs and specifications leads us to conclude that previous studies have overestimated the effects of these policies. This helps to resolve an apparent contradiction in the literature. Since studies have shown that the effects of SIP policies on social distancing and mobility are small (5, 6), it would be surprising to find large effects on COVID-19 cases, and our analyses suggest that these effects are indeed small.

Although we estimate modest effects of SIP policies, our results should not be taken to imply that the actions of government officials had little effect on the pandemic. There may have been other policies that better mitigated the spread of COVID-19, although SIP orders have been arguably the most drastic and controversial policy. Furthermore, we observe nationwide trends in all outcomes, and these trends may have been highly responsive to the public health recommendations, emergency declarations, and the behaviors of high-profile politicians. Our results also do not mean that sheltering in place per se is an ineffective way to mitigate the spread of COVID-19. If SIP policies did not meaningfully increase the extent to which people actually sheltered in place or socially distanced, our results have nothing to say about the health and societal benefits of staying at home and reducing physical contact with others.

The explanation for our null findings is likely nuanced and multifaceted. One part of the explanation is that many people were already staying at home and social distancing voluntarily even in the absence of SIP policies. Another part of the explanation is, perhaps, that few people who weren’t already changing their behavior complied with the policies. After all, SIP orders appeared to cause less than a 1% decrease in mobility. There was, however, approximately a 50% decrease in mobility nationwide between February and April. The nationwide reaction to COVID-19 almost surely decreased the spread of the disease. SIP orders likely would have been more effective in slowing the spread had more people complied with them, and future SIP orders would likely be more effective if they are coupled with greater enforcement. But we find little evidence that SIP orders, as implemented, had much effect over and above all the other public messaging and voluntary behavior changes occurring nationwide. Although we find no detectable health benefits of SIP orders, we also find that they accounted for a small share of economic costs associated with the pandemic, consistent with other studies (6–9).

Our study is certainly not the last word on this topic. Assessing the effects of SIP orders is difficult, and more information and better designs may become available in the future that enable more precise or more credible estimates. Furthermore, our study focuses on the early months of the pandemic, and the effectiveness of SIP orders could change over time. However, the previously presented evidence on the effectiveness of SIP orders appears to be misleading, and there is currently no compelling evidence to suggest that SIP policies saved a large number of lives or significantly mitigated the spread of COVID-19. However, this does not mean that voluntary social distancing—SIP practice as distinct from policy—was ineffective.