The Truth About Lockdowns: Refuting the ‘No Benefit’ Study

This transcript has been edited for clarity.

Welcome to Impact Factor, your weekly dose of commentary on a new medical study. I'm Dr F. Perry Wilson of the Yale School of Medicine.

Despite COVID-19 death rates holding relatively steady, there is a palpable sense of pandemic optimism in the US and abroad. The Omicron wave, the largest of the pandemic, has, in most of the world, peaked and is now receding.

There is broad hope that the immunity engendered by that wave, combined with that engendered by vaccines, may bring us to the end of the pandemic.

And with that hope, a slew of changes in governmental policies regarding COVID — with even liberal coastal governors like my own Ned Lamont in Connecticut starting to rescind mask mandates and other mitigation policies.

As the pandemic ebbs, it is natural and appropriate to look back on the decisions we made along the way and ask: Were they correct? Did we do the right thing?

A new "working paper" made a huge splash last week by claiming that, when it comes to lockdowns, we made a terrible mistake. The paper comes from three Johns Hopkins economists and is a meta-analysis of other studies examining the effect of lockdown policies on mortality. Meta-analyses are useful tools — combining the estimates of various studies into one overall estimate — allowing the vagaries of study design and the vicissitudes of chance to wash out via averaging and, ideally, produce an estimate of effect that is something closer to the truth.

So is their overall finding, that lockdowns do essentially nothing to reduce deaths from COVID, close to the truth? It was good enough for the media. Fox News ran multiple segments on the study, on the news shows as well as on Hannity. It garnered an editorial in the New York Post and Newsweek as well.

If true, the implications are twofold. First, the individuals who were pushing hard against government restrictions during COVID would get to drink a big glass of I-told-you-so, but more important, when the next pandemic occurs — and it will — we can handle it differently.

But is it true?

Is it possible that government restrictions have no effect on COVID mortality? That would go against the whole pandemic playbook, not just this pandemic — the sort of basic tenets of infectious disease public health. So there may be a problem here.

Before we get too deep, we need to talk about this term — "lockdown." What does that even mean?

At first blush, when I think "lockdown," I think "shelter in place": all but essential businesses closed and all of the population told to remain in their homes. This is not what the authors mean when they use the term "lockdown." They state, "Lockdowns are defined as the imposition of at least one compulsory, non-pharmaceutical intervention." So, closing schools? Lockdown. International travel ban? Lockdown. Mask mandate? Lockdown. Seems a bit hyperbolic to me, but there it is.

Okay, back to why this meta-analysis might not be telling us the whole truth. The magic of a meta-analysis arises from the studies you put into it, so let's start with the included studies. Figuring out the impact of lockdowns on COVID mortality is not straightforward. There are a myriad of approaches to try to answer the question, but they fall into a few categories.

First, you can do a modeling study. The idea here is to take the epidemic curve you observe before lockdown and predict where it would go, assuming that the conditions on the ground didn't change. Then you observe where the curve actually did go after the lockdown was instituted. The difference between predicted and observed is an estimate of lockdown effectiveness. A study in Nature showing the effect of lockdowns during the first wave in Europe used that approach. You can see here how the predicted deaths assuming no lockdown (in blue) were substantially higher than those observed once lockdowns were put in place.

This study does not appear in the meta-analysis. In fact, all studies that used a modeling approach were excluded.

You can try to get at the lockdown-mortality effect by using a before-and-after study. Here, you simply look at the death rate before lockdown and after lockdown and compare them. Of course, you need to account for other stuff that changes with time, including the epidemic curve, but good epidemiologists are pretty good at that. This is a PLOS Medicine paper that looks at COVID cases and mortality before and after lockdowns by state in the US.

Even after adjusting for things like the size of the pandemic when the restrictions were put in place, the results show a significant decrease in death rates once lockdown occurs. This study does not appear in the meta-analysis. In fact, all studies using a before-and-after approach were excluded.

You can take what's known as a difference-in-difference approach. Look at two, hopefully similar, locations over time where one institutes some sort of lockdown and the other doesn't. Then watch their respective death rates. This is the only type of study allowed in this meta-analysis. There are some good difference-in-difference studies and some bad ones. Epidemiology is hard.

In any case, after poring through 18,590 potentially eligible studies, the authors narrow the field to 34 studies for review. Twenty-four of those were included in the meta-analysis. Of those 24, 12 are preprints.

Some of the studies looked at the effect of shelter-in-place orders — the only thing that really says "lockdown" to me. Overall, the authors estimate that shelter-in-place orders decreased COVID mortality by 2.9%. That works out to around 27,000 people in the US, which isn't nothing, but it would not be described as a pandemic game changer.

The problem is that there are plenty of studies that estimate that shelter-in-place orders have a much stronger effect. The meta-analysis contains studies, like the Sears study here, that show a greater than 30% reduction in deaths, but they are given very little weight in the analysis compared to one study, Alderman, which shows a 1% reduction in death rate.

The authors describe how they create the weights for each study but they don't seem appropriate to me. In fact, the Sears study, which showed the 30% reduction in deaths, is an analysis of data from all 50 states. So is the study. Why is the Alderman study given 28 times the weight in the meta-analysis?

You see the same effect of weighting in another analysis looking at studies that used the Oxford "stringency index." The stringency index is an ordinal scale where higher values mean tighter restrictions — from recommendations to social distance, to full shelter-in-place orders.

One study in that analysis, from the CDC, showed that higher stringency was associated with lower death rates.

The authors report that the CDC study implies a 35% reduction in mortality when comparing a stringency level similar to what the US had with a lower level of stringency more consistent with advice to be careful. Now compare the weight of that CDC study, 11, with the weight of this study by Dr Carolyn Chisadza, 7390.

That huge weight, assigned to one study, drives the entire stringency score–based analysis. Given that many of these studies use national and even international databases, it seems unlikely that one study should be given so much higher importance.

And, just to mention it, the author of that study — apparently — disagreed with the meta-analysts' interpretation:

Turning back to the shelter-in-place–order analysis, studies in the meta-analysis are also given a quality score. The authors show that higher-quality studies show less effect of shelter-in-place on mortality. For example, the four studies with only 2 out of 4 quality points show a 34% reduction in deaths, whereas the four studies that got all 4 quality points show just a 1% decrease in deaths.

There are several good instruments for scoring the quality of medical studies. The 4-point system the meta-analysis authors used is not one of them.

Rather, you get a point if your paper was peer-reviewed, a point if your study ended after May 31, a point if your study shows no effect on mortality in the first 14 days, and a point if your study is written by a social scientist as opposed to an epidemiologist or other scientist. What?

With the exception of the peer-review thing, these quality metrics do not make sense. In fact, they seem designed to penalize studies that show a mortality effect of lockdown.

Now, I've been focusing a lot on shelter-in-place orders, but remember that the authors define lockdown as any nonpharmacologic mandate. And they do provide an analysis of mortality prevention stratified by various mandates. It looks like this:

As you can see, these authors find that mask mandates reduce COVID deaths by more than 20%, and that closing bars and restaurants reduce deaths by more than 10%. These findings were not reported by the media. The narrative around this study solidified firmly around "lockdowns have no effect."

Okay, here's the thing. Determining the effect of government policies on COVID mortality is really hard. There have been a lot of papers, a lot of attempts at figuring this out — so many, in fact, that you could write a meta-analysis like the one we're discussing today that says there are no real effects, or like this one, which argues that the effects are quite profound.

Personally, I think the clearest argument for the efficacy of government mandates, at least in terms of curbing COVID mortality, is China, which has experienced 3.5 COVID deaths per million population compared with 2800 in the US. Obviously, I would never accept the extremity of lockdown that China's authoritarian regime has imposed, and I doubt that my fellow American citizens would either. But from a purely scientific standpoint, it's hard to argue that lockdown — in the extreme, at least — doesn't work.

The problem in the liberal democracies is that the infection rate, government mandates, and personal choices are all interrelated.

People might be more careful about masking or social distancing because case rates are high. They might be more careful because the mere fact that a government mandate exists makes them take the virus more seriously. And, of course, they might be careful because they are forced to be careful. It can be hard to tease that all apart.

When the next pandemic occurs, this question will come up again. How much should governments do to curb the spread? It's another one of those public health outbreak tenets that you would rather do too much than too little. The problem, of course, is that if you do too much, no one will ever know how bad it could have been. And if you do too little, well, people die. It's a lose-lose from a PR standpoint. Criticism is inevitable. Of course, that's all part of the job.

Special thanks to @gidmk for advice regarding this article.

F. Perry Wilson, MD, MSCE, is an associate professor of medicine and director of Yale's Clinical and Translational Research Accelerator. His science communication work can be found in the Huffington Post, on NPR, and here on Medscape. He tweets @fperrywilson and hosts a repository of his communication work at www.methodsman.com.

Follow Medscape on Facebook, Twitter, Instagram, and YouTube

Comments

Commenting is limited to medical professionals. To comment please Log-in.

Comments on Medscape are moderated and should be professional in tone and on topic. You must declare any conflicts of interest related to your comments and responses. Please see our Commenting Guide for further information. We reserve the right to remove posts at our sole discretion.