Alex Tabarrok points me today to a longish post about how badly we managed the COVID-19 vaccine rollout, especially highlighting this:
Probably the biggest mistake was not intentionally infecting vaccinated volunteers. This could be done in 1 month, vs 6.5 months for the ecological trials that the entire world did out of misguided PR ethics. (2.5 is probably more realistic given signups, approvals, and big pharma’s slow data analysis and reporting. That’s still hundreds of thousands of lives.)
In a "challenge trial," you don't vaccinate 50,000 people and then wait six months to see how they did compared to a different 50,000 who got a placebo. Instead, you round up a thousand people, vaccinate them, and then a month later deliberately infect them all. This gives you a very fast read on whether the vaccine is any good.
So why not do it? The typical answer is that it's unethical even though everyone in the study is a volunteer, but the flip side of this is that if it speeds up vaccine development by, say, five months, it's going to save hundreds of thousands of lives. Is it still unethical even with that kind of benefit in the balance?
While you ponder the ethics of this, I'll offer up a different reason to be skeptical of challenge trials: I'm not so sure they would have speeded up anything. What matters, after all, is not producing enough vaccine doses for testing, but producing billions of vaccine doses for mass distribution. And that takes time.
How much time? It's surprisingly hard to get a good answer to that. But it's a long process. You have to engineer the flow manufacturing process. You have to source raw materials. You have to license all the tech transfers you'll need. You have to physically set up the manufacturing floor. You have to debug the manufacturing. And then, finally, you can start producing vaccines in volume.
Whoops! Don't forget about distribution! The Pfizer vaccine required storage at super-low temperatures, which in turn required the creation of a massive new distribution system. How long did that take? Months, at least.
In the case of COVID-19 vaccines, this whole process began—
Well, the truth is that I don't know when it began. It's some kind of state secret or something. Roughly, though, it seems like it began about as soon as it could have, and even so actual production didn't begin until September 2020, with significant volumes available perhaps as early as October or November.
In other words, an earlier FDA approval via challenge trials might have shaved a month or two off the schedule if they went well, but not much more than that.
But wait. That's still thousands of lives that could have been saved. True enough. And we can now go back and look at every single mistake that was made and start adding up the lives that were lost because we're all idiots. But like all of them, my basic comment is this: Is the middle of a massive pandemic really the right time to start dicking around with processes that we know work, in favor of experimenting with a whole bunch of new things that might make things better? You can decide for yourself what you believe, but I'm skeptical. When the entire planet is in a panic, I think there's good reason to believe that it's precisely the time to fall back on well-known processes and not let hysteria drag your attention away to all the bright new ideas featured on the New York Times op-ed page.
Obviously, your mileage may vary. I tend to be naturally cautious about this kind of stuff. But in the end, I suspect that although challenge trials are an interesting idea, they would have (a) made little difference in the end and (b) produced data that was hard to interpret since it's never been done before. The speeded up version of standard testing that we used instead was probably for the best.
And idiots though we may be, we ended up putting a 95% efficacy vaccine into mass production within 11 months of startup. If you think that's a disgrace, you might simply be overestimating what H. sapiens is capable of.
POSTSCRIPT: Despite all this, I still recommend reading the linked blog post about vaccine development. Partly this is because even if you disagree about this and that, it's still useful to read an after-action report and this is a decent one.
But the main reason is that there's an art to the internet rant, and this is a pretty good rant. Regardless of whether it's right or wrong, we should have more like this.
And if anyone wants to complain, there's the Cuban vaccine which used all of the old methods. It's just gotten through testing.
Maybe we could have theoretically gotten a vaccine sooner - but that's in hindsight. We should look back and see what worked and what didn't and prepare for the future.
As someone who stopped reading Marginal Revolution b/c whats-his-name and Tabarrok were consistently libertarian trolls, it's pretty difficult to take anything from that guy seriously.
What this blog post is positing, is omniscience. I'm sorry, but I was awake in Spring 2020, and I *remember* reading the articles about the previous generation of SARS-1 vaccines: vaccinated test subjects (animals) had adverse effects in their lungs when challenged with the virus. That was the state of our knowledge. It would have been extremely dangerous to just run roughshod over all the standard practices, given that we had this data from the most-recent most-similar vaccine experiments.
More generally, there's a mania for "we should have done the *most* effective, *most* efficient thing". But neither science nor society work that way: both require ample amounts of buffering time, money, retries, in order to produce decent outcomes.
The real opprobrium here goes to our political leaders, for not organizing better and more severe lockdowns, and those free-rider assholes who chose to "live their lives" instead of doing their civic duty. Period. Period.
I could not agreement more with the points you’re making. The vaccines were approved just as fast as fast could be. As you’ve observed, in every instance the past failures and current difficulties are related to the approval of the vaccines but rather are failings of social organization or political will.
My prediction is that we will eventually have some kind of obligatory vaccination program and vaccine passports because they are obvious necessities without which it could be years or even decades before life returns to normal. This was obvious from the beginning.
I've always been puzzled by this kind of thinking:
"Is it still unethical even with that kind of benefit in the balance?"
Yes. 100% no doubt still unethical. Decide if you are willing to do something unethical and be honest about it. Nothing will or should alleviate your guilt but maybe you still did the right thing.
It's like in an actual ticking time bomb scenario. You torture the terrorists and stop the bomb? You get a medal. Then after you step off the platform to jail you go. You are a hero because you did what needed to be done AND faced the consequences. You don't get a pass.
Absolutely agree! "... if it speeds up vaccine development by, say, five months, it's going to save hundreds of thousands of lives," but if it doesn't work, you've just deliberately given thousands of people a horrible disease. How would that look in the history books?
As noted many times in many places, the incredibly rapid development of nearly unbelievably effective vaccines, in roughly one year from the discovery of the disease (for which there was never any guarantee there even could be a vaccine), is a freaking miracle of science and a nearly saint-like act of beneficence by the people who worked tirelessly to do it. Any Monday-morning well-maybe-they-shoulda-woulda-coulda is WAY out of line.
Also as noted elsewhere, any blame, any second-guessing, should fall squarely on the shoulders, heads, and any other exposed anatomy of the mostly Republican idiot politicians who dithered, delayed, tried to have it both ways, or -- in many cases, including The Former Guy -- overtly sabotaged efforts to control the spread of the disease while vaccines were being developed. They deserve a slow roasting.
I don't understand why no one else has made this obvious observation.
The proposals that were briefly mooted to do challenge trials involved volunteers who were young and healthy. By the time these discussions were being had, we knew enough about Covid-19 to know that if we enrolled 1,000 such people in the trial (which would have been all that were needed), probably none would die and only one or two would become sick enough to require hospitalization. Worst case scenario might have been one or two deaths and perhaps a half dozen hospitalizations. So, no, you would not have given thousands of people a horrible disease. You would have infected one thousand people with a disease that most likely would not have been horrible to any of them, or at worst, to a small number.
Now, there remains the question whether the results of those trials would have provided the answers we needed. In particular, these trials would not have enrolled older people or people with chronic illnesses. Since it was these two groups that were most in need of an effective vaccine, one would have to, as an act of faith, believe that the results of the challenge trials apply to these other groups as well. Since one aspect of biology that we know changes with age is immune function, that's a pretty big leap of faith.
The other question is how long (if at all) the process of getting people vaccinated was ultimately delayed as a result of not using challenge trials. Since vaccine manufacturing and the development of the manufacturing process were going on in parallel with the clinical trials, and since the former seems to have been the longer process, I'm guessing that we wouldn't have gained much if any time, anyhow.
But these are not moral arguments against challenge trials, they are pragmatic ones. I have a lot of difficulty grasping the moral basis for saying that if the circumstances made it possible to save hundreds of thousands, or possibly millions, of lives by exposing a modest number of volunteers to a small risk of serious illness or death, it would be wrong to do so.
"hat if we enrolled 1,000 such people in the trial (which would have been all that were needed), probably none would die and only one or two would become sick " These numbers are off by an order of magnitude for natural infections. But we have no idea what these would be for deliberately caused infections. Note the severity of the disease is largely drien by the initial virus load which would be completely different in this scenario. And Challenge trials would not require a thousand people, it would still require thousands instead of tens of thousands. But it would also take ten times as long to find volunteers since you've deliberately infecting people.
We live in a world where no-one takes volunteering seriously any more. We have yet to see the long term consequences of this.
Even during WW2 the concept of sacrificing yourself for your people (religion, nation, friends, family, whatever) was understood, accepted, revered. But the last gasp of this was probably Kennedy with the Peace Corp and "Ask not what your country can do for you"; by the time of Carter it was considered unacceptable that he ask the country to turn down the AC a little and reduce the highway speed.
We see this same trend in action movies. Action movies used to feature heroes who saved "us" in some sense, saved humanity, or America, or at least random strangers. We still have those, but alongside them we have the action movie where the "hero" has zero interest in helping anyone but his family (or girlfriend or whatever), anyone in the way of that is irrelevant. The apotheosis of this occurs in San Andreas where the "hero" is, for gods sake, the freaking HEAD OF SEARCH AND RESCUE for Los Angeles, but as soon as disaster occurs he abandons his post and takes off with a government helicopter (and then steal multiple vehicles) so that he can save his family.
And this is not only about "other people doing things for us". Something all major religions have understood is that you have to allow sinners a way to atone, to genuinely suffer so as to repair the consciences. One way to do that was to allow such people to engage in sacrifice, in the context of war, in the context of dangerous activity, and in the context of medical experimentation. But our best and brightest "ethicists" now consider this also to be wrong -- better to let a man suffer with his conscience for fifty years than to allow him to be healed as part of helping humanity! Dostoevsky's Grand Inquisitor, supposed to be the BAD GUY in the late 19th C, comes across as a marvel of ethical subtlety compared to our modern ethicists. We've replaced genuine ethical understanding with LARPING based on pointing fingers at a few (only a few, carefully chosen) incidents from the past, and we believe strenuous ritual denunciation of these incidents (with zero interest in context, or any other similar such incidents involving other actors) makes us better than any previous nation in history. JESUS CHRIST!!!
I don't know where this ends. But I don't think a nation running on this level of moral imbecility can survive long when genuine hardship arrives.
"We live in a world where no-one takes volunteering seriously any more. We have yet to see the long term consequences of this."
the people on which the covid vaccines were tested were in fact volunteers. i usually don't rely on superhero and dwayne johnson movies as accurate barometers of anything.
Doing challenge trials in healthy young volunteers may not have provided the data needed for a disease that hits older, less healthy adults much harder. To ensure valid results, both the test group and the control group must represent the actual population that is targeted by the vaccine.
Stage 1 trials, of course, are only conducted unhealthy volunteers in order to determine if a vaccine is safe. But later trials must be more representative of the whole population.
Further, the coronavirus was spreading so rapidly in the early days of the pandemic, it is difficult to understand why challenge trials would be needed. After all, there was no shortage of infected patients.
This kind of analysis belongs in a project retrospective meeting (and, unless the people running the pharmaceutical companies that developed the vaccines are idiots, that's exactly where it has happened/is happening). Engineers of all stripes do this, and physicians do the same kind of thing in their periodic mortality and morbidity conferences.
Where it definitely does not belong is in a speculative blog post. At best, it's just Monday morning quarterbacking. More likely, it exposes the author's naïveté regarding engineering and product development processes.
The tech details belong in a project retrospective.
But the cultural aspects, the moral/ethical choices made by society are not technical matters, and they do indeed deserve to be discussed EVERYWHERE.
In particular they need to be discussed in a wide variety of fora, allowing a wide variety of different opinions to be heard. The last thing we need is the credentialed class telling us that their 15% opinion, as pushed through the mainstream media, is the last word on the subject and we don't need dissident moral views being published on "speculative blog posts" and being entertained by the other 85% who, god forbid, might decide to come to a different ethical opinion.
Did massive manufacturing begin before the vaccines were tested? I think not. So testing was still a bottleneck.
Given the efficacy of the mRNA vaccines and the speed they were originally developed there should be several ways to speed things up next time. There might be more testing of the vaccine type in the absence of actual infection to eliminate safety concerns. If its safety can be guaranteed, then in the case of a fast-spreading virus with high fatality rate it might be unethical to keep anybody from getting the vaccine, even in tests. There should be sufficient data to evaluate efficacy as the vaccination process goes on.
It most certain did begin in summer 2021.
It most certain did begin in summer 2020.
Did massive manufacturing begin before the vaccines were tested?
Unequivocally yes. Vaccines were manufactured "at risk" in large quantities. (This information is available easily via a modern technology known as "Internet search.")
If its safety can be guaranteed...
And how is that to be accomplished absent extensive testing? Animal testing and phase I trials catch obvious and serious safety issues, but animal models are imperfect and phase I trials are conducted only in very small numbers of healthy volunteers. Later phases not only demonstrate efficacy but also amass safety data in sufficient volume to conclude in accordance with accepted standards that a medication is safe in the full patient population. So the premise, alas, is faulty.
Now there IS an extensive literature on other ways testing and approval might be structured, eg in phases, with a medication approved earlier for limited populations subject to more stringent monitoring, to get medicine to people who really need it a little earlier than at present, but there are many devils in the details, and of course the first time something would go wrong with such a process, the popular and political reaction would not be very receptive to the nuances of statistical analysis and balance of risks, so it's all still very much in the discussion stage.
+1
The way that you can tell that massive manufacturing was not present before the drugs were tested is that when the drugs were approved, we didn't suddenly see vaccines rolling out to every pharmacy and hospital in the country.
Most mRNA candidates worked well nor are ready yet. From the Wikipedia list, about 15 are still in Phase I-II trials. Not efficacy, but are they safe and is there some response.
https://en.wikipedia.org/wiki/COVID-19_vaccine
A comparison on development is here:
https://www.theatlantic.com/health/archive/2021/06/novavax-now-best-covid-19-vaccine/619276/
(ok a bit too much hype for Novavax)
It's not just about typing in a new sequence that will magically work. It's also about translating a sequence so it can be expressed well enough in target cells to elicit an immune response. You have to know what you're doing and be very lucky.
Phase 3 trials started around the end of July 2020. The EUA was granted in mid-December. Phase 3 finished recruitment in mid-January 2021. The EUA was based on surprisingly good interim data at a predetermined checkpoint. Anyone who has worked in drug development knows that the noise to signal ratio is immense, and that it takes careful clinical trials and rigorous statistical analysis to tell if one's vaccine candidate is actually a working vaccine. To get a sense of how crazy this is, consider that there are a number of consulting houses that specialize in figuring out how big a batch of your candidate you'll need to brew up to run your trials. That's all they do. And, don't think of just running off another batch if you run out. You'd probably learn just as much by switching to distilled water, and we know more about distilled water than your candidate.
There's a lot of Monday morning quarterbacking because, like great quarterbacks, Pfizer and Moderna made it look easy. They went from unknown disease to effective vaccine roll out in just over a year which is pretty amazing, but they were only able to do this because they were professionals. They've been working on mRNA technology for twenty years. They had a practice run with SARS. They went by the book and didn't take short cuts. Most of this was invisible, and that made the stuff that was visible look easy.
Challenge trials make for good press, but they don't give the data that one needs. It's not like people were 100% sure of just how COVID spread. What was the typical dose, the dose duration, how far up one's nose was vulnerable and so on? Look at how many trials and doses it took to just get a sense of the dosing strategy for phase 3. How many challenge trials was it going to take to cover all the probable exposure types and come up with a protocol with some informative power?
They were suppose to have begun long before trials were complete, beginning around June.
The production of vaccine at decent scale was not possible until January 2021. Pfizer, for example, has indicated they plan to produce 2.5 to 3 billion doses in 2021 and 3 billion in 2022. That’s the best they could do. Ask them. What would faster look like?
It’s slightly off topic but faster would look like governments (which basically paid for the research anyway) allowing other countries to manufacture the vaccine themselves for free and with governments contracting with manufacturers to inexpensively make huge volumes of the vaccines rather than giving private companies the intellectual property rights to stuff we paid to research and then paying those companies even more money to buy the vaccines that the taxpayer funded or subsidized research made possible.
Setting aside the cost issues, I think it’s clear these companies are already doing that to add production volume.
But the difference is that if the government had kept the intellectual property rights to the vaccines rather than essentially gifting them to the pharmaceutical companies, there would be an immense amount of manufacturing going on all over the world, and particularly in Africa where the need to give huge windfall profits to pharmaceutical companies is limiting the production of vaccines. I say that in the future, if government money or subsidies are involved we should keep the rights to the vaccines.
Since no one has ever manufactured a mRNA vaccine before, and all of the underlying technology was patented years ago, you're ides are completely false for the Moderna and BioNtech vaccines. Further AstraZenneca did license it's vaccines to an Indian company and offered to license it to any third world vaccine maker that had the ability to manufacturer it. Note, Johnson and Johnson, even with a license given to Merk still isn't capable of manufacturing it's vaccine.
For example…
https://www.biospace.com/article/exelead-helps-manufacture-pfizer-biontech-covid-19-vaccine/
The US government contributed nothing to the development of the BioNTech vaccine. Pfizer took all of the risk on themselves.
Faster would look like making 6 billion doses in 2021. Faster would be setting up 10 production lines around the world in parallel. Expensive? You bet. Each production line would be used for about 1/5th of the time that the current lines will be used for. Faster would also be starting to set up those ten lines back in April of 2020, gambling that the vaccine would be proven effective
I'm in the phase III Moderna trial. I enrolled in August of 2020 and got the booster in September. I was 65 at the time, and was one demographic they were particularly looking for. I wouldn't have done it as a challenge trial.
RNA vaccines are bad. They have been developed for years before this branch of Coronavirus came to be. Real vaccines like Jansen's take time and it was just the first.
Biden should ban RNA vaccines and heavily push the Jansen/Novavax.
Indeed. He should. Now that 100 million plus Americans have gotten it.
If it were The Former Guy, he would, but only because he secretly had gone long on J&J and shorted Pfizer.
Lol ignorant nonsense.
What's next? They're rewriting your DNA? They magnetize you? They contain nanochips that will allow George Soros and Bill Gates to seize control of your brain?
Bottom's up, Shootie!
Could you explain to me why these vaccines are bad?
Ignorant, evidence-free trolling.
Please be quiet now. Adults are talking.
You're ignoring the fact that no one is presently capable of manufacturing the Jansen vaccine in volume.
One of the issues with approval, you need to follow people over time just to make sure nothing odd shows up. In this case, there were combined I/II trials, and what would traditionally be considered early starts to phase III. Follow up studies indicate they could have done better at dosing, as KD mentioned a while ago, but they were pushing to make approvals at the time based on limited trial data. Any challenge trial would have been phase III, and that probably would not have moved things along much faster, if at all. Indeed, it probably would have gummed up the works. The problem with a "challenge" test is that you don't know how to present the challenge in a real life manner. Fine aerosols or swabs? It makes a big difference.
Satellite records ground temperature of 145 degrees F in Wenatchee, Washington,
https://gizmodo.com/ground-temperatures-reached-an-astounding-145-degrees-i-1847193318
"Is the middle of a massive pandemic really the right time to start dicking around with processes that we know work, in favor of experimenting with a whole bunch of new things that might make things better? "
Putin dicked around to get the Russian vaccine out fast, and now few Russians want to get it. Trust is essential if you want a large proportion of the population to get the vaccine.
The challenge trial is idiotic - one reason for a 6 month trial vs placebo is to make sure there aren't lethal or horrendous side effects, that will only become apparent over a few months.
The linked blog post fails to mention the single biggest impediment to mass production of vaccines; patents. Instead, he criticizes the EU for failing to pay a price 1000% (or more) over the production costs based on a farcical analysis of the "true value". Pfizer claims that it would be impossible for a generic manufacturer to set up the supply chain required to produce the vaccine, but if so, putting the patents in the public domain wouldn't have hurt them at all. The basic research leading to mRNA vaccines was all done on the government's dime. And in Moderna's and Oxford's cases, governments picked up all of the R&D costs as well. Of course, nobody except Dean Baker would call this corruption, and Dr Baker is a persona non grata in the MSM, basically a crazy guy in sack cloth preaching out in the web wilderness.
While I have no ethical qualms about informed volunteers participating in challenge trials, I doubt that challenge trials would have sped things up much in this case.
One of the things you really want to know about prospective COVID vaccines is how they work in real-world conditions, and I don't think we knew enough to replicate those conditions in an artificial situation. In particular, you'd want volunteers to be exposed to a viral load typical of real-world conditions, and I don't think we know even now what that is.
Get the viral load too high, and even a very effective vaccine might be overwhelmed and appear ineffective. Get it too low, and an ineffective vaccine might look like it works well.
This is very different from a prospective malaria vaccine, where we have a better idea of the typical exposure: one feeding from a hungry, infected mosquito. For prospective malaria vaccines, challenge trials can simulate real-world conditions very well.
The other thing about challenge trials for e.g. a malaria vaccine is that we have very effective treatments for malaria, so if the vaccine doesn't work we don't just have to watch the patient die, or recover from his own internal resources. We don't know how to cure covid.
It's easy to second guess the procedures that were employed when we have vaccines that we now know are more that 95% effective. If it turned out that that they were less than 50% effective (as we've seen with one or two vaccines that were not approved), a challenge trial would have resulted in more people dying *as a result of the trial*. If that had occurred, we would all be armchair-quarterbacking why the challenge trial was approved.
As it is, our vaccine development worked very well, probably better than could have been expected.
The idea that you could do challenge trials in a month is completely ridiculous. It would take a month just to round up the doctors to partipate in the trial let alone the volunteers. And it would take months to round up volunteers. (It's not rounding up a thousand, you need thousands getting the placebo and thousands getting the real vaccine, since you still need to get data on the side effects.) And then more than a month to analyze the data. Giving the ease of getting volunteers and the extremely high infection rate last fall, it would have saved weeks. (Though that wouldn't have been clear to December of 2020.
As late as November both moderna and Pfizer were expecting the trials not to finish til January). And as far as how long would it take to ramp up vaccine manufacturing, Johnson & Johnson still hasn't achieved that despite their vaccine being licensed in January and cutting a deal with Merk (the number one vaccine manufacturer in the world) to produce these vaccines.
> The Pfizer vaccine required storage at super-low temperatures, which in turn required the creation of a massive new distribution system. How long did that take? Months, at least.
You need to provide a peer-reviewed reference for that. Not only did low-temp distribution systems exist before 2020, but also you could buy them from FedEx et al.
https://www.pharmalogisticsiq.com/packaging-shipping-systems/articles/guide-to-temperature-controlled-logistics
is a 2018 document that discusses the low-temperature distribution system.
> we ended up putting a 95% efficacy vaccine into mass production within 11 months of startup
Not really. In 11 months we had two manufacturing lines up and running. Production volumes continued to scale up in the U.S. for another four months. And that still wasn't enough production to significantly vaccinate most of the world.
There are two big takeaways from 2020: first, once the prototype manufacturing is set up to run the testing, we need to be able to scale up those proof-of-concept manufacturing lines at 10 times the scale. Second, the test protocol shouldn't be for a 95% effective vaccine using two doses 3 weeks apart; it should be for a single dose immediately with a second booster dose 6 months down the road. The 65% effective vaccines are fine because they 100% reduce the nastiness of catching the virus.
Fortunately, it looks like we might get a lot of practice more quickly developing, testing, and distributing vaccines.