It's sort of a truism in the social sciences that nothing works. More specifically, interventions meant to improve things almost never actually do. Megan Stevenson, an economist and criminal justice scholar at the University of Virginia law school, says this about it:
This claim will not be controversial to anyone immersed in the literature. But, like a dirty secret, it almost never gets seriously acknowledged or discussed. Nor is it widely known beyond the small circle of people trained in statistical methods of causal inference. The research that people hear about shows the rare cases of success; the remainder gets filtered from public view.
That's from a newly published article where Stevenson makes two claims about interventions that initially seem successful:
- If you test them with a randomized controlled trial, the gold standard of experimental assessments, they almost always fail.
- If they succeed, they almost always fail when someone tries to replicate them.
Stevenson is focused on the criminal justice space, though she believes her conclusion is probably true for almost all social interventions. The problem is that very often initial successful results get a lot of attention but subsequent failures don't. For example:
In 2017, an article showed that a program teaching youths to “think slow” instead of impulsively responding led to a substantial decline in violent arrests and gains in high school graduation rates.... However, a follow-up article shows that this effect was mostly prevalent in the earliest cohort analyzed: effects for subsequent cohorts were close to zero and statistically insignificant.
....Note that this disappointing follow-up result would be hard to discover. While the original success was published in economics’ most prestigious journal and received widespread media attention, the subsequent failure to replicate is mentioned only tangentially in the back pages of an unpublished working paper on a different topic.
Is Stevenson right that nothing works? I'd say almost certainly—though note that this is only the case for social interventions, not things like vaccines or environmental changes.
Also worth a note: Stevenson points out that plenty of simple, direct interventions work fine. If you give people food, they'll be less hungry. The failures arise for less obvious, more indirect questions. For example, does having more food affect graduation rates or make people less likely to steal stuff?
In any case, Stevenson's claim doesn't have a lot of political valence: it's true for both liberal and conservative interventions. However, by its nature it applies mostly to small-bore interventions, since those are the ones that can be tested. So the good news is that there's still plenty of reason to believe that big, revolutionary changes have plenty of effect.
No, she's almost certainly wrong.
Here's an easy counter example: sex education is a social intervention and it works at keeping rates of teen pregnancy and STD infection down.
https://www.aap.org/en/patient-care/adolescent-sexual-health/equitable-access-to-sexual-and-reproductive-health-care-for-all-youth/the-importance-of-access-to-comprehensive-sex-education/
The "think slow" example sure looks like a weakass example to grind an entire axe with.
BOOM.
This is what I came here to say. Given girls and women sex ed and free contraception and abortion and there will be many fewer kids with bad outcomes.
Your solution is also the best way to combat global warming, as shown by what is now a fairly large body of studies.
She didn't say that "nothing works"; she was far more nuanced than that.
On a similar vein, it is beyond doubt that the amyloid beta (Aβ) theory was false. Hundreds of millions (billions?) of dollars were spent chasing Aβ reduction. And yet, it took over a decade of Aβ-targeted drug failures for the science community to finally reject (I think?) Aβ reduction as a mechanism to treat ALZ.
The social sciences aren't the only people who are resistant to acknowledging problems of cognitive biases in research and publication.
Just look at how the FDA granted approval for a failed ALZ drug targeting Aβ reduction despite evidence showing that it had no statistically significant effect.
".. reject (I think?) Aβ reduction as a mechanism to treat ALZ."
That is just wrong.
https://en.wikipedia.org/wiki/Anti-amyloid_drugs
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10195708/
The effects of these are not that impressive, and they have serious side effects, so it is not obvious how useful they are. But they certainly still trying this approach.
Isn't it that they're saying it's not the amyloid that's the issue but the waste removal system, and that in combination with another element that I am not recalling?
They are saying a lot of things. They don't have a clear answer yet.
Read Lowe: https://www.science.org/content/blog-post/faked-beta-amyloid-data-what-does-it-mean and https://media.nature.com/original/magazine-assets/d41586-018-05719-4/d41586-018-05719-4.pdf
While Aβ is clearly involved in the process, it is not in itself the cause of cognitive decline. At best, anti-Aβ drugs slow decline by a few months, not years. How can it be that drug after drug effectively clearing Aβ does not result in the clear retardation of Aβ?
How can that be? Well, here's one that makes a lot of sense.
Correlation but not causation. Targeting reduction of Aβ has, in one drug trial, resulted in acceleration of cognitive decline. None of this makes sense unless you accept that Aβ is not the causation agent.
Did you read the references I gave?
" How can it be that drug after drug effectively clearing Aβ .."
There is no drug sthat effectively clears Aβ. They try, with varying levels of success.
The current drugs are claimed to slow the disease, though the numbers are not that convincing. They were enough to convince the FDA , though (see my references above).
If the study is focused mainly on behavior, I get that younger cohorts would benefit from early interventions. For older cohorts, interventions probably followed with sustained maintenance would be required.
Is the study really just proving you cannot teach an old dog new tricks?
At the beginning of Kevin’s post is this quote from Megan Stevenson:
“ like a dirty secret, it almost never gets seriously acknowledged or discussed.”
Apparently, she needs to read the Wikipedia article on the Hawthorne Effect.
Public education is functioning social intervention, as is Social Security, as are gun regulations, etc
I'd also add that too many of these social "science" "experiments" are crap, and the grandiose conclusions drawn from sketchy results, which play so well in the click-driven media, are unfounded. This view would seem to be borne out by the experiments' lack of repeatability.
Making some massive social change on the basis of something that appeared to have been a one-fer is not smart. In my opinion. Note that I'm biased, having been educated in the physical (that is, actual) sciences.
Honestly this is true of pretty much all science. Any excited news story about a new study is almost always pure hogwash.
true. Splashy journals like sexy articles...with the "boring" follow-ups found in less impactful journals. Only really big things get fully covered, e.g. cold fusion or the ever popular "water carries a memory".
Remember when the universe was 26 billion years old, not 13?
Talk about click bait! The title of the post just reeks of it!
Like most people, I only read the headline. Was there more to the story than that?
"Also worth a note: Stevenson points out that plenty of simple, direct interventions work fine. If you give people food, they'll be less hungry."
Even something as simple sounding as this is probably not true.
What kind of food are you providing? When and where? How are you deciding who to give the food to? Are you giving people food they actually want? I recall well walking around a homeless mission in San Jose around lunch time and seeing baloney sandwiches all over the sidewalk because even the homeless didn't want to eat that trash.
Maybe I'm going off topic, but give people money and let them buy the things they want (food, clothes, beer, whatever).
"Even something as simple sounding as this is probably not true."
You're just wrong about this.
In your sandwich example, you're missing the point that if even 1 person ate the sandwich and was less hungry than they otherwise would have been, it had an effect. What you're really saying is that it may have been an inefficient means of addressing a need, but inefficient and ineffective are NOT the same thing. An intervention can be inefficient and effective, just as it can be efficient and ineffective. Two completely different dimensions of measurement that you are conflating.
However, you do arrive at the holy grail answer: if people need shit, just give them money and let them solve their own problems. It really is that simple.
As with many such things, it helps for Americans to look beyond their borders. In other countries, "give them money" is a much more common poverty-relief strategy than it is in the US. American poverty relief tends to assume that poor people can't manage money, and so they are given a pastiche of food stamps, subsidized housing, Medicaid, school lunches, etc.
It would be helpful for serious social science researchers to look at other countries to see how effective "give them money" is as a poverty relief strategy compared to the American approach.
American poverty relief tends to assume that
poorthose people can't manage money, and so they are given a pastiche of food stamps, subsidized housing, Medicaid, school lunches, etc.FIFY.
Gee, I wonder if anybody's done any of those.
https://www.npr.org/sections/goatsandsoda/2023/12/07/1217478771/its-one-of-the-biggest-experiments-in-fighting-global-poverty-now-the-results-ar
That's definitely indicative, but I was thinking in terms of comparing American poverty-relief programs to those in other wealthy countries.
It's likely true that "small bore" interventions in the social sciences have little effect. They are, after all, typically trying to work against very large social factors.
We seem culturally disposed toward looking for silver bullets and "one weird tricks," but change is actually very hard without long-term differences in inputs. Social science experiments just don't get to change the inputs for the long term, nor do they get to change all the inputs - people still live in the wider culture.
Given the demands of publish or perish and the emphasis placed on the results of small pilot programs before wider rollouts, I'm not sure how we change things so that research is more accurately reflecting real results. We may be stuck with the status quo. At least it seems unlikely that the interventions involved will cause actual harm.
We seem culturally disposed toward looking for silver bullets and "one weird tricks," but change is actually very hard without long-term differences in inputs.
Culturally speaking, long after the low-hanging fruit has been plucked we still insist on searching there for our missing car keys. Sorry, couldn't resist.
It is not necessarily that research results are "invalid". I would suggest that they are more often limited in scope.
That is, the number of small-scale interventions that will have large impacts is probably rather small. As noted, they are generally working against a lot of much larger scale social factors.
But also important is that, if such small-scale interventions are indeed effective in some given case, it is likely that they are also highly dependent upon the specific conditions of that case, most of which cannot even be adequately described, let alone controlled for. Thus, it should not be surprising that when the same intervention is attempted in a different case, with different conditions, it turns out to be less effective, or even ineffective. After all, if it was able to have large effects due to the specific conditions in which it was tested, then there is no reason to think that the same effects will result in different conditions.
Amazing how everything conveniently fits the worldview of the anti-tax "government should do absolutely nothing but maintain an army and maybe pave roads" crowd.
Also amazing that somehow "nothing works" in social science research, and yet if you plagiarize any of that apparently-meaningless research, you're worthy of being hounded out of your job.
African American misleaders like Thomas Sowell and John McWhorter rely on this bad science.
That is suspicious. "The lady doth protest too much, methinks"
It's too easy to argue that we shouldn't even try. It's at the heart of Silicon bro flavored effective altruism. There's also lots of stuff we've seen that works like higher marginal tax rates on extremely large incomes, vigorous antitrust enforcement and high estate taxes.
I Googled the Finnish experiment with a test of guaranteed basic income given to 2000 unemployed. Seems to produce a small increase in employment. The articles I read did not discuss other aspects or the unemployed lives. Finland has a pretty good social security safety net so maybe that result is to be expected.
I'm a fan of a principle often stated by the longtime teachers' union leader Albert Shanker: "Every educational experiment is doomed to succeed." The idea is that the first implementation of a reform is always carried out by people with great enthusiasm and commitment to the new approach. Until the results can be replicated by those less invested in the success of the innovation, it cannot be considered an improvement.
High schools can get grants to implement and/or study new teaching methodologies. Of course if they need that money, that means they'll be changing methodologies every few years. It's not about what works, but about what brings in the money.
He was the head of my mother's union, the UFT. He's much missed.
I don’t know about other stuff, but I do know that the “criminal justice space” is filled with “social interventions” that don’t work. The one I remember from my days working in juvenile hall in Los Angeles in the late ‘70s was “Scared Straight” where kids would get brow beaten by ex-cons and thus see how bad it would be for them in prison. It was all the rage for a while but in reality it didn’t have any long term effect.
Kevin’s “lead crime hypothesis” was, in my opinion what actually did work.
Part of the problem is that people are looking for magic bullets that will work right now--hence "scared straight" program. What's needed is long term investment in communities and a number of programs for the kids and families in general. More lead clean up and not just a warning notice that the place you live in might have lead paint.
>> Part of the problem is that people are looking for magic bullets that will work right now<<
This is the problem with so many things, isn’t it?
Waste in government spending? Across the board 10% cut!
Unauthorized immigration? Build a wall!
Too much crime? More police and harsher punishment!
Etc.
Right?! There needs to be intervention with sustained maintenance because people live...
There's no such thing as a magic bullet because there's no magic!
[clap children, clap to save Tinkerbell, it's working...oh you saved her!]
I'd amend that to say that people are looking for magic bullets that will work right now and further, don't cost them any money.
Lately, there seems to be an increasing emphasis on "bullets" -- the non-magical kind. A number of my acquaintances were all hopped up on a couple of local incidents of shoplifting, in which the alleged perps were shot. Dead. Yippee.
I find it hard to believe that "Grand Theft T-shirt" ought to be a capital offense.
There's an old saying that has considerable truth to it. "For every problem there exists a solution that is neat, plausible, simple and wrong." The fact that there are lots of wrong solutions does not prove that there are no right ones.
I would bet that MADD (founded in 1980) worked - of course their interventions were not only limited to raising awareness of the issue to change the culture around drinking and driving, but lobbying for legal changes and enforcement, but as someone who grew up driving as a teenager in the 70s I remember how different things were back then.
The demand for drugs and black market drugs belies the assertion interventions do not work. The interventions of the ruling class to increase subjection for producing surpluses do not work very well, while the solutions discovered by their subjects to cope with the demands of a market society and alleviate stress do provide some relief. Although these solutions may only be temporary, debilitating, and deemed anti-social by the institutions tasked with coercing the working class to transfer labor to produce surpluses for the rich, they should be recognized as effective. Arthur Sackler did.
Physiological anthropology
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9999635/
A message of hope for the new year!
Nothin' really rocks and nothin' really rolls and nothin's ever worth the cost!
I think of this often in the context of mindfulness and meditation, which your "think slow" reference is related to.
A Buddhist speaker I follow, Ajahn Sumedho, openly admits that meditation won't make you a better person, and that Buddhist monasteries can be hives of bitterness and backbiting.
I meditate anyway, if only an hour or two a week, and enjoy the effects, possibly placebo, of not being ruled by my knee-jerk reactions for at least part of my day.
There is something about confronting the meaninglessness and insignificance of one's own daydream thoughts, which usually rule us.
Turns out that all of our thoughts are garbage.
Big, revolutionary changes do indeed have an effect, but it isn't necessarily a positive one, e.g., Communism. Unfortunately, absent controlled experimentation in advance, it's hard to know whether a change will be positive or negative.
Rolls eyes. Communism is evolutionary, not revolutionary. But you just had to get that dig in, didn't you?
A program teaching youths the importance of adhering to the scientific principle will cause them to think slow instead of responding impulsively, and it will lead to a substantial decline in violent arrests and gains in high school graduation rates.
A program teaching youths to think slow instead of responding impulsively will cause them to think slow until they stop thinking slow because it has little to no positive effect.
In other words, it depends on what the intervention is trying to accomplish.
To reiterate, and as per Wikipedia's scientific method article, the scientific principle involves careful observation, applying rigorous skepticism about what is observed, given that cognitive assumptions can distort how one interprets the observation. By default, being rigorously skeptical is thinking slow, but a person thinking slow is not necessarily adhering to the scientific principle.
In 1950, according to Toyota's self-assessment, their workers took 9 hours to produce what a worker at one of the big three American manufacturers produced in one hour. Also in 1950, a program teaching Japanese businesses the importance of adhering to the scientific principle beginning when the US government sent people like Edwards Deming to Japan.
It worked. While the big three were counting beans, Toyota was becoming their worst nightmare, and Deming became the central figure in a 1980 NBC documentary called "If Japan Can, Why Can't We?"
Pingback: Does DEI even work? – Kevin Drum
This sounds like a redescription (rediscovery?) of what Rittel and Webber called "wicked problems" in their now 50 year old paper "Dilemmas in a General Theory of Planning":
https://urbanpolicy.net/wp-content/uploads/2012/11/Rittel+Webber_1973_PolicySciences4-2.pdf
These are, par excellence, problems having to do with indirect causes and effects, and the very first characteristic of them that Rittel and Webber call out is that "the process of solving the problem is identical with the process of understanding its nature."
It seems to me that what you're saying is what I would say using different language. I'd say we must distinguish between deductive, inductive, and abductive reasoning.
If your conclusion is that you understand the relationship between observable phenomena that appear to be related, then you're using inductive reasoning.
If your conclusion is that you understand the relationship between observable phenomena that appear to be unrelated, then you're using abductive reasoning.
If you're rigorously skeptical of the assumptions underlying your conclusion, along with the evidence supporting those assumptions, then you're using deductive reasoning.
With inductive and abductive reasoning, you start with evidence and end with a conclusion. With deductive reasoning, you start with a conclusion and end with evidence.
The only question to ask is, "Am I willing to constrain my own ignorance (by challenging my own assumptions)?" And the people that do not answer "yes" have gone to the dark side.
Pingback: New Study Raises Important Point – How Helpful Is Ed Research? - Education