The Lehrer effect

Ever since Jonah Lehrer identified the supposed flawed nature of the scientific method, and named it the decline effect, the Intertoobz has responded by either advancing the notion this proves science is just as unreliable as other manufactroversies, or by pointing out that what Lehrer sees as problematic actually is why science is reliable: it eventually filters out human bias.

Eventhough I was somewhat late in noticing the current kerfuffle others are slightly faster to comment on the notion that science does not work. Some observations I initially missed are by Kent Anderson, responding to "a study that scientists thought proved that female barn swallows preferred mating with males who had long, symmetrical feathers:"

In this case, the raw observations about feathers are generalized first to one species, then attempts are made to generalize them to other species, and soon the attempts start to fail. This isn’t an erosion of truth. It’s a failure of theorizing. The theory was derived after the fact, and to no useful end other than to publish more papers. It wasn’t a hypothesis that was tested, but a data set shoehorned into a post hoc theory.

The scientific method itself is revealing the limitations of initial findings. It’s working. But we’re so geared to create “headline science,” and so wrapped up in ego and pride that we’ve forgotten the humility we need to exhibit before the facts. But most importantly, we may have forgotten that something is even more important than facts — and that is theory.

Writing for Big Think Matthew C. Nisbet notes that:

the reaction that the article has stirred in some cases does not match the nuance of Lehrer's arguments. The article has been unfairly critiqued by some for giving ammunition to those already committed to extreme doubt about subjects such as climate change or evolution.  As Lehrer notes at his blog, he's also been accused of being a post-modernist, arguing that there is no such thing as truth or reality.

Criticism he does not share. In his view the article, and the ensuing debate, are an opportunity to bolster science education. The added benefit is that they:

are wonderful teaching tools for science students.

A column by John Allen Paulos, for ABC News, reviews the article and elaborates on the explanations for why the "decline effect" might occur:

A greater realization of these effects by journalists, scientists, and everyone else will lead to more caution in reporting results, more realistic expectations, and, I would guess, a decline in the decline affect (more accurately, the stat-psych effect).

The criticism his article generated has led Lehrer to write a response. Nonetheless, it appears he did not grasp what was said, or as Orac puts it:

Actually, what Lehrer's critics have been doing is anything but reassuring ourselves with platitudes about the rigors of replication. Indeed, all of us who bothered to write about Lehrer's article spent considerable time pointing out how regression to the mean, publication bias, and a variety of other factors that could explain much of the decline effect. We spent a lot of effort trying to explain how it is unsurprising that initial promising results often appear less so as more and more scientists investigate a question, developing along the way better techniques and approaches to investigating the question and approaching it from different angles. We spent a lot of verbiage describing how it is not at all unsurprising that new drugs, which seem to work so well in early clinical trials, appear to lose efficacy as their indication is broadened beyond the homogeneous initial small groups of subjects to more patients whose characteristics are less tightly controlled. Indeed, one of the letter writers pointed this very fact out to Lehrer, but he chose not to address this point directly.

The decline effect is something any physician who does clinical research knows from experience (although he may not call it that) because he sees it so often. To Lehrer it seemed to be some sort of shocking revelation in clinical research. The expectation that randomized clinical trials can overestimate the efficacy of new drugs is the very reason why, after drugs are released, physicians sometimes carry out what are known as "pragmatic trials," which are designed to find out how effective a treatment is in everyday, real-world practice, where the conditions are not nearly as controlled and the patient populations not nearly as homogeneous as they are in randomized clinical trials. Efficacy results determined in pragmatic trials are virtually always less robust than what was measured in the original randomized clinical trials. Not that any of this stops Lehrer from simply repeating the same stuff about big pharma having incentives to shape the results of its science and clinical trials. We get it; we get it. Science is done by humans, and sometimes human biases and motivations other than scientific discovery influence thee humans who do science.

He continues with:

More importantly, after discussing the decline effect and impugning the reliability of science, Lehrer still can't seem to give a coherent explanation as to why AGW and evolution are such reliable, well-founded scientific theories compared to what he seems to perceive as the unreliability of the rest of science. Worse, he hasn't addressed many of the more cogent criticisms of his work, in particular the numerous attempts to explain to him why it is not at all remarkable that second generation antipsychotics have not proven to be as effective as initial results suggested or why it is not particularly surprising or disturbing that fluctuating asymmetry never panned out. Lehrer had a great opportunity to explain why making scientific conclusions is so difficult and why all scientific knowledge is provisional. Those points are in his articles on the decline effect, but they're buried in the surrounding implication that the decline effect is mysterious. Then in the last paragraph of his response to critics Lehrer has the chutzpah to declare that "there is nothing inherently mysterious about why the scientific process occasionally fails or the decline effect occurs."

Someone else not overly impressed by Lehrer is David Weisman who strongly opines:

the 'decline effect' is bullshit. Science also ignores things for long periods of time, then recognizes their importance, at which point the theory grows, strengthens, branches off into new fields, and improves. Germ theory is hardly in decline, it grows stronger with every pneumonia. DNA as genetic instruction is also a theory, one that is hardly in decline. Ditto evolution. Ditto greenhouse gas. Ditto cells, tectonic plates, gravity, sodium channels, and atoms. None seem in danger of decline.

Lehrer wrote a faulty article. It got published. Initially it must have seemed almost reasonable. But on second look, it has fatal flaws. In fact, it appears to me that he cherry picked examples to support his own pet theory, a classic fallacy of the highly biased. Now it doesn't look very reasonable at all. It looks like small minded pseudoscience.

Another take on the phenomenon is given by Mike the Mad Biologist by referring to Andrew Gelman:

Gelman (and he has some good slides over at his post) is claiming, correctly, that if the effect is weak and you don't have enough samples (e.g., subjects enrolled in the study), any statistically significant result will be so much greater than what the biology would provide that it's probably spurious. You might get lucky and have a spurious result that points in the same direction as the real phenomenon, but that's just luck.

Then he points out:

So what someone will do is report the statistically significant result (since we tend to not report the insignificant ones). But further experiments, which often aren't well designed either, fail to pick up an effect. The ones that are well designed and have a large sample size will either identify a very weak real effect, leading to a consensus in the field of "Meh", or correctly fail to find a non-existent effect.
Sounds like the Decline Effect to me.

By claiming science is unable to provide any definite answers, i.e. The Truth, Lehrer apperently agrees with the anti-science movement which claims science is merely another opinion. Once everything is opinion how could any reasonable person (you know, the "fair and balanced"-type) object to dissenting views such as: evolution is not true, global warming does not exist, vaccines are evil, HIV is harmless, et cetera? This inadvertent support of denialism, through the law of unintended consequences, I propose we call the Lehrer effect. In other words, the Lehrer effect stands for the proposition that the orchestrated efforts of misinformation-central, in time inevitably will contaminate and debilitate even the protectors of reason, i.e. they too eventually come to believe that the scientific method is no more reliable than pseudoscience and denialism.

Update: Amended definition last sentence.