The Lehrer effect

Ever since Jonah Lehrer identified the supposed flawed nature of the scientific method, and named it the decline effect, the Intertoobz has responded by either advancing the notion this proves science is just as unreliable as other manufactroversies, or by pointing out that what Lehrer sees as problematic actually is why science is reliable: it eventually filters out human bias.

Eventhough I was somewhat late in noticing the current kerfuffle others are slightly faster to comment on the notion that science does not work. Some observations I initially missed are by Kent Anderson, responding to "a study that scientists thought proved that female barn swallows preferred mating with males who had long, symmetrical feathers:"

In this case, the raw observations about feathers are generalized first to one species, then attempts are made to generalize them to other species, and soon the attempts start to fail. This isn’t an erosion of truth. It’s a failure of theorizing. The theory was derived after the fact, and to no useful end other than to publish more papers. It wasn’t a hypothesis that was tested, but a data set shoehorned into a post hoc theory.

The scientific method itself is revealing the limitations of initial findings. It’s working. But we’re so geared to create “headline science,” and so wrapped up in ego and pride that we’ve forgotten the humility we need to exhibit before the facts. But most importantly, we may have forgotten that something is even more important than facts — and that is theory.

Writing for Big Think Matthew C. Nisbet notes that:

the reaction that the article has stirred in some cases does not match the nuance of Lehrer's arguments. The article has been unfairly critiqued by some for giving ammunition to those already committed to extreme doubt about subjects such as climate change or evolution.  As Lehrer notes at his blog, he's also been accused of being a post-modernist, arguing that there is no such thing as truth or reality.

Criticism he does not share. In his view the article, and the ensuing debate, are an opportunity to bolster science education. The added benefit is that they:

are wonderful teaching tools for science students.

A column by John Allen Paulos, for ABC News, reviews the article and elaborates on the explanations for why the "decline effect" might occur:

A greater realization of these effects by journalists, scientists, and everyone else will lead to more caution in reporting results, more realistic expectations, and, I would guess, a decline in the decline affect (more accurately, the stat-psych effect).

The criticism his article generated has led Lehrer to write a response. Nonetheless, it appears he did not grasp what was said, or as Orac puts it:

Actually, what Lehrer's critics have been doing is anything but reassuring ourselves with platitudes about the rigors of replication. Indeed, all of us who bothered to write about Lehrer's article spent considerable time pointing out how regression to the mean, publication bias, and a variety of other factors that could explain much of the decline effect. We spent a lot of effort trying to explain how it is unsurprising that initial promising results often appear less so as more and more scientists investigate a question, developing along the way better techniques and approaches to investigating the question and approaching it from different angles. We spent a lot of verbiage describing how it is not at all unsurprising that new drugs, which seem to work so well in early clinical trials, appear to lose efficacy as their indication is broadened beyond the homogeneous initial small groups of subjects to more patients whose characteristics are less tightly controlled. Indeed, one of the letter writers pointed this very fact out to Lehrer, but he chose not to address this point directly.

The decline effect is something any physician who does clinical research knows from experience (although he may not call it that) because he sees it so often. To Lehrer it seemed to be some sort of shocking revelation in clinical research. The expectation that randomized clinical trials can overestimate the efficacy of new drugs is the very reason why, after drugs are released, physicians sometimes carry out what are known as "pragmatic trials," which are designed to find out how effective a treatment is in everyday, real-world practice, where the conditions are not nearly as controlled and the patient populations not nearly as homogeneous as they are in randomized clinical trials. Efficacy results determined in pragmatic trials are virtually always less robust than what was measured in the original randomized clinical trials. Not that any of this stops Lehrer from simply repeating the same stuff about big pharma having incentives to shape the results of its science and clinical trials. We get it; we get it. Science is done by humans, and sometimes human biases and motivations other than scientific discovery influence thee humans who do science.

He continues with:

More importantly, after discussing the decline effect and impugning the reliability of science, Lehrer still can't seem to give a coherent explanation as to why AGW and evolution are such reliable, well-founded scientific theories compared to what he seems to perceive as the unreliability of the rest of science. Worse, he hasn't addressed many of the more cogent criticisms of his work, in particular the numerous attempts to explain to him why it is not at all remarkable that second generation antipsychotics have not proven to be as effective as initial results suggested or why it is not particularly surprising or disturbing that fluctuating asymmetry never panned out. Lehrer had a great opportunity to explain why making scientific conclusions is so difficult and why all scientific knowledge is provisional. Those points are in his articles on the decline effect, but they're buried in the surrounding implication that the decline effect is mysterious. Then in the last paragraph of his response to critics Lehrer has the chutzpah to declare that "there is nothing inherently mysterious about why the scientific process occasionally fails or the decline effect occurs."

Someone else not overly impressed by Lehrer is David Weisman who strongly opines:

the 'decline effect' is bullshit. Science also ignores things for long periods of time, then recognizes their importance, at which point the theory grows, strengthens, branches off into new fields, and improves. Germ theory is hardly in decline, it grows stronger with every pneumonia. DNA as genetic instruction is also a theory, one that is hardly in decline. Ditto evolution. Ditto greenhouse gas. Ditto cells, tectonic plates, gravity, sodium channels, and atoms. None seem in danger of decline.

Lehrer wrote a faulty article. It got published. Initially it must have seemed almost reasonable. But on second look, it has fatal flaws. In fact, it appears to me that he cherry picked examples to support his own pet theory, a classic fallacy of the highly biased. Now it doesn't look very reasonable at all. It looks like small minded pseudoscience.

Another take on the phenomenon is given by Mike the Mad Biologist by referring to Andrew Gelman:

Gelman (and he has some good slides over at his post) is claiming, correctly, that if the effect is weak and you don't have enough samples (e.g., subjects enrolled in the study), any statistically significant result will be so much greater than what the biology would provide that it's probably spurious. You might get lucky and have a spurious result that points in the same direction as the real phenomenon, but that's just luck.

Then he points out:

So what someone will do is report the statistically significant result (since we tend to not report the insignificant ones). But further experiments, which often aren't well designed either, fail to pick up an effect. The ones that are well designed and have a large sample size will either identify a very weak real effect, leading to a consensus in the field of "Meh", or correctly fail to find a non-existent effect.
Sounds like the Decline Effect to me.

By claiming science is unable to provide any definite answers, i.e. The Truth, Lehrer apperently agrees with the anti-science movement which claims science is merely another opinion. Once everything is opinion how could any reasonable person (you know, the "fair and balanced"-type) object to dissenting views such as: evolution is not true, global warming does not exist, vaccines are evil, HIV is harmless, et cetera? This inadvertent support of denialism, through the law of unintended consequences, I propose we call the Lehrer effect. In other words, the Lehrer effect stands for the proposition that the orchestrated efforts of misinformation-central, in time inevitably will contaminate and debilitate even the protectors of reason, i.e. they too eventually come to believe that the scientific method is no more reliable than pseudoscience and denialism.

Update: Amended definition last sentence.

Why the "decline effect" shows the scientific method works

Part of the struggle between science and ideology is the recurring theme that science too can be wrong. Note the number of times we have seen scientists retreating from their initial position to a more nuanced one. This transition to less pronounced statements causes the anti-science crowd to claim this proves science is not as reliable as we think. This phenomenon is discussed by David Gorski while responding to:

an article in The New Yorker by Jonah Lehrer entitled The Truth Wears Off: Is There Something Wrong With the Scientific Method?

The above concept is called the "decline effect" which, according to Gorski, stands for:

a phenomenon in which initial results from experiments or studies of a scientific question are highly impressive, but, over time, become less so as the same investigators and other investigators try to replicate the results, usually as a means of building on them. In fact, Googling “the decline effect” brought up an entry from The Skeptic’s Dictionary, in which the decline effect is described thusly:

The decline effect is the notion that psychics lose their powers under continued investigation. This idea is based on the observation that subjects who do significantly better than chance in early trials tend to do worse in later trials.

To him this observation is is neither shocking nor new:

In medicine, in particular, early reports tend to be smaller trials and experiments that, because of their size, tend to be more prone to false positive results. Such false positive results (or, perhaps, exaggerated results that appear more positive than they really are) generate enthusiasm, and more investigators pile on. There’s often a tendency to want to publish confirmatory papers early on (the “bandwagon effect”), which might further skew the literature too far towards the positive. Ultimately, larger, more rigorous studies are done, and these studies result in a “regression to the mean” of sorts, in which the newer studies fail to replicate the large effects seen in earlier results. This is nothing more than what we’ve been writing right here on SBM ever since its inception, namely that the normal course of clinical research is to start out with observations from smaller studies, which are inherently less reliable because they are small and thus more prone to false positives or exaggerated effect sizes

His conclusion:

Although Lehrer makes some good points, where he stumbles, from my perspective, is when he appears to conflate “truth” with science or, more properly, accept the idea that there are scientific “truths,” even going so far as to use the word in the title of his article. That is a profound misrepresentation of the nature of science, in which all “truths” are provisional and all “truths” are subject to revision based on evidence and experimentation. The decline effect–or, as Lehrer describes it the title of his article, the “truth wearing off”–is nothing more than science doing what science does: Correcting itself.

Commenting on this article Steven Novella writes that the term "decline effect:"

was first applied to the parapsychological literature, and was in fact proposed as a real phenomena of ESP – that ESP effects literally decline over time. Skeptics have criticized this view as magical thinking and hopelessly naive – Occam’s razor favors the conclusion that it is the flawed measurement of ESP, not ESP itself, that is declining over time.  Lehrer, however, applies this idea to all of science, not just parapsychology.

Just like Gorski he notes:

Lehrer is ultimately referring to aspects of science that skeptics have been pointing out for years (as a way of discerning science from pseudoscience), but Lehrer takes it to the nihilistic conclusion that it is difficult to prove anything, and that ultimately “we still have to choose what to believe.” Bollocks!

His explanation for this phenomenon is the same as Gorski's. The article was also noticed by Pharyngula who stated:

I read it. I was unimpressed with the overselling of the flaws in the science, but actually quite impressed with the article as an example of psychological manipulation.

And he then remarked:

Early in any scientific career, one should learn a couple of general rules: science is never about absolute certainty, and the absence of black & white binary results is not evidence against it; you don't get to choose what you want to believe, but instead only accept provisionally a result; and when you've got a positive result, the proper response is not to claim that you've proved something, but instead to focus more tightly, scrutinize more strictly, and test, test, test ever more deeply. It's unfortunate that Lehrer has tainted his story with all that unwarranted breast-beating, because as a summary of why science can be hard to do, and of the institutional flaws in doing science, it's quite good.

In short, the article erroneously points out the inherent characteristics of the scientific method as "proof" of why science is just another opinion. While provocative it identifies weakness but fails to recognise this actually is science's strongpoint.

As an side, this flowchart on how to debate the anti-science brigade (creationists actually) is brilliant:

(h/t Pharyngula)

Update: Following the "decline effect" there now is the Lehrer effect.