Why empiricism will always be imperfect

A new year is upon us, and that’s always a great time to clean out the skeletons in your closet.  So without further ado, let’s take a look at Jonah Lehrer’s explanation of “the decline effect” (published in The New Yorker last month). Lehrer describes this odd phenomenon whereby statistical significance of previous scientific findings seems to decrease with age, as we get further and further away from the time that it was initially reported in literature.

As any scientist can tell you, the holy grail of an experiment is a low p-value, a statistical measure that tells whether your findings are indicative of an actual effect, not just randomness and chance. This sounds fairly straightforward – of course we want to find things of actual importance, rather than being lulled into a false discovery by arbitrary data – but it turns out to be  much hazier than a simple “yes” or “no.” P-values depend on a number of factors that can change the statistical outcome of your experiment. Things like experimental design, subject choice, even the time of day can have drastic effects on the results of an experiment.

Scientists’ answer to such imperfections is to run the experiment over and over in a number of different environments. This is the beauty of scientific empiricism; at its best, it has the ability to extract truth from the noisy world around us. However, as Lehrer notes, there is one variable that we never change: the fact that people are the ones running these studies. This statement may seem annoyingly obvious, but it’s incredibly important to consider for any scientific study. While the empirical process is designed to provide an objective method of analyzing data, humans are inherently imperfect at being objective and unbiased, and this can manifest itself in the conclusions we take from our studies.

Suppose that you run a study with 90 subjects. The first 80 subjects show a fantastic result. You eagerly begin working on your forthcoming journal article, ready to share your findings with the world. However, upon running the final 10 subjects, you find that this result almost totally disappears. Bummer. An objective machine might say “maybe there isn’t anything here after all” and move on.  But people aren’t objective, and they’ve got a stake in giving the world something that is deemed significant. So you decide to leave out those last few subjects, citing them as outliers and thus non-representative of the general population, and publish an article. A few years pass, and a number of researchers (less invested in your discovery) decide to take another look at that paper.  They replicate your experimental design, but they fail to replicate your stellar results.

I don’t mean to cast a shadow of doom and gloom over the scientific enterprise. I just want to remind everyone that as long as human beings carry out scientific work, human faults will continue to plague our results. If we hope to come to an understanding of the world around us, it is important that we accept and anticipate the flaws inherent in our system.

Leave a Reply

6 comments

  1. Being objective is hard for us because most of us have one track minds

  2. Pingback: Berkeley Science Review Blog

  3. Anna Goldstein

    I like the idea of publishing negative results, if we could figure out a practical way to do it… This reminds me of an argument I had once about Ken Jennings on Jeopardy. Jennings won 74 consecutive games before losing- clearly a spectacular player. But my claim is that there could have been other contestants who were just as skilled as him (or more), but their “negative result” happened to come on the first or second game, instead of the 75th.

  4. Check this out Anna:

    http://www.jnrbm.com/

    It looks like people across many disciplines are realizing the importance of negative results…now if only there were a better way to get other people to look at them.

    @Hania, did you happen to host a radio show at Berkeley called The Graduate once upon a time?

  5. Pingback: The Berkeley Science Review » Dead salmon finds home. Still dead.