“We are not interested in the logic itself, nor will we argue for replacing the .05 alpha with another level of alpha, but at this point in our discussion we only wish to emphasize that dichotomous significance testing has no ontological basis. That is, we want to underscore that, surely, God loves the .06 nearly as much as the .05. Can there be any doubt that God views the strength of evidence for or against the null as a fairly continuous function of the magnitude of p?”

Rosnow, R.L. & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276-1284.

This colorful quote came to mind while discussing significance testing procedures with colleagues over lunch. In Cognitive Neuroscience, with it’s enormous boon of obfuscated data, it seems we are so often met with these kinds of seemingly absurd, yet important statistical decisions. Should one correct p-values over the lifetime, as often suggested by our resident methodology expert? I love this suggestion; imagine an academia where the fossilized experts (no offense experts) are tossed aside for the newest and greenest researchers whose pool of p-values remains untapped!

Really though, just how many a priori anatomical hypothesis should one have sealed up in envelopes? As one colleague joked, it seems advantageous to keep a drawer full of wild speculations sealed away in case one’s whole-brain analysis fails to yield results. Of course we must observe and follow best scientific and statistical procedures to their maximum, but in truth a researcher often finds themselves at these obscure impasses, thousands of dollars in scanning funding spent, trying to decide whether or not they predicted a given region’s involvement. In these circumstances, it has even been argued that there is a certain ethical need to explore one’s data and not merely throw away all non-hypothesis fitting findings. While I do not support this claim, I believe it is worth considering. And further, I believe that a vast majority of the field, from the top institutions to the most obscure, often dip into these murky ethical realms.

This is one area I hope “data-driven” science, as in the Human Genome and Human Connectome projects, can succeed. It also points to a desperate need for publishing reform; surely what matters is not how many blobs fall on one side of an arbitrary distinction, but rather a full and accurate depiction of one’s data and it’s implications. In a perfect world, we would not need to obscure the truth hidden in these massive datasets while we hunt for sufficiently low p-values.

Rather we should publish a clear record, showing exactly what was done, what correlated with what, and also where significance and non-significance lie. Perhaps we might one day dream of combing through such datasets, actually explaining what drove the .06′s vs the .05′s. For now however, we must be careful not to look at our uncorrected statistical maps; for that way surely voodoo lie! And that is perhaps the greatest puzzle of all; two datasets, all things being equal. In one case the researcher writes down on paper, “blobs A, B, and C I shall see” and conducts significant ROI analyses on these regions. In the other he first examines the uncorrected map, notices blobs A, B, and C, and then conducts a region of interest analysis. In both cases, the results and data are the same. And yet one is classic statistical voodoo- double dipping- and the other perfectly valid hypothesis testing. It seems thus that our truth criterion lay not only with our statistics, but also in some way, in the epistemological ether.

Of course, it’s really more of a pragmatic distinction than an ontological one. The voodoo distinction serves not to delineate true from false results but rather to discourage researchers from engaging in risky practices that could inflate the risk of false-positives. All-in-all, I agree with Dorothy Bishop: we need to stop chasing the novel, typically spurious and begin to share and investigate our data in ways that create lasting, informative truths. The brain is simply too complex and expensive an object of study to let these practices build into an inevitable file-drawer of doom. It infuriates me how frustratingly obtuse many published studies are, even in top journals, regarding the precise methods and analysis that went into the paper. Wouldn’t we all rather share our data, and help explain it cohesively? I dread the coming collision between the undoubtably monolithic iceberg of unpublished negative findings, spurious positive findings, and our most trusted brain mapping paradigms.