Neurovault: a must-use tool for every neuroimaging paper!

Something that has long irked me about cognitive neuroscience is the way we share our data. I still remember the very first time I opened a brain imaging paper and was struck dumbfounded by the practice of listing activation results in endless p-value tables and selective 2D snapshots. How could anyone make sense of data this way? Now having several years experience creating such papers, I am only more dumbfounded that we continue to present our data in this way. What purpose can be served by taking a beautiful 3-dimensional result and filtering it through an awkward foci ‘photoshoot’? While there are some standards you can use to improve the 2D presentation of 3D brain maps, for example showing only peak activation and including glass-brains, this is an imperfect solution – ultimately the best way to assess the topology of a result is by directly examining the full 3D result.

Just imagine how improved every fMRI paper would be, if instead of a 20+ row table and selective snapshot, results were displayed in a simple 3D viewing widget right in the paper? Readers could assess the underlying effects at whatever statistical threshold they feel is most appropriate, and PDF versions could be printed at a particular coordinate and threshold specified by the author. Reviewers and readers alike could get a much fuller idea of the data, and meta-analysis would be vastly improved by the extensive uploading of well-categorized contrast images. More-over, all this can be easily achieved while excluding worries about privacy and intellectual property, using only group-level contrast images, which are inherently without identifying features and contain only those effects included in the published manuscript!

Now imagine my surprise when I learned that thanks to Chris Gorgolewksi and colleagues, all of this is already possible! Chris pioneered the development of neurovault.org, an extremely easy to use data sharing site backed by the International Neuroinformatics Coordinating Facility. To use it, researchers simply need to create a new ‘collection’ for their study and then upload whatever images they like. Within about 15 minutes I was able to upload both the T- and contrast-images from my group level analysis, complete with as little or as much meta-data as I felt like including. Collections can be easily linked to paper DOIs and marked as in-review, published, etc. Collections and entries can be edited or added to at any time, and the facilities allow quick documentation of imaging data at any desired level, from entire raw imaging datasets to condition-specific group contrast images. Better still, neurovault seamlessly displays these images on a 3D MNI standard brain with flexible options for thresholding, and through a hookup to neurosynth.org can even seamlessly find meta-analytic feature loadings for your images! Check out these t-map display and feature loadings for the stimulus intensity contrast for my upcoming somatosensory oddball paper, which correctly identified the modality of stimulation!

T-map in the neurovault viewer.
T-map in the neurovault viewer.
Decoded features for my contrast image.
Decoded features for my contrast image, with accurate detection of stimulation modality!

Neurovault.org doesn’t yet support embedding the viewer, but it is easy to imagine that with collaboration from publishers, future versions could be embedded directly within HTML full-text for imaging papers. For now, the site provides the perfect solution for researchers looking to make their data available to others and to more fully present their results, simply by providing supplementary links either to the neurovault collection or directly to individual viewer results. This is a tool that everyone in cognitive neuroscience should be using – I fully intend to do so in all future papers!

We the Kardashians are Democratizing Science

I had a good laugh this weekend at a paper published to Genome Biology. Neil Hall, the author of the paper and well-established Liverpool biologist, writes that in the brave new era of social media, there “is a danger that this form of communication is gaining too high a value and that we are losing sight of key metrics of scientific value, such as citation indices.” Wow, what a punchline! According to Neil, we’re in danger of forgetting that tweets and blogposts are, according to him, the worthless gossip of academia. After all, who reads Nature and Science these days?? I know so many colleagues getting big grants and tenure track jobs just over their tweets! Never mind that Neil himself has about 11 papers published in Nature journals – or perhaps we are left to sympathize with the poor, untweeted author? Outside of bitter sarcasm, the article is a fun bit of satire, and I’d like to think charitably that it was aimed not only at ‘altmetrics’, but at the metric enterprise in general. Still, I agree totally with Kathryn Clancy that the joke fails insofar as it seems to be ‘punching down’ at those of us with less established CVs than Neil, who take to social media in order to network and advance our own fledgling research profiles. I think it also belies a critical misapprehension of how social media fits into the research ecosystem common among established scholars. This sentiment is expressed rather precisely by Neil when discussing his Kardashian index:

The Kardashian Index
The Kardashian Index

“In an age dominated by the cult of celebrity we, as scientists, need to protect ourselves from mindlessly lauding shallow popularity and take an informed and critical view of the value we place on the opinion of our peers. Social media makes it very easy for people to build a seemingly impressive persona by essentially ‘shouting louder’ than others. Having an opinion on something does not make one an expert.”

So there you have it. Twitter equals shallow popularity. Never mind the endless possibilities of having seamless networked interactions with peers from around the world. Never mind sharing the latest results, discussing them, and branching these interactions into blog posts that themselves evolve into papers. Forget entirely that without this infosphere of interaction, we’d be left totally at the whims of Impact Factor to find interesting papers among the thousands published daily. What it’s really all about is building a “seemingly impressive persona” by “shouting louder than others”. What then does constitute effective scientific output, Neil? The answer it seems – more high impact papers:

“I propose that all scientists calculate their own K-index on an annual basis and include it in their Twitter profile. Not only does this help others decide how much weight they should give to someone’s 140 character wisdom, it can also be an incentive – if your K-index gets above 5, then it’s time to get off Twitter and write those papers.”

Well then, I’m glad we covered that. I’m sure there were many scientists or scholars out there who amid the endless cycle of insane job pressure, publish or perish horse-racing, and blood feuding for grants thought, ‘gee I’d better just stop this publishing thing entirely and tweet instead’. And likewise, I’m sure every young scientist looks at ‘Kardashians’ and thinks, ‘hey I’d better suspend all critical thinking, forget all my training, and believe everything this person says’. I hope you can feel me rolling my eyes.  Seriously though – this represents a fundamental and common misunderstanding of the point of all this faffing about on the internet. Followers, impact, and notoriety are all poorly understood side-effects of this process; they are neither the means nor goal. And never mind those less concrete (and misleading) contributions like freely shared code, data, or thoughts – the point here is to blather and gossip!

While a (sorta) funny joke, it is this point that is done the most disservice by Neil’s article. We (the Kardashians) are democratizing science. We are filtering the literally unending deluge of papers to try and find the most outrageous, the most interesting, and the most forgotten, so that they can see the light of day beyond wherever they were published and forgotten. We seek these papers to generate discussion and to garner attention where it is needed most. We are the academy’s newest, first line of defense, contextualizing results when the media runs wild with them. We tweet often because there is a lot to tweet, and we gain followers because the things we tweet are interesting. And we do all of this without the comfort of a lofty CV or high impact track record, with little concrete assurance that it will even benefit us, all while still trying to produce the standard signs of success. And it may not seem like it now – but in time it will be clear that what we do is just as much a part of the scientific process as those lofty Nature papers. Are we perfect? No. Do we sometimes fall victim to sensationalism or crowd mentality? Of course – we are only fallible human beings, trying to find and create utility within a new frontier. We may not be the filter science deserves – but we are the one it needs. Wear your Kardshian index with pride.

#MethodsWeDontReport – brief thought on Jason Mitchell versus the replicators

This morning Jason Mitchell self-published an interesting essay espousing his views on why replication attempts are essentially worthless. At first I was merely interested by the fact that what would obviously become a topic of heated debate was self-published, rather than going through the long slog of a traditional academic medium. Score one for self publication, I suppose. Jason’s argument is essentially that null results don’t yield anything of value and that we should be improving the way science is conducted and reported rather than publicising our nulls. I found particularly interesting his short example list of things that he sees as critical to experimental results which nevertheless go unreported:

These experimental events, and countless more like them, go unreported in our method section for the simple fact that they are part of the shared, tacit know-how of competent researchers in my field; we also fail to report that the experimenters wore clothes and refrained from smoking throughout the session.  Someone without full possession of such know-how—perhaps because he is globally incompetent, or new to science, or even just new to neuroimaging specifically—could well be expected to bungle one or more of these important, yet unstated, experimental details.

While I don’t agree with the overall logic or conclusion of Jason’s argument (I particularly like Chris Said’s Bayesian response), I do think it raises some important or at least interesting points for discussion. For example, I agree that there is loads of potentially important stuff that goes on in the lab, particularly with human subjects and large scanners, that isn’t reported. I’m not sure to what extent that stuff can or should be reported, and I think that’s one of the interesting and under-examined topics in the larger debate. I tend to lean towards the stance that we should report just about anything we can – but of course publication pressures and tacit norms means most of it won’t be published. And probably at least some of it doesn’t need to be? But which things exactly? And how do we go about reporting stuff like how we respond to random participant questions regarding our hypothesis?

To find out, I’d love to see a list of things you can’t or don’t regularly report using the #methodswedontreport hashtag. Quite a few are starting to show up- most are funny or outright snarky (as seems to be the general mood of the response to Jason’s post), but I think a few are pretty common lab occurrences and are even though provoking in terms of their potentially serious experimental side-effects. Surely we don’t want to report all of these ‘tacit’ skills in our burgeoning method sections; the question is which ones need to be reported, and why are they important in the first place?

Birth of a New School: PDF version and Scribus Template!

As promised, today we are releasing a copy-edited PDF of my “Birth of a New School” essay, as well as a Scribus template that anyone can use to quickly create their own professional quality PDF manuscripts. Apologies for the lengthy delay, as i’ve been in the middle of a move to the UK. We hope folks will iterate and optimize these templates for a variety of purposes, especially post-publication peer review, commentary, pre-registration, and more. Special thanks to collaborator Kate Mills, who used Scribus to create the initial layout. You might notice we deliberately styled the manuscript around the format of one of those Big Sexy Journals (see if you can guess which one). I’ve heard this elaborate process should cost somewhere in the tens of thousands of dollars per article, so I guess I owe Kate a few lunches! Seriously though, the entire copy-editing and formatting process only took about 3 or 4 hours total (most of which was just getting used to the Scribus interface), less than the time you would spend formatting and reformatting your article for a traditional publisher. With a little practice Scribus or similar tools can be used to quickly turn out a variety of high quality article types.

Here is the article on Figshare, and the direct download link:

Screen Shot 2013-12-12 at 11.50.42
The formatted manuscript. Easy!

What do you think? Personally, I’m really pleased with it! We’ve also gone ahead and uploaded the Scribus template to Figshare. You can use this to easily publish your own post-publication peer reviews, commentaries, and whatever else you like. Just copy-paste your own text into the text fields, replace the images, upload to Figshare or a similiar service, and you are good to go! In general Scribus is a really awesome open source tool for publishing, both easy to learn and cross platform. Another great alternative is Fidus. For now we’re still not exactly sure how to generate citations – in theory if you format your manuscripts according to these guidelines, Google Scholar will pick them up anywhere on the net and generate alerts. For now we are recommending everyone upload their self-publications to Figshare or a similar service, who are already working on a streamlined citation generation scheme. We hope you find these useful; now go out and publish some research!

The template:

An easy to use Scribus template for self-publishing
Our Scribus template, for quick creation of research proofs.