The_Good_The_Bad_and_The_Ugly

The Wild West of Publication Reform Is Now

It’s been a while since I’ve tried out my publication reform revolutionary hat (it comes in red!), but tonight as I was winding down I came across a post I simply could not resist. Titled “Post-publication peer review and the problem of privilege” by evolutionary ecologist Stephen Heard, the post argues that we should be cautious of post-publication review schemes insofar as they may bring about a new era of privilege in research consumption. Stephen writes:

“The packaging of papers into conventional journals, following pre-publication peer review, provides an important but under-recognized service: a signalling system that conveys information about quality and breath of relevance. I know, for instance, that I’ll be interested in almost any paper in The American Naturalist*. That the paper was judged (by peer reviewers and editors) suitable for that journal tells me two things: that it’s very good, and that it has broad implications beyond its particular topic (so I might want to read it even if it isn’t exactly in my own sub-sub-discipline). Take away that peer-review-provided signalling, and what’s left? A firehose of undifferentiated preprints, thousands of them, that are all equal candidates for my limited reading time (such that it exists). I can’t read them all (nobody can), so I have just two options: identify things to read by keyword alerts (which work only if very narrowly focused**), or identify them by author alerts. In other words, in the absence of other signals, I’ll read papers authored by people who I already know write interesting and important papers.”

In a nutshell, Stephen turns the entire argument for PPPR and publishing reform on its head. High impact[1] journals don’t represent elitism; rather they provide the no name rising young scientist a chance to have their work read and cited. This argument really made me pause for a second as it represents the polar opposite of almost my entire worldview on the scientific game and academic publishing. In my view, top-tier journals represent an entrenched system of elitism masquerading as meritocracy. They make arbitrary, journalistic decisions that exert intense power over career advancement. If anything the self-publication revolution represents the ability of a ‘nobody’ to shake the field with a powerful argument or study.

Needless to say I was at first shocked to see this argument supported by a number of other scientists on Twitter, who felt that it represented “everything wrong with the anti-journal rhetoric” spouted by loons such as myself. But then I remembered that in fact this is a version of an argument I hear almost weekly when similar discussions come up with colleagues. Ever since I wrote my pie-in-the sky self-publishing manifesto (don’t call it a manifesto!), I’ve been subjected (and rightly so!) to a kind of trial-by-peers as a de facto representative of the ‘revolution’. Most recently I was even cornered at a holiday party by a large and intimidating physicist who yelled at me that I was naïve and that “my system” would never work, for almost the exact reasons raised in Stephen’s post. So lets take a look at what these common worries are.

The Filter Problem

Bar none the first, most common complaint I hear when talking about various forms of publication reform is the “filter problem”. Stephen describes the fear quite succinctly; how will we ever find the stuff worth reading when the data deluge hits? How can we sort the wheat from the chaff, if journals don’t do it for us?

I used to take this problem seriously, and try to dream up all kinds of neato reddit-like schemes to solve it. But the truth is, it just represents a way of thinking that is rapidly becoming irrelevant. Journal based indexing isn’t a useful way to find papers. It is one signal in a sea of information and it isn’t at all clear what it actually represents. I feel like people who worry about the filter bubble tend to be more senior scientists who already struggle to keep up with the literature. For one thing, science is marked by an incessant march towards specialization. The notion that hundreds of people must read and cite our work for it to be meaningful is largely poppycock. The average paper is mostly technical, incremental, and obvious in nature. This is absolutely fine and necessary – not everything can be ground breaking and even the breakthroughs must be vetted in projects that are by definition less so. For the average paper then, being regularly cited by 20-50 people is damn good and likely represents the total target audience in that topic area. If you network to those people using social media and traditional conferences, it really isn’t hard to get your paper in their hands.

Moreover, the truly ground breaking stuff will find its audience no matter where it is published. We solve the filter problem every single day, by publically sharing and discussing papers that interest us. Arguing that we need journals to solve this problem ignores the fact that they obscure good papers behind meaningless brands, and more importantly, that scientists are perfectly capable of identifying excellent papers from content alone. You can smell a relevant paper from a mile away – regardless of where it is published! We don’t need to wait for some pie in the sky centralised service to solve this ‘problem’ (although someday once the dust settles i’m sure such things will be useful). Just go out and read some papers that interest you! Follow some interesting people on twitter. Develop a professional network worth having! And don’t buy into the idea that the whole world must read your paper for it to be worth it.

The Privilege Problem 

Ok, so lets say you agree with me to this point. Using some combination of email, social media, alerts, and RSS you feel fully capable of finding relevant stuff for your research (I do!). But your worried about this brave new world where people archive any old rubbish they like and embittered post-docs descend to sneer gleefully at it from the dark recesses of pubpeer. Won’t the new system be subject to favouritism, cults of personality, and the privilege of the elite? As Stephen says, isn’t it likely that popular persons will have their papers reviewed and promoted and all the rest will fade to the back?

The answer is yes and no. As I’ve said many times, there is no utopia. We can and must fight for a better system, but cheaters will always find away[2]. No matter how much transparency and rigor we implement, someone is going to find a loophole. And the oldest of all loopholes is good old human-corruption and hero worship. I’ve personally advocated for a day when data, code, and interpretation are all separate publishable, citable items that each contribute to ones CV. In this brave new world PPPRs would be performed by ‘review cliques’ who build up their reputation as reliable reviewers by consistently giving high marks to science objects that go on to garner acclaim, are rarely retracted, and perform well on various meta-analytic robustness indices (reproducibility, transparency, documentation, novelty, etc). They won’t replace or supplant pre-publication peer review. Rather we can ‘let a million flowers bloom’. I am all for a continuum of rigor, ranging from preregistered, confirmatory research with pre and post peer review, to fully exploratory, data driven science that is simply uploaded to a repository with a ‘use at your peril’ warning’. We don’t need to pit one reform tool against another; the brave new world will be a hybrid mixture of every tool we have at our disposal. Such a system would be massively transparent, but of course not perfect. We’d gain a cornucopia of new metrics by which to weight and reward scientists, but assuredly some clever folks would benefit more than others. We need to be ready when that day comes, aware of whatever pitfalls may bely our brave new science.

Welcome to the Wild West

Honestly though, all this kind of talk is just pointless. We all have our own opinions of what will be the best way to do science, or what will happen. For my own part I am sure some version of this sci-fi depiction is inevitable. But it doesn’t matter because the revolution is here, it’s now, it’s changing the way we consume and produce science right before our very eyes. Every day a new preprint lands on twitter with a massive splash. Just last week in my own field of cognitive neuroscience a preprint on problems in cluster inference for fMRI rocked the field, threatening to undermine thousands of existing papers while generating heated discussion in the majority of labs around the world. The week before that #cingulategate erupted when PNAS published a paper which was met with instant outcry and roundly debunked by an incredibly series of thorough post-publication reviews. A multitude of high-profile fraud cases have been exposed, and careers ended, via anonymous comments on pubpeer. People are out there, right now finding and sharing papers, discussing the ones that matter, and arguing about the ones that don’t. The future is now and we have almost no idea what shape it is taking, who the players are, or what it means for the future of funding and training. We need to stop acting like this is some fantasy future 10 years from now; we have entered the wild west and it is time to discuss what that means for science.

Authors note: In case it isn’t clear, i’m quite glad that Stephen raised the important issue of privilege. I am sure that there are problems to be rooted out and discussed along these lines, particularly in terms of the way PPPR and filtering is accomplished now in our wild west. What I object to is the idea that the future will look like it does now; we must imagine a future where science is radically improved!

[1] I’m not sure if Stephen meant high impact as I don’t know the IF of American Naturalist, maybe he just meant ‘journals I like’.

[2] Honestly this is where we need to discuss changing the hyper-capitalist system of funding and incentives surrounding publication but that is another post entirely! Maybe people wouldn’t cheat so much if we didn’t pit them against a thousand other scientists in a no-holds-barred cage match to the death.

8 thoughts on “The Wild West of Publication Reform Is Now

  1. Thanks Micah. As you know, I largely stole the idea for my revolution of science publishing from you (and Dorothy Bishop to some degree): https://neuroneurotic.wordpress.com/2015/07/17/revolutionise-the-publication-process/

    I understand the worry by some people, especially those with limited experience with present-day information technology, to sift through the volume of unvalidated “preprints” (the term is not really accurate but it’s the best we have right now). This is why I have been very clear in my own concept that pre-acceptance peer review will always be needed. I think any reader who has no interest in actually reviewing a study should not bother reading preprints. They should wait until a criterion level of peer review has taken place. This should be very easy to implement. You simply have a default filter that blanks out all the unvalidated rubbish and shows you accepted studies. You could customise this to your own liking and only search for those papers highly rated by reviewers to be more selective in your reading. Nobody is stopping anyone in this system to promote the unvalidated preprint via social media or other ways – as long as their status is made abundantly clear.

    I do think we need to have a central platform for the preprint and peer reviews to take place. Or at the very least there should be a central platform to curate them (i.e. link to manuscript and copy/paste the text). So not really all that Wild West. I don’t think a social media style filtering is sufficient.

    Anyway, we were planning to run a Cognitive Drinks about this topic in February. I believe you were copied into those tweets but not sure you said you’d come?

    • Hey Sam. Actually I don’t agree on two points:
      ‘prepublication peer review is always needed’ – this is just not true, or at least it depends on what you mean by needed. There is no doubt that prepublication review – and publishing of those reviews – adds massive value to any publication. But as I argue in my manifesto, there is still plenty of value in publishing everything. Conservatives often talk about the deluge as if its a bad thing, as if any one person needs to read a statistically representation sample of papers for science to function. This really isn’t true. If properly documented data can be assessed via metanalytic methods, can contribute to reanalyses, and can serve as pilot data for open access power analyses. Of course the value increases when something gets pre-reviewed, and in some areas like clinical trials preregistration should simply be mandatory. But I can see plenty of value in people archiving everything they do long before peer review. It’s all a part of opening up science and breaking apart the paper into multiple usable objects.

      It seems like you agree anyway – obviously we need to make it abundantly clear that such data are not peer reviewed and are ‘use at your own risk’. But I do believe sharing science outputs at the earliest convenience will improve science communication. In my view open science will continue to evolve as a variety of praciteces ranging from data and code sharing to preprints, preregistration, and PPPR. All of these things add value and the more openly and centrally documented the better.

      As for the wild west, we’re still quite far from the point where any centralized system is likely to see wide adoption. A point I consistently make in my writing on this topic is that it’s really about changing the culture of science, not coming up with geeky platforms. Platforms a plenty exist. What is missing is widespread incentive and a culture of practice. This is why I point out we’re in the wild west – science has changed massively over the last decade. Filtration IS happening in the wild west of social media. Unpeer reviewed articles ARE having a massive impact. As these practices continue to grow with each new generation it will be easier to produce centralized systems that maximize their benefit while limiting their harm. So we need to encourage these practices in every form while also being mindful of the new challenges they bring about.

      In the end I suspect the sea change will come about in the next ten years. Once publishers see profits decline they will rush to fill the space created by open science practices. The biggest challenge remains convincing the funders to value these activities.

      • We should just stop using terms like preprint and prepublication. Publication means making something public – that’s what a “preprint” is. All I am saying is that you should be able to easily filter the stuff that experts in the field regard as high quality from unverified rubbish. There is a lot of rubbish out there and a completely uncurated system will only make that worse.

        I don’t think a central platform is all that hard to achieve, especially if it easily integrates information from different channels. If we can convince journals to let peer review take place on the platform (which could still be their own editors and reviewers – only in public and recorded on the platform) it will be adopted in no time. The fact that this will put an end to the tedious descend of the impact factor ladder should make it widely appealing.

        We also need to to herald the masses. Most scientistis, especially older more senior ones but also many junior ones, are not as familiar with the social media world or willing to engage with it. Giving people a place where both the new and the old peer review happens makes it easier for them.

  2. I believe the scientific community could take a useful cue from the tech world and consider their corpus of research as a vast knowledgebase, which it essentially is — just dithered throughout static, printed repositories. The collected knowledge of scientific communities would best be served by being subject to the same guidelines as the vast (and operationally critical) knowledgebases of service organizations in multinational corporations — where KB articles are collected as they are created, then they are technically reviewed by acknowledged subject matter experts, and leveraged by those in need, based on their rankings by users and review status.

    Having been in the technology knowledge management space for >15 years, it’s clear to me that this is really a knowledge issue, gummed up by politics and folks who are still hooked into the Gutenberg-style print era, when only the rich and privileged owned / had access to printing presses, and wealth and privilege were equated with education and cultivation (since nobody else had the leisure for / access to real schooling, back in the day). Those are no longer safe assumptions, but when the waves and wind pick up, people won’t let go of the lifesaver that’s kept them afloat in calm seas, even if there’s a helicopter hovering overhead to take them to safety.

    Ultimately, never fear – results will speak for themselves. Case in point: I’ve been working with a well-established neuropsychologist for the past ~7 years on personal neuro issues. The good doctor has been practicing for 40+ years, and shuns all manner of “new media”. Won’t go on Facebook, won’t consider Twitter, avoids Wikipedia like the plague, doesn’t look around on their own for new research, gets their info from a set group of specialized pubs, and resists anything “fringe” with gleeful dismissal.

    I, on the other hand, have been searching far and wide for neuro recovery resources, ideas, concept, new research, and yes, YouTube videos, Facebook posts, and Tweets among them. I’ve applied plenty of what I’ve found, tried a lot of new things, discarded a bunch of them, but kept open to the possibilities w/ and open mind.

    The results: The most phenomenal progress my neuropsych has ever seen happen. I’m back to earning at a rate equivalent to my experience, and my living situation is more stable than it’s ever been. These are quantifiable results directly related to “questionable” research I’ve found and applied.

    We live in a results-driven world, where the proof is in the application. Ultimately, support and success goes to those who can demonstrate value. Social media and the new knowledge infrastructure of scientific research will certainly give rise to new forms of support which will cancel out the chilling effect of the Gutenberg-era publishing/privilege model. It may not eradicate it, but it will give it a run for its money, and in the end, a new generation will make up its own mind about where it gets its information.

    I have a pretty good idea where they’ll look.

  3. Thanks, Micah, for posting this and linking to my post. These are important conversations, precisely because I think your final point is quite right: increasing prevalence of preprint servers and preprints is inevitable, and that means the time is now to think about how we want these to work, and how we want to conceptualize their content. Your Wild West metaphor might be more telling than you meant; as I understand it, the Wild West was a place of huge privilege (for instance, of men over women, Europeans over First Nations and Latinos, and wealthy over poor). We romanticize it, of course, as the ultimate in ilbertarian freedom, and that’s fine to make an enjoyable movie; but it wasn’t really that way. I’m rambling (I have a tendency to do that), but my point is that no matter how inevitable the preprint world is, we need to air and think carefully about how it’s going to work. Your post and mine, I think, together help us do that. Cheers!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s