Thelma_Louise_cliff (1)

Is Frontiers in Trouble?

Lately it seems like the rising tide is going against Frontiers. Originally hailed as a revolutionary open-access publishing model, the publishing group has been subject to intense criticism in recent years. Recent issues include being placed on Beall’s controversial ‘predatory publisher list‘, multiple high profile disputes at the editorial level, and controversy over HIV and vaccine denialist articles published in the journal seemingly without peer review. As a proud author of two Frontiers articles and former frequent reviewer, these issues compounded with a general poor perception of the journal recently led me to stop all publication activities at Frontiers outlets. Although the official response from Frontiers to these issues has been mixed, yesterday a mass-email from a section editor caught my eye:

Dear Review Editors, Dear friends and colleagues,

As some of you may know, Prof. Philippe Schyns recently stepped down from his role as Specialty Chief Editor in Frontiersin Perception Science, and I have been given the honor and responsibility of succeeding him into this function. I wish to extend to him my thanks and appreciation for the hard work he has put in building this journal from the ground up. I will strive to continue his work and maintain Frontiers in Perception Science as one of the primary journals of the field. This task cannot be achieved without the support of a dynamic team of Associate Editors, Review Editors and Reviewers, and I am grateful for all your past, and hopefully future efforts in promoting the journal.

It am aware that many scientists in our community have grown disappointed or even defiant of the Frontiers publishing model in general, and Frontiers in Perception Science is no exception here. Among the foremost concerns are the initial annoyance and ensuing disinterest produced by the automated editor/reviewer invitation system and its spam-like messages, the apparent difficulty in rejecting inappropriate manuscripts, and (perhaps as a corollary), the poor reputation of the journal, a journal to which many authors still hesitate before submitting their work. I have experienced these troubles myself, and it was only after being thoroughly reassured by the Editorial office on most of these counts that I accepted to get involved as Specialty Chief Editor. Frontiers is revising their system, which will now leave more time for Associate Editors to mandate Review Editors before sending out automated invitations. When they occur, automated RE invitations will be targeted to the most relevant people (based on keyword descriptors), rather than broadcast to the entire board. This implies that it is very important for each of you to spend a few minutes editing the Expertise keywords on your Loop profile page. Most of these keywords were automatically collected within your publications, and they may not reflect your true area of expertise. Inappropriate expertise keywords are one of the main reasons why you receive inappropriate reviewing invitations! In the new Frontiers system, article rejection options will be made more visible to the handling Associate Editor. Although my explicit approval is still required for any manuscript rejection, I personally vow to stand behind all Associate Editors who will be compelled to reject poor-quality submissions. (While perceived impact cannot be used as a rejection criterion, poor research or writing quality and objective errors in design, analysis or interpretation can and should be used as valid causes for rejection). I hope that these measures will help limit the demands on the reviewers’ time, and contribute to advancing the standards and reputation of Frontiers in Perception Science. Each of you can also play a part in this effort by continuing to review articles that fall into your area of expertise, and by submitting your own work to the journal.

I look forward to working with all of you towards establishing Frontiers in Perception Science as a high-standard journal for our community.

It seems Frontiers is indeed aware of the problems and is hoping to bring back wary reviewers and authors. But is it too little too late? Discussing the problems at Frontiers is often met with severe criticism or outright dismissal by proponents of the OA publishing system, but I felt these neglected a wider negative perception of the publisher that has steadily grown over the past 5 years. To get a better handle on this I asked my twitter followers what they thought. 152 persons responded as follows:

As some of you requested control questions, here are a few for comparison:

 

That is a stark difference between the two top open access journals – whereas only 19% said there was no problem at Frontiers, a full 50% say there is no problem at PLOS ONE. I think we can see that even accounting for general science skepticism, opinions of Frontiers are particularly negative.

Sam Schwarzkopf also lent some additional data, comparing the whole field of major open access outlets – Frontiers again comes out poorly, although strangely so does F1000:

These data confirm what I had already feared: public perception among scientists (insofar as we can infer anything from such a poll) is lukewarm at best. Frontiers has a serious perception problem. Only 19% of 121 respondents were willing to outright say there was no problem at the journal. A full 45% said there was a serious problem, and 36% were unsure. Of course to fully evaluate these numbers, we’d like to know the baserate of similiar responses for other journals, but I cannot imagine any Frontiers author, reviewer, or editor feeling joy at these numbers – I certainly do not. Furthermore they reflect a widespread negativity I hear frequently from colleagues across the UK and Denmark.

What underlies this negative perception? As many proponents point out, Frontiers has been actually quite diligent at responding to user complaints. Controversial papers have been put immediately under review, overly spammy-review invitations and special issue invites largely ceased, and so on. I would argue the issue is not any one single mistake on the part of Frontiers leadership, but a growing history of errors contributing to a perception that the journal is following a profit-led ‘publish anything’ model. At times the journal feels totally automated, within little human care given to publishing and extremely high fees. What are some of the specific complaints I regularly hear from colleagues?

  • Spammy special issue invites. An older issue, but at Frontier’s inception many authors were inundated with constant invites to special issues, many of which were only tangentially related to author’s specialties.
  • Spammy review invites. Colleagues who signed on to be ‘Review Editors’ (basically repeat reviewers) reported being hit with as many as 10 requests to review in a month, again many without relevance to their interest
  • Related to both of the above, a perception that special issues and articles are frequently reviewed by close colleagues with little oversight. Similiarly, many special issues were edited by junior researchers at the PhD level.
  • Endless review. I’ve heard numerous complaints that even fundamentally flawed or unpublishable papers are impossible or difficult to reject. Reviewers report going through multiple rounds of charitable review, finding the paper only gets worse and worse, only to be removed from the review by editors and the paper published without them.

Again, Frontiers has responded to each of these issues in various ways. For example, Frontiers originally defended the special issues, saying that they were intended to give junior researchers an outlet to publish their ideas. Fair enough, and the spam issues have largely ceased. Still, I would argue it is the build up and repetition of these issues that has made authors and readers wary of the journal. This coupled with the high fees and feeling of automation leads to a perception that the outlet is mostly junk. This is a shame as there are certainly many high-value articles in Frontiers outlets. Nevertheless, academics are extremely bloodshy, and negative press creates a vicious feedback loop. If researchers feel Frontiers is a low-quality, spam-generating publisher who relies on overly automated processes, they are unlikely to submit their best work or review there. The quality of both drops, and the cycle intensifies.

For my part, I don’t intend to return to Frontiers unless they begin publishing reviews. I think this would go a long way to stemming many of these issues and encourage authors to judge individual articles on their own merits.

What do you think? What can be done to stem the tide? Please add your own thoughts, and stories of positive or negative experiences at Frontiers, in the comments.

____

Edit:

A final comparison question

 

 

#MethodsWeDontReport – brief thought on Jason Mitchell versus the replicators

This morning Jason Mitchell self-published an interesting essay espousing his views on why replication attempts are essentially worthless. At first I was merely interested by the fact that what would obviously become a topic of heated debate was self-published, rather than going through the long slog of a traditional academic medium. Score one for self publication, I suppose. Jason’s argument is essentially that null results don’t yield anything of value and that we should be improving the way science is conducted and reported rather than publicising our nulls. I found particularly interesting his short example list of things that he sees as critical to experimental results which nevertheless go unreported:

These experimental events, and countless more like them, go unreported in our method section for the simple fact that they are part of the shared, tacit know-how of competent researchers in my field; we also fail to report that the experimenters wore clothes and refrained from smoking throughout the session.  Someone without full possession of such know-how—perhaps because he is globally incompetent, or new to science, or even just new to neuroimaging specifically—could well be expected to bungle one or more of these important, yet unstated, experimental details.

While I don’t agree with the overall logic or conclusion of Jason’s argument (I particularly like Chris Said’s Bayesian response), I do think it raises some important or at least interesting points for discussion. For example, I agree that there is loads of potentially important stuff that goes on in the lab, particularly with human subjects and large scanners, that isn’t reported. I’m not sure to what extent that stuff can or should be reported, and I think that’s one of the interesting and under-examined topics in the larger debate. I tend to lean towards the stance that we should report just about anything we can – but of course publication pressures and tacit norms means most of it won’t be published. And probably at least some of it doesn’t need to be? But which things exactly? And how do we go about reporting stuff like how we respond to random participant questions regarding our hypothesis?

To find out, I’d love to see a list of things you can’t or don’t regularly report using the #methodswedontreport hashtag. Quite a few are starting to show up- most are funny or outright snarky (as seems to be the general mood of the response to Jason’s post), but I think a few are pretty common lab occurrences and are even though provoking in terms of their potentially serious experimental side-effects. Surely we don’t want to report all of these ‘tacit’ skills in our burgeoning method sections; the question is which ones need to be reported, and why are they important in the first place?

Effective connectivity or just plumbing? Granger Causality estimates highly reliable maps of venous drainage.

update: for an excellent response to this post, see the comment by Anil Seth at the bottom of this article. Also don’t miss the extended debate regarding the general validity of causal methods for fMRI at Russ Poldrack’s blog that followed this post. 

While the BOLD signal can be a useful measurement of brain function when used properly, the fact that it indexes blood flow rather than neural activity raises more than a few significant concerns. That is to say, when we make inferences on BOLD, we want to be sure the observed effects are causally downstream of actual neural activity, rather than the product of physiological noise such as fluctuations in breath or heart rate. This is a problem for all fMRI analyses, but is particularly tricky for resting state fMRI, where we are interested in signal fluctuations that fall in the same range as respiration and pulse. Now a new study has extended these troubles to granger causality modelling (GCM), a lag-based method for estimating causal interactions between time series, popular in the resting state literature. Just how bad is the damage?

In an article published this week in PLOS ONE, Webb and colleagues analysed over a thousand scans from the Human Connectome database, examining the reliability of GCM estimates and the proximity of the major ‘hubs’ identified by GCM with known major arteries and veins. The authors first found that GCM estimates were highly robust across participants:

Plot showing robustness of GCM estimates across 620 participants. The majority of estimated causes did not show significant differences within or between participants (black datapoints).
Plot showing robustness of GCM estimates across 620 participants. The majority of estimated causes did not show significant differences within or between participants (black datapoints).

They further report that “the largest [most robust] lags are for BOLD Granger causality differences for regions close to large veins and dural venous sinuses”. In other words, although the major ‘upstream’ and ‘downstream’ nodes estimated by GCM are highly robust across participants, regions primarily effecting other regions (e.g. causal outflow) map onto major arteries, whereas regions primarily receiving ‘inputs’  (e.g.  causal inflow) map onto veins. This pattern of ‘causation’ is very difficult to explain as anything other than a non-neural artifact, as it seems like the regions mostly ‘causing’ activity in others are exactly where you would have fresh blood coming into the brain, and regions primarily being influenced by others seem to be areas of major blood drainage. Check out the arteriogram and venogram provided by the authors:

Depiction of major arteries (top image) and veins (bottom). Not overlap with areas of greatest G-cause (below).
Depiction of major arteries (top image) and veins (bottom). Note overlap with areas of greatest G-cause (below).

Compare the above to their thresholded z-statistic map for significant granger causality; white are areas of significant g-causation overlapping with an ateriogram mask, green are significant areas overlapping with a venogram mask:

journal.pone.0084279.g005
From paper:
“Figure 5. Mean Z-statistic for significant Granger causality differences to seed ROIs. Z-statistics were averaged for a given target ROI with the 264 seed ROIs to which it exhibited significantly asymmetric Granger causality relationship. Masks are overlaid for MRI arteriograms (white) and MRI venograms (green) for voxels with greater than 2 standard deviations signal intensity of in-brain voxels in averaged images from 33 (arteriogram) and 34 (venogram) subjects. Major arterial inflow and venous outflow distributions are labeled.”

It’s fairly obvious from the above that a significant proportion of the areas typically G-causing other areas overlap with arteries, whereas areas typically being g-caused by others overlap with veins. This is a serious problem for GCM of resting state fMRI, and worse, these effects were also observed for a comprehensive range of task-based fMRI data. The authors come to the grim conclusion that “Such arterial inflow and venous drainage has a highly reproducible pattern across individuals where major arterial and venous distributions are largely invariant across subjects, giving the illusion of reliable timing differences between brain regions that may be completely unrelated to actual differences in effective connectivity”. Importantly, this isn’t the first time GCM has been called into question. A related concern is the impact of spatial variation in the lag between neural activation and the BOLD response (the ‘hemodynamic response function’, HRF) across the brain. Previous work using simultaneous intracranial and BOLD recordings has shown that due to these lags, GCM can estimate a causal pattern of A then B, whereas the actual neural activity was B then A.

This is because GCM acts in a relatively simple way; given two time-series (A & B), if a better estimate of the future state of B can be predicted by the past fluctation of both A and B than that provided by B alone, then A is said to G-cause B.  However, as we’ve already established, BOLD is a messy and complex signal, where neural activity is filtered through slow blood fluctuations that must be carefully mapped back onto to neural activity using deconvolution methods. Thus, what looks like A then B in BOLD, can actually be due to differences in HRF lags between regions – GCM is blind to this as it does not consider the underlying process producing the time-series. Worse, while this problem can be resolved by combining GCM (which is naïve to the underlying cause of the analysed time series) with an approach that de-convolves each voxel-wise time-series with a canonical HRF, the authors point out that such an approach would not resolve the concern raised here that granger causality largely picks up macroscopic temporal patterns in blood in- and out-flow:

“But even if an HRF were perfectly estimated at each voxel in the brain, the mechanism implied in our data is that similarly oxygenated blood arrives at variable time points in the brain independently of any neural activation and will affect lag-based directed functional connectivity measurements. Moreover, blood from one region may then propagate to other regions along the venous drainage pathways also independent of neural to vascular transduction. It is possible that the consistent asymmetries in Granger causality measured in our data may be related to differences in HRF latency in different brain regions, but we consider this less likely given the simpler explanation of blood moving from arteries to veins given the spatial distribution of our results.”

As for correcting for these effects, the authors suggest that a nuisance variable approach estimating vascular effects related to pulse, respiration, and breath-holding may be effective. However, they caution that the effects observed here (large scale blood inflow and drainage) take place over a timescale an order of magnitude slower than actual neural differences, and that this approach would need extremely precise estimates of the associated nuisance waveforms to prevent confounded connectivity estimates. For now, I’d advise readers to be critical of what can actually  be inferred from GCM until further research can be done, preferably using multi-modal methods capable of directly inferring the impact of vascular confounds on GCM estimates. Indeed, although I suppose am a bit biased, I have to ask if it wouldn’t be simpler to just use Dynamic Causal Modelling, a technique explicitly designed for estimating causal effects between BOLD timeseries, rather than a method originally designed to estimate influences between financial stocks.

References for further reading:

Friston, K. (2009). Causal modelling and brain connectivity in functional magnetic resonance imaging. PLoS biology, 7(2), e33. doi:10.1371/journal.pbio.1000033

Friston, K. (2011). Dynamic causal modeling and Granger causality Comments on: the identification of interacting networks in the brain using fMRI: model selection, causality and deconvolution. NeuroImage, 58(2), 303–5; author reply 310–1. doi:10.1016/j.neuroimage.2009.09.031

Friston, K., Moran, R., & Seth, A. K. (2013). Analysing connectivity with Granger causality and dynamic causal modelling. Current opinion in neurobiology, 23(2), 172–8. doi:10.1016/j.conb.2012.11.010

Webb, J. T., Ferguson, M. a., Nielsen, J. a., & Anderson, J. S. (2013). BOLD Granger Causality Reflects Vascular Anatomy. (P. A. Valdes-Sosa, Ed.)PLoS ONE, 8(12), e84279. doi:10.1371/journal.pone.0084279

Chang, C., Cunningham, J. P., & Glover, G. H. (2009). Influence of heart rate on the BOLD signal: the cardiac response function. NeuroImage, 44(3), 857–69. doi:10.1016/j.neuroimage.2008.09.029

Chang, C., & Glover, G. H. (2009). Relationship between respiration, end-tidal CO2, and BOLD signals in resting-state fMRI. NeuroImage, 47(4), 1381–93. doi:10.1016/j.neuroimage.2009.04.048

Lund, T. E., Madsen, K. H., Sidaros, K., Luo, W.-L., & Nichols, T. E. (2006). Non-white noise in fMRI: does modelling have an impact? Neuroimage, 29(1), 54–66.

David, O., Guillemain, I., Saillet, S., Reyt, S., Deransart, C., Segebarth, C., & Depaulis, A. (2008). Identifying neural drivers with functional MRI: an electrophysiological validation. PLoS biology, 6(12), 2683–97. doi:10.1371/journal.pbio.0060315

Update: This post continued into an extended debate on Russ Poldrack’s blog, where Anil Seth made the following (important) comment 

Hi this is Anil Seth.  What an excellent debate and I hope I can add few quick thoughts of my own since this is an issue close to my heart (no pub intended re vascular confounds).

First, back to the Webb et al paper. They indeed show that a vascular confound may affect GC-FMRI but only in the resting state and given suboptimal TR and averaging over diverse datasets.  Indeed I suspect that their autoregressive models may be poorly fit so that the results rather reflect a sort-of mental chronometry a la Menon, rather than GC per se.
In any case the more successful applications of GC-fMRI are those that compare experimental conditions or correlate GC with some behavioural variable (see e.g. Wen et al.http://www.ncbi.nlm.nih.gov/pubmed/22279213).  In these cases hemodynamic and vascular confounds may subtract out.
Interpreting findings like these means remembering that GC is a description of the data (i.e. DIRECTED FUNCTIONAL connectivity) and is not a direct claim about the underlying causal mechanism (e.g. like DCM, which is a measure of EFFECTIVE connectivity).  Therefore (model light) GC and (model heavy) DCM are to a large extent asking and answering different questions, and to set them in direct opposition is to misunderstand this basic point.  Karl, Ros Moran, and I make these points in a recent review (http://www.ncbi.nlm.nih.gov/pubmed/23265964).
Of course both methods are complex and ‘garbage in garbage out’ applies: naive application of either is likely to be misleading or worse.  Indeed the indirect nature of fMRI BOLD means that causal inference will be very hard.  But this doesn’t mean we shouldn’t try.  We need to move to network descriptions in order to get beyond the neo-phrenology of functional localization.  And so I am pleased to see recent developments in both DCM and GC for fMRI.  For the latter, with Barnett and Chorley I have shown that GC-FMRI is INVARIANT to hemodynamic convolution given fast sampling and low noise (http://www.ncbi.nlm.nih.gov/pubmed/23036449).  This counterintuitive finding defuses a major objection to GC-fMRI and has been established both in theory, and in a range of simulations of increasing biophysical detail.  With the development of low-TR multiband sequences, this means there is renewed hope for GC-fMRI in practice, especially when executed in an appropriate experimental design.  Barnett and I have also just released a major new GC software which avoids separate estimation of full and reduced AR models, avoiding a serious source of bias afflicting previous approaches (http://www.ncbi.nlm.nih.gov/pubmed/24200508).
Overall I am hopeful that we can move beyond premature rejection of promising methods on the grounds they fail when applied without appropriate data or sufficient care.  This applies to both GC and fMRI. These are hard problems but we will get there.

Birth of a New School: How Self-Publication can Improve Research

Edit: click here for a PDF version and citable figshare link!

Preface: What follows is my attempt to imagine a radically different future for research publishing. Apologies for any overlooked references – the following is meant to be speculative and purposely walks the line between paper and blog post. Here is to a productive discussion regarding the future of research.

Our current systems of producing, disseminating, and evaluating research could be substantially improved. For-profit publishers enjoy extremely high taxpayer-funded profit margins. Traditional closed-door peer review is creaking under the weight of an exponentially growing knowledge base, delaying important communications and often resulting in seemingly arbitrary publication decisions1–4. Today’s young researchers are frequently dismayed to find their pain-staking work producing quality reviews overlooked or discouraged by journalistic editorial practices. In response, the research community has risen to the challenge of reform, giving birth to an ever expanding multitude of publishing tools: statistical methods to detect p-hacking5, numerous open-source publication models6–8, and innovative platforms for data and knowledge sharing9,10.

While I applaud the arrival and intent of these tools, I suspect that ultimately publication reform must begin with publication culture – with the very way we think of what a publication is and can be. After all, how can we effectively create infrastructure for practices that do not yet exist? Last summer, shortly after igniting #pdftribute, I began to think more and more about the problems confronting the publication of results. After months of conversations with colleagues I am now convinced that real reform will come not in the shape of new tools or infrastructures, but rather in the culture surrounding academic publishing itself. In many ways our current publishing infrastructure is the product of a paper-based society keen to produce lasting artifacts of scholarly research. In parallel, the exponential arrival of networked society has lead to an open-source software community in which knowledge is not a static artifact but rather an ever-expanding living document of intelligent productivity. We must move towards “research 2.0” and beyond11.

From Wikipedia to Github, open-source communities are changing the way knowledge is produced and disseminated. Already this movement has begun reach academia, with researchers across disciplines flocking to social media, blogs, and novel communication infrastructures to create a new movement of post-publication peer review4,12,13. In math and physics, researchers have already embraced self-publication, uploading preprints to the online repository arXiv, with more and more disciplines using the site to archive their research. I believe that the inevitable future of research communication is in this open-source metaphor, in the form of pervasive self-publication of scholarly knowledge. The question is thus not where are we going, but rather how do we prepare for this radical change in publication culture. In asking these questions I would like to imagine what research will look like 10, 15, or even 20 years from today. This post is intended as a first step towards bringing to light specific ideas for how this transition might be facilitated. Rather than this being a prescriptive essay, here I am merely attempting to imagine what that future may look like. I invite you to treat what follows as an ‘open beta’ for these ideas.

Part 1: Why self-publication?

I believe the essential metaphor is within the open-source software community. To this end over the past few months I have  feverishly discussed the merits and risks of self-publishing scholarly knowledge with my colleagues and peers. While at first I was worried many would find the notion of self-publication utterly absurd, I have been astonished at the responses – many have been excitedly optimistic! I was surprised to find that some of my most critical and stoic colleagues have lost so much faith in traditional publication and peer review that they are ready to consider more radical options.

The basic motivation for research self-publication is pretty simple: research papers cannot be properly evaluated without first being read. Now, by evaluation, I don’t mean for the purposes of hiring or grant giving committees. These are essentially financial decisions, e.g. “how do I effectively spend my money without reading the papers of the 200+ applicants for this position?” Such decisions will always rely on heuristics and metrics that must necessarily sacrifice accuracy for efficiency. However, I believe that self-publication culture will provide a finer grain of metrics than ever dreamed of under our current system. By documenting each step of the research process, self-publication and open science can yield rich information that can be mined for increasingly useful impact measures – but more on that later.

When it comes to evaluating research, many admit that there is no substitute for opening up an article and reading its content – regardless of journal. My prediction is, as post-publication peer review gains acceptance, some tenured researcher or brave young scholar will eventually decide to simply self-publish her research directly onto the internet, and when that research goes viral, the resulting deluge of self-publications will be overwhelming. Of course, busy lives require heuristic decisions and it’s arguable that publishers provide this editorial service. While I will address this issue specifically in Part 3, for now I want to point out that growing empirical evidence suggests that our current publisher/impact-based system provides an unreliable heuristic at best14–16. Thus, my essential reason for supporting self-publication is that in the worst-case scenario, self-publications must be accompanied by the disclaimer: “read the contents and decide for yourself.” As self-publishing practices are established, it is easy to imagine that these difficulties will be largely mitigated by self-published peer reviews and novel infrastructures supporting these interactions.

Indeed, with a little imagination we can picture plenty of potential benefits of self-publication to offset the risk that we might read poor papers. Researchers spend exorbitant amounts of their time reviewing, commenting on, and discussing articles – most of that rich content and meta-data is lost under the current system. In documenting the research practice more thoroughly, the ensuing flood of self-published data can support new quantitative metrics of reviewer trust, and be further utlized in the development of rich information about new ideas and data in near real-time. To give just one example, we might calculate how many subsequent citations or retractions a particular reviewer generates, generating a reviewer impact factor and reliability index. The more aspects of research we publish, the greater the data-mining potential. Incentivizing in-depth reviews that add clarity and conceptual content to research, rather than merely knocking down or propping up equally imperfect artifacts, will ultimately improve research quality. By self-publishing well-documented, open-sourced pilot data and accompanying digital reagents (e.g. scripts, stimulus materials, protocols, etc), researchers can get instant feedback from peers, preventing uncounted research dollars from being wasted. Previously closed-door conferences can become live records of new ideas and conceptual developments as they unfold. The metaphor here is research as open-source – an ever evolving, living record of knowledge as it is created.

Now, let’s contrast this model to the current publishing system. Every publisher (including open-access) obliges researchers to adhere to randomly varied formatting constraints, presentation rules, submission and acceptance fees, and review cultures. Researchers perform reviews for free for often publically subsidized work, so that publishers can then turn around and sell the finished product back to those same researchers (and the public) at an exorbitant mark-up. These constraints introduce lengthy delays – ranging from 6+ months in the sciences all the way up to two years in some humanities disciplines. By contrast, how you self-publish your research is entirely up to you – where, when, how, the formatting, and the openness. Put simply, if you could publish your research how and when you wanted, and have it generate the same “impact” as traditional venues, why would you use a publisher at all?

One obvious reason to use publishers is copy-editing, i.e. the creation of pretty manuscripts. Another is the guarantee of high-profile distribution. Indeed, under the current system these are legitimate worries. While it is possible to produce reasonably formatted papers, ideally the creation of an open-source, easy to use copy-editing software is needed to facilitate mainstream self-publication. Innovators like figshare are already leading the way in this area. In the next section, I will try to theorize some different ways in which self-publication can overcome these and other potential limitations, in terms of specific applications and guidelines for maximizing the utility of self-published research. To do so, I will outline a few specific cases with the most potential for self-publication to make a positive impact on research right away, and hopefully illuminate the ‘why’ question a bit further with some concrete examples.

 Part 2: Where to begin self-publishing

What follows is the “how-to” part of this document. I must preface by saying that although I have written so far with researchers across the sciences and humanities in mind, I will now focus primarily on the scientific examples with which I am more experienced.  The transition to self-publication is already happening in the forms of academic tweets, self-archives, and blogs, at a seemingly exponential growth rate. To be clear, I do not believe that the new publication culture will be utopian. As in many human endeavors the usual brandism3, politics, and corruption can be expected to appear in this new culture. Accordingly, the transition is likely to be a bit wild and woolly around the edges. Like any generational culture shift, new practices must first emerge before infrastructures can be put in place to support them. My hope is to contribute to that cultural shift from artifact to process-based research, outlining particularly promising early venues for self-publication. Once these practices become more common, there will be huge opportunities for those ready and willing to step in and provide rich informational architectures to support and enhance self-publication – but for now we can only step into that wild frontier.

In my discussions with others I have identified three particularly promising areas where self-publication is either already contributing or can begin contributing to research. These are: the publication of exploratory pilot-data, post-publication peer reviews, and trial pre-registration. I will cover each in turn, attempting to provide examples and templates where possible. Finally, Part 3 will examine some common concerns with self-publication. In general, I think that successful reforms should resemble existing research practices as much as possible: publication solutions are most effective when they resemble daily practices that are already in place, rather than forcing individuals into novel practices or infrastructures with an unclear time-commitment. A frequent criticism of current solutions such as the comments section on Frontiers, PLOS One, or the newly developed PubPeer, is that they are rarely used by the general academic population. It is reasonable to conclude that this is because already over-worked academics currently see little plausible benefit from contributing to these discussions given the current publishing culture (worse still, they may fear other negative repercussions, discussed in Part 3). Thus a central theme of the following examples is that they attempt to mirror practices in which many academics are already engaged, with complementary incentive structures (e.g. citations).

Example 1: Exploratory Pilot Data 

This previous summer witnessed a fascinating clash of research cultures, with the eruption of intense debate between pre-registration advocates and pre-registration skeptics. I derived some useful insights from both sides of that discussion. Many were concerned about what would happen to exploratory data under these new publication regimes. Indeed, a general worry with existing reform movements is that they appear to emphasize a highly conservative and somewhat cynical “perfect papers” culture. I do not believe in perfect papers – the scientific model is driven by replication and discovery. No paper can ever be 100% flawless – otherwise there would be no reason for further research! Inevitably, some will find ways to cheat the system. Accordingly, reform must incentivize better reporting practices over stricter control, or at least balance between the two extremes.

Exploratory pilot data is an excellent avenue for this. By their very nature such data are not confirmatory – they are exciting in that they do not conform well to prior predictions. Such data benefit from rapid communication and feedback. Imagine an intuition-based project – a side or pet project conducted on the fly for example. The researcher might feel that the project has potential, but also knows that there could be serious flaws. Most journals won’t publish these kinds of data. Under the current system these data are lost, hidden, obscured, or otherwise forgotten.

Compare to a self-publication world: the researcher can upload the data, document all the protocols, make the presentation and analysis scripts open-source, and provide some well-written documentation explaining why she thinks the data are of interest. Some intrepid graduate student might find it, and follow up with a valuable control analysis, pointing out an excellent feature or fatal flaw, which he can then upload as a direct citation to the original data. Both publications are citable, giving credit to originator and reviewer alike. Armed with this new knowledge, the original researcher could now pre-register an altered protocol and conduct a full study on the subject (or alternatively, abandon the project entirely). In this exchange, it is likely that hundreds of hours and research dollars will have been saved. Additionally, the entire process will have been documented, making it both citable and minable for impact metrics. Tools already exist for each of these steps – but largely cultural fears prevent it from happening. How would it be perceived? Would anyone read it? Will someone steal my idea? To better frame these issues, I will now examine a self-publication practice that has already emerged in force.

 Example 2: Post-publication peer review

This is a particularly easy case, precisely because high-profile scholars are already regularly engaged in the practice. As I’ve frequently joked on twitter, we’re rapidly entering an era where publishing in a glam-mag has no impact guarantee if the paper itself isn’t worthwhile – you may as well hang a target on your head for post-publication peer reviewers. However, I want to emphasize the positive benefits and not just the conservative controls. Post-publication peer review (PPPR) has already begun to change the way we view research, with reviewers adding lasting content to papers, enriching the conclusions one can draw, and pointing out novel connections that were not extrapolated upon by the authors themselves. Here I like to draw an analogy to the open source movement, where code (and its documentation) is forkable, versioned, and open to constant revision – never static but always evolving.

Indeed, just last week PubMed launched their new “PubMed Commons” system, an innovative PPPR comment system, whereby any registered person (with at least one paper on PubMed) can leave scientific comments on articles.  Inevitably, the reception on twitter and Facebook mirrored previous attempts to introduce infrastructure-based solutions – mixed excitement followed by a lot of bemused cynicism – bring out the trolls many joked. To wit, a brief scan of the average comment on another platform, PubPeer, revealed a generally (but not entirely) poor level of comment quality. While many comments seem to be on topic, most had little to no formatting and were given with little context. At times comments can seem trollish, pointing out minor flaws as if they render the paper worthless. In many disciplines like my own, few comments could be found at all. This compounds the central problem with PPPR; why would anyone acknowledge such a system if the primary result is poorly formed nitpicking of your research? The essential problem here is again incentive – for reviews to be quality there needs to be incentive. We need a culture of PPPR that values positive and negative comments equally. This is common to both traditional and self-publication practices.

To facilitate easy, incentivized self-publication of comments and PPPRs, my colleague Hauke Hillebrandt and I have attempted to create a simple template that researchers can use to quickly and easily publish these materials. The idea is that by using these templates and uploading them to figshare or similar services, Google Scholar will automatically index them as citations, provide citation alerts to the original authors, and even include the comments in its h-index calculation. This way researchers can begin to get credit for what they are already doing, in an easy to use and familiar format. While the template isn’t quite working yet (oddly enough, Scholar is counting citations from my blog, but not the template), you can take a look at it here and maybe help us figure out why it isn’t working! In the near future we plan to get this working, and will follow-up this post with the full template, ready for you to use.

Example 3: Pre-registration of experimental trials

As my final example, I suggest that for many researchers, self-publication of trial pre-registrations (PR) may be an excellent way to test the waters of PR in a format with a low barrier to entry. Replication attempts are a particularly promising venue for PR, and self-publication of such registrations is a way to quickly move from idea to registration to collection (as in the above pilot data example), while ensuring that credit for the original idea is embedded in the infamously hard to erase memory of the internet.

A few benefits of PR self-publication, rather than relying on for-profit publishers, is that PR templates can be easily open-sourced themselves, allowing various research fields to generate community-based specialized templates adhering to the needs of that field. Self-published PRs, as well as high quality templates, can be cited – incentivizing the creation and dissemination of both. I imagine the rapid emergence of specialized templates within each community, tailored to the needs of that research discipline.

Part 3: Criticism and limitations

Here I will close by considering some common concerns with self-publication:

Quality of data

A natural worry at this point is quality control. How can we be sure that what is published without the seal of peer review isn’t complete hooey? The primary response is that we cannot, just like we cannot be sure that peer reviewed materials are quality without first reading them ourselves. Still, it is for this reason that I tried to suggest a few particularly ripe venues for self-publication of research. The cultural zeitgeist supporting full-blown scholarly self-publication has not yet arrived, but we can already begin to prepare for it. With regards to filtering noise, I argue that by coupling post-publication peer review and social media, quality self-publications will rise to the top. Importantly, this issue points towards flaws in our current publication culture. In many research areas there are effects that are repeatedly published but that few believe, largely due to the presence of biases against null-findings. Self-publication aims to make as much of the research process publicly available as possible, preventing this kind of knowledge from slipping through the editorial cracks and improving our ability to evaluate the veracity of published effects. If such data are reported cleanly and completely, existing quantitative tools can further incorporate them to better estimate the likelihood of p-hacking within a literature. That leads to the next concern – quality of presentation.

Hemingway's thoughts on data.

Quality of presentation

Many ask: how in this brave new world will we separate signal from noise? I am sure that every published researcher already receives at least a few garbage citations a year from obscure places in obscure journals with little relevance to actual article contents. But, so the worry goes, what if we are deluged with a vast array of poorly written, poorly documented, self-published crud. How would we separate the signal from the noise?

 The answer is Content, Presentation, and Clarity. These must be treated as central guidelines for self-publication to be worth anyone’s time. The Internet memesphere has already generated one rule for ranking interest: content rules. Content floats and is upvoted, blogspam sinks and is downvoted. This is already true for published articles – twitter, reddit, facebook, and email circles help us separate the wheat from the chaff at least as much as impact factor if not more. But presentation and clarity are equally important. Poorly conducted research is not shared, or at least is shared with vehemence. Similarly, poorly written self-publications, or poorly documented data/reagents are unlikely to generate positive feedback, much less impact-generating eyeballs. I like to imagine a distant future in which self-publication has given rise to a new generation of well-regarded specialists: reviewers who are prized for their content, presentation, and clarity; coders who produce cleanly documented pipelines; behaviorists producing powerful and easily customized paradigm scripts; and data collection experts who produce the smoothest, cleanest data around. All of these future specialists will be able to garner impact for the things they already do, incentivizing each step of the research processes rather than only the end product.

Being scooped, intellectual credit

Another common concern is “what if my idea/data/pilot is scooped?” I acknowledge that particularly in these early days, the decision to self-publish must be weighted against this possibility. However, I must also point out that in the current system authors must also weight the decision to develop an idea in isolation against the benefits of communicating with peers and colleagues. Both have risks and benefits – an idea or project in isolation can easily over-estimate its own quality or impact. The decision to self-publish must similarly be weighted against the need for feedback. Furthermore, a self-publication culture would allow researchers to move more quickly from project to publication, ensuring that they are readily credited for their work. And again, as research culture continues to evolve, I believe this concern will increasingly fade. It is notoriously difficult to erase information from The Internet (see the “Streisand effect”) – there is no reason why self-published ideas and data cannot generate direct credit for the authors. Indeed, I envision a world in which these contributions can themselves be independently weighted and credited.

 Prevention of cheating, corruption, self-citations

To some, this will be an inevitable point of departure. Without our time-tested guardian of peer review, what is to prevent a flood of outright fabricated data? My response is: what prevents outright fabrication under the current system? To misquote Jeff Goldblum in Jurassic Park, cheaters will always find a way. No matter how much we tighten our grip, there will be those who respond to the pressures of publication by deliberate misconduct. I believe that the current publication system directly incentivizes such behavior by valuing end product over process. By creating incentives for low-barrier post-publication peer review, pre-registration, and rich pilot data publication, researchers are given the opportunity to generate impact for each step of the research process. When faced with the vast penalties of cheating due to a null finding, versus doing one’s best to turn those data into something useful for someone, I suspect most people will choose the honest and less risky option.

 Corruption and self-citations are perhaps a subtler, more sinister factor. In my discussions with colleagues, a frequent concern is that there is nothing to prevent high-impact “rich club” institutions from banding together to provide glossy post-publication reviews, citation farming, or promoting one another’s research to the top of the pile regardless of content. I again answer: how is this any different from our current system? Papers are submitted to an editor who makes a subjective evaluation of the paper’s quality and impact, before sending it to four out of a thousand possible reviewers who will make an obscure  decision about the content of the paper. Sometimes this system works well, but increasingly it does not2. Many have witnessed great papers rejected for political reasons, or poor ones accepted for the same. Lowering the barrier to post-publication peer review means that even when these factors drive a paper to the top, it will be far easier to contextualize that research with a heavy dose of reality. Over time, I believe self-publication will incentivize good research. Cheating will always be a factor – and this new frontier is unlikely to be a utopia. Rather, I hope to contribute to the development of a bridge between our traditional publishing models and a radically advanced not-too-distant future.

Conclusion

Our current systems of producing, disseminating, and evaluating research increasingly seem to be out of step with cultural and technological realities. To take back the research process and bolster the ailing standard of peer-review I believe research will ultimately adopt an open and largely publisher-free model. In my view, these new practices will be entirely complementary to existing solutions including such as the p-curve5, open-source publication models6–8, and innovative platforms for data and knowledge sharing such as PubPeer, PubMed Commons, and figshare9,10. The next step from here will be to produce useable templates for self-publication. You can expect to see a PDF version of this post in the coming weeks as a further example of self-publishing practices. In attempting to build a bridge to the coming technological and social revolution, I hope to inspire others to join in the conversation so that we can improve all aspects of research.

 Acknowledgments

Thanks to Hauke Hillebrandt, Kate Mills, and Francesca Fardo for invaluable discussion, comments, and edits of this work. Many of the ideas developed here were originally inspired by this post envisioning a self-publication future. Thanks also to PubPeer, PeerJ,  figshare, and others in this area for their pioneering work in providing some valuable tools and spaces to begin engaging with self-publication practices.

Addendum

Excellent resources already exist for the many of the ideas presented here. I want to give special notice to researchers who have already begun self-publishing their work either as preprints, archives, or as direct blog posts. Parallel publishing is an attractive transitional option where researchers can prepublish their work for immediate feedback before submitting it to a traditional publisher. Special notice should be given to Zen Faulkes whose excellent pioneering blog posts demonstrated that it is reasonably easy to self-produce well formatted publications. Here are a few pioneering self-published papers you can use as examples – feel free to add your own in the comments:

The distal leg motor neurons of slipper lobsters, Ibacus spp. (Decapoda, Scyllaridae), Zen Faulkes

http://neurodojo.blogspot.dk/2012/09/Ibacus.html

Eklund, Anders (2013): Multivariate fMRI Analysis using Canonical Correlation Analysis instead of Classifiers, Comment on Todd et al. figshare.

http://dx.doi.org/10.6084/m9.figshare.787696

Automated removal of independent components to reduce trial-by-trial variation in event-related potentials, Dorothy Bishop

http://bishoptechbits.blogspot.dk/2011_05_01_archive.html

Deep Impact: Unintended consequences of journal rank

Björn Brembs, Marcus Munafò

http://arxiv.org/abs/1301.3748

A novel platform for open peer to peer review and publication:

http://thewinnower.com/

A platform for open PPPRs:

https://pubpeer.com/

Another PPPR platform:

http://f1000.com/

References

1. Henderson, M. Problems with peer review. BMJ 340, c1409 (2010).

2. Ioannidis, J. P. A. Why Most Published Research Findings Are False. PLoS Med 2, e124 (2005).

3. Peters, D. P. & Ceci, S. J. Peer-review practices of psychological journals: The fate of published articles, submitted again. Behav. Brain Sci. 5, 187 (2010).

4. Hunter, J. Post-publication peer review: opening up scientific conversation. Front. Comput. Neurosci. 6, 63 (2012).

5. Simonsohn, U., Nelson, L. D. & Simmons, J. P. P-Curve: A Key to the File Drawer. (2013). at <http://papers.ssrn.com/abstract=2256237>

6.  MacCallum, C. J. ONE for All: The Next Step for PLoS. PLoS Biol. 4, e401 (2006).

7. Smith, K. A. The frontiers publishing paradigm. Front. Immunol. 3, 1 (2012).

8. Wets, K., Weedon, D. & Velterop, J. Post-publication filtering and evaluation: Faculty of 1000. Learn. Publ. 16, 249–258 (2003).

9. Allen, M. PubPeer – A universal comment and review layer for scholarly papers? | Neuroconscience on WordPress.com. Website/Blog (2013). at <https://neuroconscience.com/2013/01/25/pubpeer-a-universal-comment-and-review-layer-for-scholarly-papers/>

10. Hahnel, M. Exclusive: figshare a new open data project that wants to change the future of scholarly publishing. Impact Soc. Sci. blog (2012). at <http://eprints.lse.ac.uk/51893/1/blogs.lse.ac.uk-Exclusive_figshare_a_new_open_data_project_that_wants_to_change_the_future_of_scholarly_publishing.pdf>

11. Yarkoni, T., Poldrack, R. A., Van Essen, D. C. & Wager, T. D. Cognitive neuroscience 2.0: building a cumulative science of human brain function. Trends Cogn. Sci. 14, 489–496 (2010).

12. Bishop, D. BishopBlog: A gentle introduction to Twitter for the apprehensive academic. Blog/website (2013). at <http://deevybee.blogspot.dk/2011/06/gentle-introduction-to-twitter-for.html>

13. Hadibeenareviewer. Had I Been A Reviewer on WordPress.com. Blog/website (2013). at <http://hadibeenareviewer.wordpress.com/>

14. Tressoldi, P. E., Giofré, D., Sella, F. & Cumming, G. High Impact = High Statistical Standards? Not Necessarily So. PLoS One 8, e56180 (2013).

15.  Brembs, B. & Munafò, M. Deep Impact: Unintended consequences of journal rank. (2013). at <http://arxiv.org/abs/1301.3748>

16.  Eisen, J. A., Maccallum, C. J. & Neylon, C. Expert Failure: Re-evaluating Research Assessment. PLoS Biol. 11, e1001677 (2013).

http://wl.figshare.com/articles/875339/embed?show_title=1

Can compassion be trained like a muscle? Active-controlled fMRI of compassion meditation.

Among the cognitive training literature, meditation interventions are particularly unique in that they often emphasize emotional or affective processing at least as much as classical ‘top-down’ attentional control. From a clinical and societal perspective, the idea that we might be able to “train” our “emotion muscle” is an attractive one. Recently much has been made of the “empathy deficit” in the US, ranging from empirical studies suggesting a relationship between quality-of-care and declining caregiver empathy, to a recent push by President Obama to emphasize the deficit in numerous speeches.

While much of the training literature focuses on cognitive abilities like sustained attention and working memory, many investigating meditation training have begun to study the plasticity of affective function, myself included.  A recent study by Helen Weng and colleagues in Wisconsin investigated just this question, asking if compassion (“loving-kindness”) meditation can alter altruistic behavior and associated neural processing. Her study is one of the first of its kind, in that rather than merely comparing groups of advanced practitioners and controls, she utilized a fully-randomized active-controlled design to see if compassion responds to brief training in novices while controlling for important confounds.

As many readers should be aware, a chronic problem in training studies is a lack of properly controlled longitudinal design. At best, many rely on “passive” or “no-contact” controls who merely complete both measurements without receiving any training. Even in the best of circumstances “active” controls are often poorly matched to whatever is being emphasized and tested in the intervention of interest. While having both groups do “something” is better than a passive or no-control design, problems may still arise if the measure of interest is mismatched to the demand characteristics of the study.  Stated simply, if your condition of interest receives attention training and attention tests, and your control condition receives dieting instruction or relaxation, you can expect group differences to be confounded by an explicit “expectation to improve” in the interest group.

In this regard Weng et al present an almost perfect example of everything a training study should be. Both interventions were delivered via professionally made audio CDs (you can download them yourselves here!), with participants’ daily practice experiences being recorded online. The training materials were remarkably well matched for the tests of interest and extra care was taken to ensure that the primary measures were not presented in a biased way. The only thing they could have done further would be a single blind (making sure the experimenters didn’t know the group identity of each participant), but given the high level of difficulty in blinding these kinds of studies I don’t blame them for not undertaking such a manipulation. In all the study is extremely well-controlled for research in this area and I recommend it as a guideline for best practices in training research.

Specifically, Weng et al tested the impact of loving-kindness compassion meditation or emotion reappraisal training on an emotion regulation fMRI task and behavioral economic game measuring altruistic behavior. For the fMRI task, participants viewed emotional pictures (IAPS) depicting suffering or neutral scenarios and either practiced a compassion meditation or reappraisal strategy to regulate their emotional response, before and after training. After the follow-up scan, good-old fashion experimental deception was used to administer a dictator economics-game that was ostensibly not part of the primary study and involved real live players (both deceptions).

For those not familiar with the dictator game, the concept is essentially that a participant watches a “dictator” endowed with 100$ give “unfair” offers to a “victim” without any money. Weng et al took great care in contextualizing the test purely in economic terms, limiting demand confounds:

Participants were told that they were playing the game with live players over the Internet. Effects of demand characteristics on behavior were minimized by presenting the game as a unique study, describing it in purely economic terms, never instructing participants to use the training they received, removing the physical presence of players and experimenters during game play, and enforcing real monetary consequences for participants’ behavior.

This is particularly important, as without these simple manipulations it would be easy for stodgy reviewers like myself to worry about subtle biases influencing behavior on the task. Equally important is the content of the two training programs. If for example, Weng et al used a memory training or attention task as their active-control group, it would be difficult not to worry that behavioral differences were due to one group expecting a more emotional consequence of the study, and hence acting more altruistic. In the supplementary information, Weng et al describe the two training protocols in great detail:

Compassion

… Participants practiced compassion for targets by 1) contemplating and envisioning their suffering and then 2) wishing them freedom from that suffering. They first practiced compassion for a Loved One, such as a friend or family member. They imagined a time their loved one had suffered (e.g., illness, injury, relationship problem), and were instructed to pay attention to the emotions and sensations this evoked. They practiced wishing that the suffering were relieved and repeated the phrases, “May you be free from this suffering. May you have joy and happiness.” They also envisioned a golden light that extended from their heart to the loved one, which helped to ease his/her suffering. They were also instructed to pay attention to bodily sensations, particularly around the heart. They repeated this procedure for the Self, a Stranger, and a Difficult Person. The Stranger was someone encountered in daily life but not well known (e.g., a bus driver or someone on the street), and the Difficult Person was someone with whom there was conflict (e.g., coworker, significant other). Participants envisioned hypothetical situations of suffering for the stranger and difficult person (if needed) such as having an illness or experiencing a failure. At the end of the meditation, compassion was extended towards all beings. For each new meditation session, participants could choose to use either the same or different people for each target category (e.g., for the loved one category, use sister one day and use father the next day).

Reappraisal

… Participants were asked to recall a stressful experience from the past 2 years that remained upsetting to them, such as arguing with a significant other or receiving a lower-than- expected grade. They were instructed to vividly recall details of the experience (location, images, sounds). They wrote a brief description of the event, and chose one word to best describe the feeling experienced during the event (e.g., sad, angry, anxious). They rated the intensity of the feeling during the event, and the intensity of the current feeling on a scale (0 = No feeling at all, 100 = Most intense feeling in your life). They wrote down the thoughts they had during the event in detail. Then they were asked to reappraise the event (to think about it in a different, less upsetting way) using 3 different strategies, and to write down the new thoughts. The strategies included 1) thinking about the situation from another person’s perspective (e.g., friend, parent), 2) viewing it in a way where they would respond with very little emotion, and 3) imagining how they would view the situation if a year had passed, and they were doing very well. After practicing each strategy, they rated how reasonable each interpretation was (0 = Not at all reasonable, 100 = Completely reasonable), and how badly they felt after considering this view (0 = Not bad at all, 100 = Most intense ever). Day to day, participants were allowed to practice reappraisal with the same stressful event, or choose a different event. Participants logged the amount of minutes practiced after the session.

In my view the active control is extremely well designed for the fMRI and economic tasks, with both training methods explicitly focusing on the participant altering an emotional response to other individuals.  In tests of self-rated efficacy, both groups showed significant decreases in negative emotion, further confirming the active control. Interestingly when Weng et al compared self-ratings over time, only the compassion group showed significant reduction from the first half of training sessions to the last. I’m not sure if this constitutes a limitation, as Weng et al further report that on each individual training day the reappraisal group reported significant reductions, but that the reductions themselves did not differ significantly over time. They explain this as being likely due to the fact that the reappraisal group frequently changed emotional targets, whereas the compassion group had the same 3 targets throughout the training. Either way the important point is that both groups self-reported similar overall reductions in negative emotion during the course of the study, strongly supporting the active control.

Now what about the findings? As mentioned above, Weng et al tested participants before and after training on an fMRI emotion regulation task. After the training, all participants performed the “dictator game”, shown below. After rank-ordering the data, they found that the compassion group showed significantly greater redistribution:

The dictator task (left) and increased redistribution (right).

For the fMRI analysis, they analyzed BOLD responses to negative vs neutral images at both time points, subtracted the beta coefficients, and then entered these images into a second-level design matrix testing the group difference, with the rank-ordered redistribution scores as a covariate of interest. They then tested for areas showing group differences in the correlation of redistribution scores and changes of BOLD response to negative vs neutral images (pre vs post), across the whole brain and in several ROIs, while properly correcting for multiple comparisons. Essentially this analysis asks, where in the brain do task-related changes in BOLD correlate more or less with the redistribution score in one group or another. For the group x covariate interaction they found significant differences (increased BOLD-covariate correlation) in the right inferior parietal cortex (IPC), a region of the parietal attention network, shown on the left-hand panel:

Increased correlation between negative vs neutral imagery related BOLD and redistribution scores (left), connectivity with DLPFC (right).

They further extracted signal from the IPC cluster and entered it into a conjunction analysis, testing for areas showing significant correlation  with the IPC activity, and found a strong effect in right DLPFC (right panel). Finally they performed a psychophysiological interaction (PPI) analysis with the right DLPFC activity as the seed, to determine regions showing significant task-modulated connectivity with that DLPFC activity. The found increased emotion-modulated DLPFC connectivity to nucleus accumbens, a region involved in encoding positive rewards (below, right).

Screen shot 2013-05-23 at 3.21.15 PM
PPI shows increased emotion-modulated connectivity of nucleus accumbens and DLPFC.

Together these results implicate training-related BOLD activity increases to emotional stimuli in the parietal attention network and increased parietal connectivity with regions implicated in cognitive control and reward processing, in the observed altruistic behavior differences. The authors conclude that compassion training may alter emotional processing through a novel mechanism, where top-down central-executive circuits redirect emotional information to areas associated with positive reward, reflecting the role of compassion meditation in emphasizing increased positive emotion to the aversive states of others. A fitting and interesting conclusion, I think.

Overall, the study should receive high marks for its excellent design and appropriate statistical rigor. There is quite a bit of interesting material in the supplementary info, a strategy I dislike, but that is no fault of the authors considering the publishing journal (Psych Science). The question itself is extremely novel, in terms of previous active-controlled studies. To date only one previous active-controlled study investigated the role of compassion meditation on empathy-related neuroplasticity. However that study compared compassion meditation with a memory strategy course, which (in my opinion) exposes it to serious criticism regarding demand characteristic. The authors do reference that study, but only briefly to state that both studies support a role of compassion training in altering positive emotion- personally I would have appreciated a more thorough comparison, though I suppose I can go and to that myself if I feel so inclined :).

The study does have a few limitations worth mentioning. One thing that stood out to me was that the authors never report the results of the overall group mean contrast for negative vs neutral images. I would have liked to know if the regions showing increased correlation with redistribution actually showed higher overall mean activation increases during emotion regulation. However as the authors clearly had quite specific hypotheses, leading them to  restrict their alpha to 0.01 (due to testing 1 whole-brain contrast and 4 ROIs), I can see why they left this out. Given the strong results of the study, it would in retrospect perhaps have been more prudent to skip  the ROI analysis (which didn’t seem to find much) and instead focus on testing the whole brain results.  I can’t blame them however, as it is surprising not to see anything going on in insula or amygdala for this kind of training.  It is also a bit unclear to me why the DLPFC was used as the PPI seed as opposed to the primary IPC cluster, although I am somewhat unfamiliar with the conjunction-connectivity analysis used here. Finally, as the authors themselves point out, a major limitation of the study is that the redistribution measure was collected only at time two, preventing a comparison to baseline for this measure.

Given the methodological state of the topic (quite poor, generally speaking), I am willing to grant them these mostly minor caveats. Of course, without a baseline altruism measure it is difficult to make a strong conclusion about the causal impact of the meditation training on altruism behavior, but at least their neural data are shielded from this concern. So while we can’t exhaustively conclude that compassion can be trained, the results of this study certainly suggest it is possible and perhaps even likely, providing a great starting point for future research. One interesting thing for me was the difference in DLPFC. We also found task-related increases in dorsolateral prefrontal cortex following active-controlled meditation, although in the left hemisphere and for a very different kind of training and task. One other recent study of smoking cessation also reported alteration in DLPFC following mindfulness training, leading me to wonder if we’re seeing the emergence of empirical consensus for this region’s specific involvement in meditation training. Another interesting point for me was that affective regulation here seems to involve primarily top-down or attention related neural correlates,  suggesting that bottom-up processing (insula, amygdala) may be more resilient to brief training, something we also found in our study. I wonder if the group mean-contrasts would have been revealing here (i.e. if there were differences in bottom-up processing that don’t correlate with redistribution). All together a great study that raises the bar for training research in cognitive neuroscience!

A brave new default mode in meditation practitioners- or just confused controls? Review of Brewer (2011)

Given that my own work focuses on cognitive control, intrinsic connectivity, and mental-training (e.g. meditation) I was pretty excited to see Brewer et al’s paper on just these topics appear in PNAS just in time for the winter holidays. I meant to review it straight away but have been buried under my own data analysis until recently. Sadly, when I finally got around to delving into it, my overall reaction was lukewarm at best. Without further ado, my review of:

“Meditation experience is associated with differences in default mode network activity and connectivity

Abstract:

“Many philosophical and contemplative traditions teach that “living in the moment” increases happiness. However, the default mode of humans appears to be that of mind-wandering, which correlates with unhappiness, and with activation in a network of brain areas associated with self-referential processing. We investigated brain activity in experienced meditators and matched meditation-naive controls as they performed several different meditations (Concentration, Loving-Kindness, Choiceless Awareness). We found that the main nodes of the default mode network(medial prefrontal and posterior cingulate cortices) were relatively deactivated in experienced meditators across all meditation types. Furthermore, functional connectivity analysis revealed stronger coupling in experienced meditators between the posterior cingulate, dorsal anterior cingulate, and dorsolateral prefrontal cortices (regions previously implicated in self- monitoring and cognitive control), both at baseline and during meditation. Our findings demonstrate differences in the default-mode network that are consistent with decreased mind-wandering. As such, these provide a unique understanding of possible neural mechanisms of meditation.”

Summary:

Aims: 9/10

Methods: 5/10

Interpretation: 7/10

Importance/Generalizability: 4/10

Overall: 6.25/10

The good: simple, clear cut design, low amount of voodoo, relatively sensible findings

The bad: lack of behavioral co-variates to explain neural data, yet another cross-sectional design

The ugly: prominent reporting of uncorrected findings, comparison of meditation-naive controls to practitioners using meditation instructions (failure to control task demands).

Take-home: Some interesting conclusions, from a somewhat tired and inconclusive design. Poor construction of baseline condition leads to a shot-gun spattering of brain regions with a few that seem interesting given prior work. Let’s move beyond poorly controlled cross-sections and start unravelling the core mechanisms (if any) involved in mindfulness.

Extended Review:
Although this paper used typical GLM and functional connectivity analyses, it loses points in several areas. First, although the authors repeatedly suggest that their “relative paucity of findings” may be “driven by the sensitivity of GLM analysis to fluctuations at baseline… and since meditation practitioners may be (meditating) at baseline…” the contrast would be weak. However, I will side with Jensen et al (2011) here in saying: Meditation naive controls receiving less than 5 minutes of instruction in “focused attention, loving-kindness and choiceless awareness” are simply no controls at all. The argument that the inability of the GLM to detect differences that are quite obviously confounded by a lack of an appropriately controlled baseline is galling at best. This is why we use a GLM-approach; it’s senseless to make conclusions about brain activity when your baseline is no baseline at all. Telling meditation-naive controls to utilize esoteric cultural practices of which they have only just been introduced too, and then comparing that to highly experienced practitioners is a perfect storm of cognitive confusion and poorly controlled demand characteristic. Further, I am disappointed in the review process that allowed the following statement “We found a similar pattern in the medial prefrontal cortex (mPFC), another primary node of the DMN, although it did not survive whole-brain correction for signifigance” followed by this image:

image

These results are then referred to repeatedly in the following discussion. I’m sorry, but when did uncorrected findings suddenly become interpretable? I blame the reviewers here over the authors- they should have known better. The MPFC did not survive correction and hence should not be included in anything other than a explicitly stated as such “exploratory analysis”. In fact it’s totally unclear from the methods section of this paper how these findings where at all discovered: did the authors first examine the uncorrected maps and then re-analyze them using the FWE correction? Or did they reduce their threshold in an exploratory post-hoc fashion? These things make a difference and I’m appalled that the reviewers let the article go to print as it is, when figure 1 and the discussion clearly give the non-fMRI savy reader the impression that a main finding of this study is MPFC activation during meditation. Can we please all agree to stop reporting uncorrected p-values?

I will give the authors this much; the descriptions of practice, and the theoretical guideposts are all quite coherent and well put-together. I found their discussion of possible mechanisms of DMN alteration in meditation to be intriguing, even if I do not agree with their conclusion. Still, it pains me to see a paper with so much potential fail to address the pitfalls in meditation research that should now be well known. Indeed the authors themselves make much ado about how difficult proper controls are, yet seem somehow oblivious to the poorly controlled design they here report. This leads me to my own reinterpretation of their data.

A new default mode, or confused controls?

Brewer et al (2011) report that, when using a verbally guided meditation instruction with meditation naive-controls and experienced practitioners, greater activations in PCC, temporal regions, and for loving-kindness, amygdala are found. Given strong evidence by colleagues Christian Jensen et al (2011) that these kinds of contrasts better represent differences in attentional effort than any mechanism inherent to meditation, I can’t help but wonder if what were seeing here is simply some controls trying to follow esoteric instructions and getting confused in the process. Consider the instruction for the choiceless awareness condition:

“Please pay attention to whatever comes into your awareness, whether it is a thought, emotion, or body sensation. Just follow it until something else comes into your awareness, not trying to hold onto it or change it in any way. When something else comes into your awareness, just pay attention to it until the next thing comes along”

Given that in most contemplative traditions, choiceless awareness techniques are typically late-level advanced practices, in which the very concept of grasping to a stimulus is distinctly altered and laden with an often spiritual meaning, it seems obvious to me that such an instruction constitutes and excellent mindwandering inducement for naive-controls. Do you meditate? I do a little, and yet I find these instructions extremely difficult to follow without essentially sending my mind in a thousand directions. Am I doing this correctly?  When should I shift? Is this a thought or am I just feeling hungry? These things constitute mind-wandering but for the controls, I would argue they constitute following the instructions. The point is that you simply can’t make meaningful conclusions about the neural mechanisms involved in mindfulness from these kinds of instructions.

Finally, let’s examine the functional-connectivity analysis. To be honest, there isn’t a whole lot to report here; the functional connectivity during meditation is perhaps confounded by the same issues I list above, which seems to me a probable cause for the diverse spread of regions reported between controls and meditators. I did find this bit to be interesting:

“Using the mPFC as the seed region, we found increased connectivity with the fusiform gyrus, inferior temporal and parahippocampal gyri, and left posterior insula (among other regions) in meditators relative to controls during meditation (Fig. 3, Fig. S1H, and Table S3). A subset of those regions showed the same relatively increased connectivity in meditators during the baseline period as well (Fig. S1G and Table1)

I found it interesting that the meditation conditions appear to co-activate MPFC and insula, and would love to see this finding replicated in properly controlled design. I also have a nagging wonder as to why the authors didn’t bother to conduct a second-level covariance analysis of their findings with the self-reported mind-wandering scores. If these findings accurately reflect meditation-induced alterations in the DMN, or as the authors more brazenly suggest “a entirely new default network”, wouldn’t we expect their PCC modulations to be predicted by individual variability in mind-wandering self-reports? Of course, we could open the whole can of worms that is “what does it mean when you ask participants if they ‘experienced mind wandering” but I’ll leave that for a future review. At least the authors throw a bone to neurophenomenology, suggesting in the discussion that future work utilize first-person methodology. Indeed.

Last, it occurs to me that the primary finding, of increased DLPFC and ACC in meditation>Controls, also fits well with my intepretation that this design is confounded by demand characteristics. If you take a naive subject and put them in the scanner with these instructions, I’ve argued that their probably going to do something a whole lot like mind-wandering. On the other hand, an experienced practitioner has a whole lot of implicit pressure on them to live up to their tradition. They know what they are their for, and hence they know that they should be doing their thing with as much effort as possible. So what does the contrast meditation>naive really give us? It gives us mind-wandering in the naive group, and increased attentional effort in the practitioner group. We can’t conclude anything from this design regarding mechanisms intrinsic to mindfulness; I predict that if you constructed a similar setting with any kind of dedicated specialist, and gave instructions like “think about your profession, what it means to you, remember a time you did really well” you would see the exact same kind of results. You just can’t compare the uncomparable.

Disclaimer: as usual, I review in the name of science, and thank the authors whole-heartily for the great effort and attention to detail that goes into these projects.  Also it’s worth mentioning that my own research focuses on many of these exact issues in mental training research, and hence i’m probably a bit biased in what I view as important issues.

New Meditation Study in Neuroimage: “Meditation training increases brain efficiency in an attention task”

Just a quick post to give my review of the latest addition to imaging and mindfulness research. A new article by Kozasa et al, slated to appear in Neuroimage, investigates the neural correlates of attention processing in a standard color-word stroop task. A quick overview of the article reveals it is all quite standard; two groups matched for age, gender, and years of education are administered a standard RT-based (i.e. speeded) fMRI paradigm. One group has an average of 9 years “meditation experience” which is described as “a variety of OM (open monitoring) or FA (focused attention) practices such as “zazen”, mantra meditation, mindfulness of breathing, among others”. We’ll delve into why this description should give us pause for thought in a moment, for now let’s look at the results.

Amplitude of bold responses in the lentiform nucleus, medial frontal gyrus, middle temporal gyrus and precentral gyrus during the incongruent and congruent conditions in meditators and non-meditators.
Results from incon > con, non-meditators vs meditators

In a nutshell, the authors find that meditation-practitioners show faster reaction times with reduced BOLD-signal for the incongruent (compared to congruent and neutral) condition only. The regions found to be more active for non-meditators compared to meditators are the (right) “lentiform nucleus, medial frontal gyrus, and pre-central gyrus” . As this is not accompanied by any difference in accuracy, the authors interpret the finding as demonstrating  that “meditators may have maintained the focus in naming the colour with less interference of reading the word and consequently have to exert less effort to monitor the conflict and less adjustment in the motor control of the impulses to choose the correct colour button.” The authors in the conclusion review related findings and mention that differences in age could have contributed to the effect.

So, what are we to make of these findings? As is my usual style, I’ll give a bulleted review of the problems that immediately stand out, and then some explanation afterwards. I’ll preface my critique by thanking the authors for their hard work; my comments are intended only for the good of our research community.

The good:

  • Sensible findings; increases in reaction time and decreases in bold are demonstrated in areas previously implicated in meditation research
  • Solid, easy to understand behavioral paradigm
  • Relatively strong main findings ( P< .0001)
  • A simple replication. We like replications!
The bad:
  • Appears to report uncorrected p-values
  • Study claims to “match samples for age” yet no statistical test demonstrating no difference is shown. Qualitatively, the ages seem different enough to be cause for worry (77.8% vs 65% college graduates). Always be suspicious when a test is not given!
  • Extremely sparse description of style of practice, no estimate of daily practice hours given.
  • Reaction-time based task with no active control

I’ll preface my conclusion with something Sara Lazar, a meditation researcher and neuroimaging expert at the Harvard MGH told me last summer; we need to stop going for the “low hanging fruit of meditation research”. There are now over 20 published cross-sectional reaction-time based fMRI studies of “meditators” and “non-meditators”. Compare that to the incredibly sparse number of longitudinal, active controlled studies, and it is clear that we need to stop replicating these findings and start determining what they actually tell us. Why do we need to active control our meditation studies? For one thing, we know that reaction-time based tests are heavily based by the amount of effort one expends on the task. Effort is in turn influenced by task-demands (e.g. how you treat your participants, expectations surrounding the experiment). To give one in-press example, my colleagues Christian Gaden Jensen at the Copenhagen Neurobiology Research recently conducted a study demonstrating just how strong this confounding effect can be.

To briefly summarize, Christian recruited over 150 people for randomization to four experimental groups: mindfulness-based stress reduction (MBSR), non-mindfulness stress reduction (NMSR), wait-listed controls, and financially-motivated wait-listed controls. This last group is the truly interesting one; they were told that if they had top performance on the experimental tasks (a battery of classical reaction-time based and unspeeded perceptual threshold tasks) they’d receive a reward of approximately 100$. When Christian analyzed the data, he found that the financial incentive eliminated all reaction-time based differences between the MBSR, NMSR, and financially motivated groups! It’s important to note that this study, fully randomized and longitudinal, showed something not reflected in the bulk of published studies: that meditation may actually train more basic perceptual sensitivities rather than top-down control. This is exactly why we need to stop pursuing the low-hanging fruit of uncontrolled experimental design; it’s not telling us anything new! Meditation research is no longer exploratory.

In addition to these issues, there is another issue a bit more specific to meditation research. That is the totally sparse description of the practice- less than one sentence total, with no quantitative data! In this study we are not even told what the daily practice actually consists of, or its quality or length. These practitioners report an average of 8 years practice, yet that could be 1 hour per week of mantra meditation or 12 hours a week of non-dual zazen! These are not identical processes and our lack of knowledge for this sample severely limits our ability to assess the meaning of  these findings. For the past two years (and probably longer) of the Mind & Life Summer Research Institute, Richard Davidson and others have repeatedly stated that we must move beyond studying meditation as “a loose practice of FA and OM practices including x, y, z, & and other things”. Willoughby Britton suggested at a panel discussion that all meditation papers need to have at least one contemplative scholar on them or risk rejection. It’s clear that this study was most likely not reviewed by anyone with any serious academic background in meditation research.

My supervisor Antoine Lutz and his colleague John Dunne, authors of the paper that launched the “FA/OM” distinction, have since stated emphatically that we must go beyond these general labels and start investigating effects of specific meditation practices. To quote John, we need to stop treating meditation like a “black box” if we ever want to understand the actual mechanisms behind it. While I thank the authors of this paper for their earnest contribution, we need to take this moment to be seriously skeptical. We can only start to understand processes like meditation from a scientific point of view if we are willing to hold them to the highest of scientific standards. It’s time for us to start opening the black box and looking inside.

Intrinsic correlations between Salience, Primary Sensory, and Default Mode Networks following MBSR

Going through my RSS backlog today, I was excited to see Kilpatrick et al.’s “Impact of Mindfulness-Based Stress Reduction Training on Intrinsic Brain Connectivity” appear in this week’s early view Neuroimage. Although I try to keep my own work focused on primary research in cognition and connectivity, mindfulness-training (MT) is a central part of my research. Additionally, there are few published findings on intrinsic connectivity in this area. Previous research has mainly focused on between-group differences in anatomical structure (gray and white matter for example) and task-related activity. A few more recent studies have gone as far as to randomize participants into wait-listed control and MT groups.

While these studies are interesting, they are of course limited in their scope by several factors. My supervisor Antoine Lutz emphasizes that in addition to our active-controlled research here in Århus, his group at Wisconsin-Madison and others are actively preparing such datasets. Active controls are simply ‘mock’ interventions (or real ones) designed to control for every possible aspect of being involved in an intervention (placebo, community, motivation) in order to isolate the variables specific to that treatment (in this case, meditation, but not sitting, breathing, or feeling special).  Active controls are important as there is a great deal of research demonstrating that cognition itself is susceptible to placebo-like motivational effects. All and all, I’ve seen several active-controlled, cognitive-behavioral studies in review that suggest we should be strongly skeptical of any non-active controlled findings. While I can’t discuss these in detail, I will mention some of these issues in my review of the neuroimage manuscript. It suffices to say however, that if you are working on a passive-controlled study in this area, you had better get it out fast as you can expect reviewers to be greatly tightening their expectations in the coming months, as more and more rigorous papers appear. As Sara Lazar put it during my visit to her lab last summer “the low-hanging fruit of MBSR brain research are rapidly vanishing”. Overall this is a good thing for the community and we’ll see why in a moment.

Now let us turn to the paper at hand. Kilpatrick et al start with a standard summary of MBSR and rsfMRI research, focusing on findings indicating MBSR trains focused attention, sensory introspection/interception and perception. They briefly review now well-established findings indicating that rsfMRI is sensitive to training related changes, including studies that demonstrate the sensitivity of the resting state to conditions such as fatigue, eyes-open vs eyes-closed, and recent sleep. This is all pretty well and good, but I think it’s a bit odd when we see just how they collect their data.

Briefly, they recruited 32 healthy adults for randomization to MBSR and waitlist controls. Controls then complete the Mindfulness Attention Awareness Scale (MAAS) and receive 8 weeks of diary-logged standard MBSR training. After training, participants are recalled for the rsfMRI scan. An important detail here is that participants are not scanned before and after training, rendering the fMRI portion of the experiment closer to a cross-section than true longitudinal design. At the time of scan, the researchers then give two ‘task-free states’, with and without auditory white noise. The authors indicate that the noise condition is included “to enable new analysis methods not conducted here”, presumably to average out scanner-noise related affects. They later indicate no differences between the two conditions, which causes me to ask how much here is meditation vs focusing-on-scanner-noise specific. Finally, they administer the ‘task free’ states with a slight twist:

“”During this baseline scan of about 5 min, we would like you to again stay as still as possible and be mindfully aware of your surroundings. Please keep your eyes closed during this procedure. Continue to be mindfully aware of whatever you notice in your surroundings and your own sensations. Mindful awareness means that you pay attention to your present moment experience, in this case the changing sounds of the scanner/changing background sounds played through the headphones, and to bring interest and curiosity to how you are responding to them.”

While the manipulation makes sense given the experimenter’s hypothesis concerning sensory processing, an ongoing controversy in resting-state research is just what it is that constitutes ‘rest’. Research here suggests that functional connectivity is sensitive to task-instructions and variations in visual stimulation, and many complain about the lack of specificity within different rest conditions. Kilpatrick et al’s manipulation makes sense given that what they really want to see is meditation-related alterations, but it’s a dangerous leap without first establishing the relationship between ‘true rest’ and their ‘auditory meditation’ condition. Research on the impact of scanner-noise indicates some degree of noise-related nuisance effects, and also some functionally significant effects. If you’ve never been in an MR experiment, the scanner is LOUD. During my first scan I actually started feeling claustrophobic due to the oppressive machine-gun like noise of the gradient coil. Anyway, it’s really troubling that Kilpatrick et al don’t include a totally task-free set for comparison, and I’m hesitant to call this a resting-state finding without further clarification.

The study is extremely interesting, but it’s important to note it’s limitations:

  1. Lack of active control- groups are not controlled for motivation.
  2. No pre/post scan.
  3. Novel resting state without comparison condition.
  4. Findings are discussed as ‘training related’ without report of correlation with reported practice hours.
  5. Anti-correlations reported with global-signal nuisance regression. No discussion of possible regression related inducement (see edit).
  6. Discussion of findings is unclear; reported as greater DMN x Auditory correlation, but the independent component includes large portions of the salience network.

Ultimately they identify a “auditory/salience” independent component network (ICN) (primary auditory, STG, posterior Insula, ACC, and lateral frontal cortex) and then conduct seed-regression analyses of the network with areas of the DMN and Dorsal Attention Network (DAN). I find it highly strange that they pick up a network that seems to conflate primary sensory and salience regions, as do the researchers who state “Therefore, the ICN was labeled as “auditory/salience”. It is unclear why the components split differently in our sample, perhaps the instructions that brought attention to auditory input altered the covariance structure somewhat.” Given the lack of motivational control in the study, the issues in this study begin to pile onto one another and I am not sure what we can really conclude. They further find that the MBSR group demonstrates greater “auditory/salience x DMN connectivity”, “greater visual and auditory functional connectivity” (see image below). They also report several increased anti-correlations, between the aud/sal network, dMPFC and visual regions. I find this to be an extremely tantalizing finding as it would reflect a decrease in processing automaticity amongst the SAL, CEN, and DMN networks. There are some serious problems with these kinds of analysis that the authors don’t address, and so we again must reserve any strong conclusions. Here is what Kilpatrick et al conclude:

“The current findings extend the results of prior studies that showed meditation-related changes in specific brain regions active during attention and sensory processing by providing evidence that MBSR trained compared to untrained subjects, during a focused attention instruction, have increased connectivity within sensory networks and between regions associated with attentional processes and those in the attended sensory cortex. In addition they show greater differentiation between regions associated with attentional processes and the unattended sensory cortex as well as greater differentiation between attended and unattended sensory networks”

As is typical, the list of findings is quite long and I won’t bother re-stating it all here. Given the resting instructions it seems clear that the freshly post-MBSR participants are likely to have engaged a pretty dedicated set of cognitive operations during the scan. Yet it’s totally unclear what the control group would do given these contemplative instructions. Presumably they’d just lie in the scanner and try not to tune out the noise- but you can see here how it’s not clear that these conditions are really that comparable without having some idea of what’s going on. In essence what you (might) have here is one group actually doing something (meditation) and the other group not doing much at all. Ideally you want to see how training impacts the underlying process in a comparable way. Motivation has been repeatedly linked to BOLD signal intensity and in this case, it could very well be that these findings are simple artifacts of motivation to perform. If one group is actually practicing mindfulness and the other isn’t, you have not isolated the variable of interest. The authors could have somewhat alleviated this by including data from the additional pain task (“not reported here”) and/or at least giving us a correlation of the findings with the MAAS scale. I emphasize that I do find the findings of this paper interesting- they map extremely well onto my own hypotheses about how RSNs interact with mindfulness training, but that we must interpret them with caution.

Overall I think this was a project with a strong theoretical motivation and some very interesting ideas. One problem with looking at state-mindfulness in the scanner is the cramped, noisy environment. I think Kilpatrick et al had a great idea in their attempt to use the noise itself as a manipulation. Further, the findings make a good deal of sense. Still, given the above limitation, it’s important to be really careful with our conclusions. At best, this study warrants an extremely rigorous follow-up, and I wish neuroimage had published it with a bit more information, such as the status of any rest-MAAS correlations. Anyway, this post has gotten quite long and I think I’d best get back to work- for my next post I think I’ll go into more detail about some of the issues confront resting state (what is “rest”?) and mindfulness (role of active controls for community, motivation, and placebo effects) and what they mean for resting-state research.

edit: just realized I never explained limitation #5. See my “beautiful noise” slides (previous post) regarding the controversy of global signal regression and anti-correlation. Simply put, there is somewhat convincing evidence that this procedure (designed to eliminate low-frequency nuisance co-variates) may actually mathematically induce anti-correlations where none exist, probably due to regression to the mean. While it’s not a slam-dunk (see response by Fox et al), it’s an extremely controversial area and all anti-correlative findings should be interpreted in light of this possibility.

If you like this post please let me know in the comments! If I can get away with rambling about this kind of stuff, I’ll do so more frequently.