Look guys, we're headed to PNAS! No, don't bring your research integrity, we won't be needing it!

A walk in the park increases poor research practices and decreases reviewer critical thinking

Or so i’m going to claim because science is basically about making up whatever qualitative opinion you like and hard-selling it to a high impact journal right? Last night a paper appeared in PNAS early access entitled “Nature experience reduces rumination and subgenual prefrontal cortex activation” as a contributed submission. Like many of you I immediately felt my neurocringe brain area explode with activity as I began to smell the sickly sweet scent of gimmickry. Now I don’t have a lot of time so I was worried I wouldn’t be able to cover this paper in any detail. But never to worry, because the entire paper is literally two ANOVAs!

Don't think about it too much.
Look guys, we’re headed to PNAS! No, no, leave the critical thinking skills, we won’t be needing those where we’re going!

The paper begins with a lofty appeal to our naturalistic sensibilities; we’re increasingly living in urban areas, this trend is associated with poor mental health outcomes, and by golly-gee, shouldn’t we have a look at the brain to figure this all out? The authors set about testing their hypothesis by sending 19 people out into the remote wilderness of the Stanford University campus, or an urban setting:

The nature walk took place in a greenspace near Stanford University spanning an area ∼60 m northwest of Junipero Serra Boulevard and extending away from the street in a 5.3-km loop, including a significant stretch that is far (>1 km) from the sounds and sights of the surrounding residential area. As one proxy for urbanicity, we measured the proportion of impervious surface (e.g., asphalt, buildings, sidewalks) within 50 m of the center of the walking path (Fig. S4). Ten percent of the area within 50 m of the center of the path comprised of impervious surface (primarily of the asphalt path). Cumulative elevation gain of this walk was 155 m. The natural environment of the greenspace comprises open California grassland with scattered oaks and native shrubs, abundant birds, and occasional mammals (ground squirrels and deer). Views include neighboring, scenic hills, and distant views of the San Francisco Bay, and the southern portion of the Bay Area (including Palo Alto and Mountain View to the south, and Menlo Park and Atherton to the north). No automobiles, bicycles, or dogs are permitted on the path through the greenspace.

Wow, where can I sign up for this truly Kerouac-inspired bliss? The control group on the other hand had to survive the horrors of the palo-alto urban wasteland:

The urban walk took place on the busiest thoroughfare in nearby Palo Alto (El Camino Real), a street with three to four lanes in each direction and a steady stream of traffic. Participants were instructed to walk down one side of the street in a southeasterly direction for 2.65 km, before turning around at a specific point marked on a map. This spot was chosen as the midpoint of the walk for the urban walk to match the nature walk with respect to total distance and exercise. Participants were instructed to cross the street at a pedestrian crosswalk/stoplight, and return on the other side of the street (to simulate the loop component of the nature walk and greatly reduce repeated encounters with the same environmental stimuli on the return portion of the walk), for a total distance of 5.3 km; 76% of the area within 50mof the center of this section of El Camino was comprised of impervious surfaces (of roads and buildings) (Fig. S4). Cumulative elevation gain of this walk was 4 m. This stretch of road consists of a significant amount of noise from passing cars. Buildings are almost entirely single- to double-story units, primarily businesses (fast food establishments, cell phone stores, motels, etc.). Participants were instructed to remain on the sidewalk bordering the busy street and not to enter any buildings. Although this was the most urban area we could select for a walk that was a similar distance from the MRI facility as the nature walk, scattered trees were present on both sides of El Camino Real. Thus, our effects may represent a conservative estimate of effects of nature experience, as our urban group’s experience was not devoid of natural elements.

And they got that approved by the local ethics board? The horror!

The authors gave both groups a self-reported rumination questionnaire before and after the walk, and also acquired some arterial spin labeling MRIs. Here is where the real fun gets started – and basically ends – as the paper is almost entirely comprised of group by time ANOVAs on these two measures. I wish I could say I was suprised by what I found in the results:
whattf

That’s right folks – the key behavioral interaction of the paper – is non-significant. Measly. Minuscule. Forget about p-values for a second and consider the gall it takes to not only completely skim over this fact (nowhere in the paper is it mentioned) and head right to the delicious t-tests, but to egregiously promote this ‘finding’ in the title, abstract, and discussion as showing evidence for an effect of nature on rumination! Erroneous interaction for the win, at least with PNAS contributed submissions right?! The authors also analyzed the brain data in the same way – this time actually sticking with their NHST – and find that some brain area that has been previously related to some bad stuff showed reduced activity. And that – besides a heart rate and respiration control analyses – is it. No correlations with the (non-significant) behavior. Just pure and simple reverse inference piled on top of fallacious interpretation of a non-significant interaction. Never-mind the wonky and poorly operationalized research question!

See folks, high impact science is easy! Just have friends in the National Academy…

I’ll leave you with this gem from the methods:

“‘One participant was eliminated in analysis of self-reported rumination due to a decrease in rumination after nature experience that was 3 SDs below the mean.'”

That dude REALLY got his time’s worth from the walk. Or did the researchers maybe forget to check if anyone smoked a joint during their nature walk?

26 thoughts on “A walk in the park increases poor research practices and decreases reviewer critical thinking

  1. I only read one sentence, on the perfusion method, and there’s an error in it.

    “…a pulsed continuous ASL flow alternating inversion recovery (FAIR) sequence…”

    FAIR is a pulsed ASL (PASL) method. There are also continuous ASL (CASL) and pseudo-continuous ASL (PCASL) methods. I wonder why nobody from the dozens of MRI experts at Stanford was involved in the work?

  2. Ahhh, I made the mistake of reading on. Another mistake in the acquisition parameters for the FAIR sequence:

    “…matrix size = 512 × 8…”

    Not likely, not with voxels having dimensions of 1.875 x 1.875 mm in-plane. This is what happens when you let non-experts play with the fancy apparatus.

    • Thanks for finding this PractiCal! My general reading of this paper is that it is really a low-effort in general. Very little attempt was made to do any kind of rigorous approach. I’m guessing this is a first or early paper for the lead author, probably during PhD. Since it is contributed this has to be the work of an overzelous PI…

      • I’ve got another headline for you:

        “PNAS reinvents itself as a pilot study blog!”

        Let’s face it, this entire piece of work is a pilot study and is blog- worthy, requiring much critical input (such as ours) before the authors scurry off back to the lab to reemerge in six months’ time with a final, actual conclusion.

        In some part the experiment is rather careful. (Yes, I did scan it through for anything else that caught my eye.) For instance, they realize that CBF is considered a more stable method than BOLD over days/weeks because of the baseline subtraction. And they bother to measure & check respiration and HR as possible confounds during the imaging. The report reads like a circumspect study. So it really bites that they then didn’t take their interesting pilot finding and try to replicate it in a separate group. (Readers, I have this opinion of ANY and ALL findings across ALL areas of science but especially those with fMRI and its ilk. I make an exception for people researching language acquisition in primates, in which case I’ll take a single French-speaking monkey.)

  3. Saw this headline immediately after reading your piece:

    “Going for a countryside walk really DOES reduce stress and anxiety: Strolling in natural surroundings ‘is good for the brain’, study reveals”

    Too painful…

  4. Micah,

    Your post reminded me of one I just wrote. Here’s the link if you want to take a look. http://davefleming.org/blog/2015/6/11/research-showsso-you-can-rest-easier

    The digital world allows for the dissemination of information like never before. That is good. However, it is problematic when research is presented in a “one and done” and “take it to the bank” fashion. The exclusion of validity and reliability in the presentation, leaves the reader to believe the idea is generalizable. And, of course, all it really may be is one small study that means almost nothing.

    Dave

  5. Micah,

    Couldn’t agree more. We possess the ability to disseminate information like never before. That is good. But, it can backfire. Research that is presented in an article often presented in a “one and done” “take it to the bank” manner. There is often no discussion as to validity or reliability. What was once in an obscure journal awaiting more research, can go viral in a day–and presto we have trouble.

    I wrote a blog post on this as well. You can see it here.

    http://davefleming.org/blog/2015/6/11/research-showsso-you-can-rest-easier

    I like your site.

    Dave

  6. I am doubting the significance of the question posed and the application of findings. Nature walk may have a positive effect on emotional wellbeing of some people. I think that personal preferences related to background differences may play a role in this (were these people brought up in urban or natural setting? what are their preferred modes?). All that statistical analysis reveals in this case is the distribution of people numbers with nature/urban environment preference. If it is meant to be used in therapy, would it be not more valuable to test a particular therapeutic approach?

  7. Good post!

    It’s not clear that the imaging adds anything to this study. It is quite plausible that nice scenery improves mood to a greater extent than ugly scenery. But this doesn’t mean that nice scenery exterts a specific effect on brain activity. Any effect of nice scenery on brain activity might be a nonspecific, downstream effect of mood.

    We know that mood affects brain activity. Therefore anything which effects mood will also affect brain activity for that reason. We don’t need a brain scan to know that.

    If it turned out that a nature walk exerted a different effect on brain activity compared to another equally pleasant experience (a walk around an art gallery, say) then and only then could we say that nature has a specific effect on the brain.

  8. I’m poking at this paper, and saw this:

    “Heart rate showed a main effect of time, but no interaction effect of time by
    environment [F(1,26) = 0.08, not significant]”

    Someone on twitter also pointed out that the nature walkers had a higher level of rumination before the walk.

  9. Uhhh… but wouldn’t a nice screen saver have the same effect? Watching paint dry? There seems to have been no attempt to design an experiment that isolates the obviously hoped-for contributing factor.

  10. Reblogged this on Surface Your Real Self and commented:
    “That’s right folks – the key behavioral interaction of the paper – is non-significant. Measly. Minuscule. Forget about p-values for a second and consider the gall it takes to not only completely skim over this fact (nowhere in the paper is it mentioned) and head right to the delicious t-tests, but to egregiously promote this ‘finding’ in the title, abstract, and discussion as showing evidence for an effect of nature on rumination!…No correlations with the (non-significant) behavior. Just pure and simple reverse inference piled on top of fallacious interpretation of a non-significant interaction.”

  11. I have another issue with the methods. While the walks were of equal length (5.3 km) and an equal duration (90 mins) was claimed, the urban walk necessitated multiple stops in order to cross intersections. I can’t thus see how the durations were actually the same, but that is a secondary point. The ability to wander while ruminating in the urban environment was interrupted by the frequent need to avoid being run over. To be equivalent, then, the nature walk would require multiple interruptions of thought necessary to preserve life. What if the nature walk had a few hazards posted along it? Encounters with a venomous snake or an angry bear, perhaps. How might the preservation of one’s life affect the rumination in nature then? Even if there is an effect, I would want to see an interrupted walk in both cases. A third category might even be considered: wander on a treadmill for 5.3 km. How does that compare to the urban and nature walks?

  12. I couldn’t help but notice that your blog post seems to have cherry-picked the non-significant findings and deliberately ignored the significant findings. Quoted *and highlighted* in the blog is the statement “This analysis revealed an interaction between time and environment [F(1,35) = 3.51, P = 0.07, eta-sq = 0.09]” (p. 8568). What is NOT quoted or even alluded to is the following statement, that appears one paragraph later: “sgPFC perfusion showed an interaction effect of time by environment, indicating an effect of the nature walk vs. the urban walk […F(1,29) = 23.41, P < 0.0001, eta-sq = 0.45]". Kind of important when you are assailing the worthiness of a publication to note all of the reported effects, and not just the one you think makes the authors look the worst, don't you think?

    • No, actually I directly addressed the brain results in two different places. One, I note that it is curious the authors felt the need to stick to their NHST in one measure but not the other, and two when I accurately describe their intepretation of those results as reverse inference. Without a solid relationship to behavior those brain results in and of themselves are close to totally uninformative, and especially DO NOT support the claim that this study shows walking in nature reduces rumination and unhappy thoughts. I suggest you read the paper on reverse inference by Russ Poldrack, linked in the post itself. Also I’m not sure you understand what “cherry picking” means. The headline of the study is that walking in nature reduces rumination. The study only includes two variables – one brain and one behavior- and the key behavioral finding isn’t significant. Considering I discuss how neither of these findings tells us anything about the hypothesis in question, I’d hardly call that cherry picking. I think the entire study makes the authors look bad, not just their poor grasp of NHST. The study is poorly designed and the experimental hypothesis is barely operationalized yet alone the weak findings. I chose to focus on the p-value because frankly, it’s just a funny example of the overall failure of peer review going on with this paper.

  13. Hi Micah
    I’ve admired you for criticising bullying, intimidation and punching down in online debate. But then I read this post. How come the change in tone and approach?

    • This post is from last year. I don’t see anything in this post that counts as the kind of ad hominem attacks I’ve spoken against. That being said, I’ve never said you can’t be a harsh critic of shoddy science. For what its worth the post is negative in tone and I made a new years resolution to be positive in my writing and social media activities. Sorry but I stand by this post and don’t believe it is an example of bully, intimidation, etc.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s