Top tips for new experimenters

I set out to write my top five tips for new experimenters today and found there were really only two universal suggestions I felt I could and should make:

  1. Simplify your design. There is no complex question that can’t be better asked with a simple one. Simple design means stronger statistics, a clearer interpretation, and less variables to control. If you cannot phrase your core question in a sentence, you need to drastically reduce the scope of your experiment.
  2. Know your design. Before collecting the data, you should know exactly what kind of data it will be, how many variables, and what kind of statistical test you will use to analyze it. Then you need to collect 4-5 “throw-away” participants and run them through this analysis. This ensures that the data can be readily analyzed in a rigorous way, from start to finish. You will know you are ready when your looking at the successful results of a pilot, which will discover study-killing bugs (of which there are MANY)
Those are honestly the two most important guidelines I can think of! Everything else is secondary to achieving those goals. If you pull those off, you’ll have beaten 80% of the crap that can destroy your data. In my experience the biggest mistake most people make when starting out is telling themselves that simple questions are not worth their time. You’ll build a more stable career by doing something less innovative but more solid, and knowing it more thoroughly. A lot of people don’t do this and end up with total shitpiles of worthless data- myself included. Don’t let bad data happen to you- simplify and know your design! I’d love to hear about your number 1 tips in the comments!
Edit: Tip .3 comes from a great comment by Neuroskeptic:

Good post. I would add a #3, it’s kind of an aspect of #2 although important enough to stand alone –

Make sure you are one of the pilot subjects. It’s amazing what kind of things you notice when you’re actually in the scanner that you never otherwise would – anything from the fact that the stimuli aren’t very visible, to the fact that the sequence you’re using makes the bed shake, to the fact that the task is just so long & boring that you fall asleep by the end (which is so much easier in the scanner than when you’re sitting up at a computer, which is when you probably piloted the task!)

If you’re not MRI safe, get a trusted fellow researcher to do it. But never assume that non-scientist volunteers will tell you these things because they don’t (I think because they don’t want to look stupid by questioning your authority.)

6 thoughts on “Top tips for new experimenters

  1. Excellent advice, particularly number 2; people generally just run a pilot of the experimental session without thinking about the analysis, yet (particularly with fMRI data) this is where many of the issues with a design become apparent.

    • Totally agree. I think it’s crucial to experience the experiment you’ll be putting dozens of individuals through. For one thing, you’ll probably have a lot less noisy data as I think better experimenter-subject empathy would result in a smoother data collection.

  2. Good post. I would add a #3, it’s kind of an aspect of #2 although important enough to stand alone –

    Make sure you are one of the pilot subjects. It’s amazing what kind of things you notice when you’re actually in the scanner that you never otherwise would – anything from the fact that the stimuli aren’t very visible, to the fact that the sequence you’re using makes the bed shake, to the fact that the task is just so long & boring that you fall asleep by the end (which is so much easier in the scanner than when you’re sitting up at a computer, which is when you probably piloted the task!)

    If you’re not MRI safe, get a trusted fellow researcher to do it. But never assume that non-scientist volunteers will tell you these things because they don’t (I think because they don’t want to look stupid by questioning your authority.)

    • I love this suggestion and totally agree with it. I’ve always done this, but never thought of it quite the way you put it. Of course participants are unlikely to comment on mistakes you’ve made! Although they say Danes love to complain and they do tend to point out my mistakes (even when I haven’t made any!) more often than American participants did. I’m going to add this one to the post, as it’s really crucial and helps you assess the true cognitive load of your experiment.

  3. I would actually say don’t get a psychologist/neuroscientist to be a subject, because they will 1) spend the whole time trying to work out what you’re trying to do instead of actually doing the experiment 2) break it 3) complain that your experiment doesn’t answer a question that it wasn’t asking anyway.

    Or perhaps I just mean don’t get me to be a subject.

    (seriously, though don’t *just* get psych/neuros, because they know too much and may overlook things that would confuse the hell out of a naive participant; get a trusted pool of grouches who won’t spare your feelings, as well as yourself and your labmate!)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s