Review: Big Correlations in Little Studies

The article reviewed here is ‘Big Correlations in Little Studies’ by Yarkoni and freely available here. This is another in the long list of responses to Vul et al’s paper (which was originally reviewed here). In the introduction Yarkoni summarises the responses to Vul et al’s assertion that the correlations in studies was spuriously high and adds a third. Thus the three responses are that

1. The correlations were indeed spuriously high and in this regards Yarkoni suggests that empirically this is to a greater extent than asserted by Vul et al

2. The correlations are valid

3. The correlations are too  high but less so than stated by Vul et al.

Yarkoni then looks at other factors which influence the correlations and focuses on the power of study – a topic which he has written about previously. Yarkoni illustrates the role of power calculations with some worked examples. He argues that if an effect exists, then to detect it within the correlations at a certain level of significance, the effect size will need to be much larger in order for it to be detected. Those large effects that are detected however might also represent smaller effects that have been magnified as a result of variance. Furthermore a larger variance is evident with a smaller sample size – as the sample size increases so do the confidence intervals shrink to the effect size. He also argues that the power of between-subject effects is much smaller than that of within-subject effects. Yarkoni makes an interesting point here –

an investigator who believes in big r’s has to explain why it is that most within-subject contrasts identify relatively distributed patterns of activation, whereas correlation analyses do not

Perhaps the comments of Lindquist and Gelman are relevant here (paper reviewed here) who suggest in their analysis that in comparing groups there is no reason to suppose that the difference between mean activations in one area for the same task should be zero. In other words brain activation patterns are governed by so many factors that you could argue that there will always be differences between groups. Thus if your looking at effect sizes of significance under such circumstances you could find them in every region of interest. Indeed they also argue that the same could hold for within-subject designs as factors influencing blood flow will vary across time also.

If as he asserts the power of between-subjects comparisons is lower than that of within-subject contrasts, then the implication of the above is that behavioural correlates should be represented by distributed patterns of activation as detected in the higher powered within-subject studies. Further he writes

If one believes that brain-behaviour correlations of that strength exist in the population, the absence of any large-sample studies reporting such correlations is inexplicable….In fact, what tends to happen is exactly the opposite: As sample size grows, effects shrink

Yarkoni then argues that studies should have large sample sizes, the r values should be discounted for the sake of consistency and the focus should be on within-subject designs which are higher powered.

This is a well-argued paper that shows the immense complexity of the subject that Vul et al have chosen to comment on in their original paper as the points made here are different still to those in the previous responses to Vul et al that have been reviewed here.

Addendum 12.12.09

See this blog post by Yarkoni for an update


You can follow ‘The Amazing World of Psychiatry’ Twitter by clicking on this link


You can listen to this post on Odiogo by clicking on this link (there may be a small delay between publishing of the blog article and the availability of the podcast).

TAWOP Channel

You can follow the TAWOP Channel on YouTube by clicking on this link


If you have any comments, you can leave them below or alternatively e-mail


The comments made here represent the opinions of the author and do not represent the profession or any body/organisation. The comments made here are not meant as a source of medical advice and those seeking medical advice are advised to consult with their own doctor. The author is not responsible for the contents of any external sites that are linked to in this blog.


  1. I found very informative. The article is professionally written and I feel like the author knows the subject very well. keep it that way.


  2. […] Blog Review: Citation Needed Posted in Blog Review, psychiatry by Dr Justin Marley on December 12, 2009 The blog reviewed here is ‘Citation Needed‘. The author is a neuroscientist who describes himself in the about section in more detail. Incidentally Yarkoni was one of the authors who published a response to Vul et al’s noted paper on fMRI (see review here). […]


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s