The article reviewed here is ‘Big Correlations in Little Studies’ by Yarkoni and freely available here. This is another in the long list of responses to Vul et al’s paper (which was originally reviewed here). In the introduction Yarkoni summarises the responses to Vul et al’s assertion that the correlations in studies was spuriously high and adds a third. Thus the three responses are that
1. The correlations were indeed spuriously high and in this regards Yarkoni suggests that empirically this is to a greater extent than asserted by Vul et al
2. The correlations are valid
3. The correlations are too high but less so than stated by Vul et al.
Yarkoni then looks at other factors which influence the correlations and focuses on the power of study – a topic which he has written about previously. Yarkoni illustrates the role of power calculations with some worked examples. He argues that if an effect exists, then to detect it within the correlations at a certain level of significance, the effect size will need to be much larger in order for it to be detected. Those large effects that are detected however might also represent smaller effects that have been magnified as a result of variance. Furthermore a larger variance is evident with a smaller sample size – as the sample size increases so do the confidence intervals shrink to the effect size. He also argues that the power of between-subject effects is much smaller than that of within-subject effects. Yarkoni makes an interesting point here –
‘an investigator who believes in big r’s has to explain why it is that most within-subject contrasts identify relatively distributed patterns of activation, whereas correlation analyses do not‘
Perhaps the comments of Lindquist and Gelman are relevant here (paper reviewed here) who suggest in their analysis that in comparing groups there is no reason to suppose that the difference between mean activations in one area for the same task should be zero. In other words brain activation patterns are governed by so many factors that you could argue that there will always be differences between groups. Thus if your looking at effect sizes of significance under such circumstances you could find them in every region of interest. Indeed they also argue that the same could hold for within-subject designs as factors influencing blood flow will vary across time also.
If as he asserts the power of between-subjects comparisons is lower than that of within-subject contrasts, then the implication of the above is that behavioural correlates should be represented by distributed patterns of activation as detected in the higher powered within-subject studies. Further he writes
‘If one believes that brain-behaviour correlations of that strength exist in the population, the absence of any large-sample studies reporting such correlations is inexplicable….In fact, what tends to happen is exactly the opposite: As sample size grows, effects shrink‘
Yarkoni then argues that studies should have large sample sizes, the r values should be discounted for the sake of consistency and the focus should be on within-subject designs which are higher powered.
This is a well-argued paper that shows the immense complexity of the subject that Vul et al have chosen to comment on in their original paper as the points made here are different still to those in the previous responses to Vul et al that have been reviewed here.
See this blog post by Yarkoni for an update
You can follow ‘The Amazing World of Psychiatry’ Twitter by clicking on this link
You can listen to this post on Odiogo by clicking on this link (there may be a small delay between publishing of the blog article and the availability of the podcast).
You can follow the TAWOP Channel on YouTube by clicking on this link
If you have any comments, you can leave them below or alternatively e-mail firstname.lastname@example.org
The comments made here represent the opinions of the author and do not represent the profession or any body/organisation. The comments made here are not meant as a source of medical advice and those seeking medical advice are advised to consult with their own doctor. The author is not responsible for the contents of any external sites that are linked to in this blog.