The article reviewed here is ‘Response to Voodoo Correlations in Social Neuroscience by Vul et al – summary information for the press’ by Jabbi et al and freely available here. The original Vul paper offers a useful focal point for discussion of the principles of fMRI image analysis and this article continues the discussion. This review will be integrated with the original analysis of the Vul paper. The authors have organised their response to the paper into 8 sections. In the first point the authors argue that there are corrections for multiple comparisons which control for the large number of voxels that have been sampled. They also argue that producing p-values and effect sizes for correlations complies with the American Psychological Association guidelines for statistical analysis. In their second point, the authors essentially argue that the upper limit value of Vul et al can be exceeded and cite a reliability of 0.98 in an fMRI study of language lateralisation to support their argument. The authors go on to comment on the simulation that Vul et al have run. I have commented on in this in an extension of the original analysis (pointed to by the link above) and was quite sceptical about the simulations that have been run as we have few details of them. Here the authors add that the simulations should have contained a family wise error correction. If I understand this correctly this is essentially the same as point 2. As the number of pairwise comparisons increases so too does the number of false positives and therefore there need to be corrections for this. The Bonferrini correction is an example and essentially allows significance of multiple comparisons on the same data set to be calculated (to the best of my understanding).
In the fourth point, the authors argue that Vul et al have ignored papers that report non-significant correlations. However I would argue that such correlations would exist even if there is a bias in the analysis. In the fifth point the authors argue that it is not only the numbers that matter but also the biological validity of the correlations. Nevertheless the statistical correlation is central to the argument as without it even biologically valid associations are nothing more than speculation. Point 6 is a very strong argument – namely that correlations have been replicated using different statistical methodologies for the analysis. Here we see the importance of constraining the argument to a specific area of the social neurosciences as without such a constraint an extremely large number of counterexamples can be produced. Also with such generalisations much of the meaning can be lost. In the seventh point, the authors comment on the questionnaire being invalid. They quite rightly point out that questions on the secondary analysis should have been included. However a much more significant point here in my opinion (and mentioned in the extended analysis linked to above) is that the questionnaires have not been validated, some of the responses have not been reported on and the methodology for qualitative analysis is not given. This is in my opinion a more significant weakness in the original paper which traverses both quantitative and qualitative analysis without a clearly reported methodology. The final point is one that I have difficulty with. The authors criticise Vul et al’s suggestion of having a split-half analysis. The authors use the term ‘commonly’ to suggest that Vul et al’s suggestion ignores statistical techniques commonly applied for these reasons. However Vul et al are proposing something slightly different.
This article contains some interesting responses to Vul et al’s paper. This paper also highlights the complexity of the process of reading a paper as well as criticising such a paper on the path to a deeper understanding of the subject area.
If you have any comments, you can leave them below or alternatively e-mail firstname.lastname@example.org
The comments made here represent the opinions of the author and do not represent the profession or any body/organisation. The comments made here are not meant as a source of medical advice and those seeking medical advice are advised to consult with their own doctor. The author is not responsible for the contents of any external sites that are linked to in this blog.