
- Vol 39, Issue 10
Exploring Meta-Analyses: An Example
What are the elements of a good meta-analysis?
(This article addresses methodological and statistical concerns about a recent study titled “The Serotonin Theory of Depression: A Systematic Umbrella Review of the Evidence.”1 For further discussion, please see the
The brain is the most complicated thing in the universe. Problems with the brain, such as depression, are also complex and involve manifold factors including genes, other biological conditions, general medical conditions, past and current environmental exposures, and psychological capabilities and weaknesses. Multifactorial causation means it will always be hard to detect the signal of a single contributing cause. Lack of proof or imperfect proof of a derangement having a causal role is not proof of a lack of causation.
A good review is based on an exhaustive search for data. The review by Moncrieff et al looked at only 3 databases. When the authors found more than 5 reviews or large analyses, they included only the most recent 5. The investigators did not find a systematic review or
Unfortunately, the review misrepresented some of the statistics.
First, see the Moncrieff et al Table 2, in which “No effect” was listed based on 95% confidence intervals. As described in the companion piece in this issue, confidence intervals shrink as the number of included studies increases. More importantly, confidence intervals measure only how well we have estimated the mean. Confidence intervals do not measure heterogeneity, which is what we want to know in interpreting whether a result might represent a meaningful effect in some populations.
Second, the review in question did not report the prediction interval, which is the statistic that represents heterogeneity. It may be that the underlying meta-analyses and studies sometimes did not report prediction intervals, but if so, that should have been noted to address heterogeneity.
Third, in the Moncrieff et al Table 1, the I2 statistic is listed as a measure of heterogeneity, but I2 is a ratio, not a measure of absolute heterogeneity.
Fourth, effect sizes are more important than statistical significance. The review dismissed findings as statistically insignificant by calculating p values relative to confidence intervals. However, as previously noted, the confidence interval is not even the correct statistic to test. In the Moncrieff et al Table 1, some of the effect sizes, such as those found by Wang, Nikolaus, Kambeitz, Gryglewski, and Ruhé, are of modest to large size considering that we expect any 1 factor to be only 1 of many contributing to a condition such as depression.3-7
Unfortunately, the investigators performed only a review and did not complete a statistical synthesis of the data. Thus, the most we can draw from the review are nonquantitative impressions. Unfortunately, the impressions are not trustworthy given the misinterpretations of the statistics.
The review did not say anything about the possibility that derangements in serotonin functioning could be 1 factor among many that contribute to a loss of neuroplasticity, which may be an important unifying principle in understanding depression.
We should trust studies and reviews only when their methodology and statistics are sound. When the methodology and statistics are not sound, as in this case, the study and its conclusions must be treated with skepticism at minimum and potentially even disregarded completely.
Dr Moore is clinical associate professor of psychiatry at Texas A&M University College of Medicine and works at the Baylor Scott & White Health Mental Health Clinic in Temple, Texas.
References
1. Moncrieff J, Cooper RE, Stockmann T, et al.
2. Pies RW, Dawson G. The serotonin fixation: much ado about nothing new. Psychiatric Times. August 3, 2022.
3. Wang L, Zhou C, Zhu D, et al.
4. Nikolaus S, Müller H-W, Hautzel H.
5. Kambeitz JP, Howes OD.
6. Gryglewski G, Lanzenberger R, Kranz GS, Cumming P.
7. Ruhé HG, Mason NS, Schene AH.
Articles in this issue
almost 3 years ago
Tackling Treatment Issues in Adult ADHDalmost 3 years ago
Important Lessons for Psychiatry and Beyondalmost 3 years ago
Mental Health’s Most Toxic Mythalmost 3 years ago
Exploring DTx for Clinical Carealmost 3 years ago
COVID-19 and Mental Health: Global Consequences and CAM Approachesalmost 3 years ago
Exploring Integrative Medicine and Nutrition for PTSDalmost 3 years ago
Neurobiological Mechanisms of Mind-Body Medicinealmost 3 years ago
Herbal Medicine: What Psychiatrists Need to Knowalmost 3 years ago
Contemplating RetirementNewsletter
Receive trusted psychiatric news, expert analysis, and clinical insights — subscribe today to support your practice and your patients.