Exploring Meta-Analyses: An Example

(This article addresses methodological and statistical concerns about a recent study titled “The Serotonin Theory of Depression: A Systematic Umbrella Review of the Evidence.”¹ For further discussion, please see the analysis by Ron W. Pies, MD, and George Dawson, MD, in last month’s issue of Psychiatric Times™.²)

The brain is the most complicated thing in the universe. Problems with the brain, such as depression, are also complex and involve manifold factors including genes, other biological conditions, general medical conditions, past and current environmental exposures, and psychological capabilities and weaknesses. Multifactorial causation means it will always be hard to detect the signal of a single contributing cause. Lack of proof or imperfect proof of a derangement having a causal role is not proof of a lack of causation.

A good review is based on an exhaustive search for data. The review by Moncrieff et al looked at only 3 databases. When the authors found more than 5 reviews or large analyses, they included only the most recent 5. The investigators did not find a systematic review or meta-analysis on serotonin depletion within the last 10 years and so included only the 10 most recent studies.

Unfortunately, the review misrepresented some of the statistics.

First, see the Moncrieff et al Table 2, in which “No effect” was listed based on 95% confidence intervals. As described in the companion piece in this issue, confidence intervals shrink as the number of included studies increases. More importantly, confidence intervals measure only how well we have estimated the mean. Confidence intervals do not measure heterogeneity, which is what we want to know in interpreting whether a result might represent a meaningful effect in some populations.

Second, the review in question did not report the prediction interval, which is the statistic that represents heterogeneity. It may be that the underlying meta-analyses and studies sometimes did not report prediction intervals, but if so, that should have been noted to address heterogeneity.

Third, in the Moncrieff et al Table 1, the I2 statistic is listed as a measure of heterogeneity, but I2 is a ratio, not a measure of absolute heterogeneity.

Fourth, effect sizes are more important than statistical significance. The review dismissed findings as statistically insignificant by calculating p values relative to confidence intervals. However, as previously noted, the confidence interval is not even the correct statistic to test. In the Moncrieff et al Table 1, some of the effect sizes, such as those found by Wang, Nikolaus, Kambeitz, Gryglewski, and Ruhé, are of modest to large size considering that we expect any 1 factor to be only 1 of many contributing to a condition such as depression.^3-7

Unfortunately, the investigators performed only a review and did not complete a statistical synthesis of the data. Thus, the most we can draw from the review are nonquantitative impressions. Unfortunately, the impressions are not trustworthy given the misinterpretations of the statistics.

The review did not say anything about the possibility that derangements in serotonin functioning could be 1 factor among many that contribute to a loss of neuroplasticity, which may be an important unifying principle in understanding depression.

We should trust studies and reviews only when their methodology and statistics are sound. When the methodology and statistics are not sound, as in this case, the study and its conclusions must be treated with skepticism at minimum and potentially even disregarded completely.

Dr Moore is clinical associate professor of psychiatry at Texas A&M University College of Medicine and works at the Baylor Scott & White Health Mental Health Clinic in Temple, Texas.

References

1. Moncrieff J, Cooper RE, Stockmann T, et al. The serotonin theory of depression: a systematic umbrella review of the evidence. Mol Psychiatry. 2022;10.1038/s41380-022-01661-0.

2. Pies RW, Dawson G. The serotonin fixation: much ado about nothing new. Psychiatric Times. August 3, 2022. https://www.psychiatrictimes.com/view/the-serotonin-fixation-much-ado-about-nothing-new

3. Wang L, Zhou C, Zhu D, et al. Serotonin-1A receptor alterations in depression: a meta-analysis of molecular imaging studies. BMC Psychiatry. 2016;16(1):319.

4. Nikolaus S, Müller H-W, Hautzel H. Different patterns of 5-HT receptor and transporter dysfunction in neuropsychiatric disorders – a comparative analysis of in vivo imaging findings. Rev Neurosci. 2016;27(1):27-59.

5. Kambeitz JP, Howes OD. The serotonin transporter in depression: meta-analysis of in vivo and post mortem findings and implications for understanding and treating depression. J Affect Disord. 2015;186:358-366.

6. Gryglewski G, Lanzenberger R, Kranz GS, Cumming P. Meta-analysis of molecular imaging of serotonin transporters in major depression. J Cereb Blood Flow Metab. 2014;34(7):1096-1103.

7. Ruhé HG, Mason NS, Schene AH. Mood is indirectly related to serotonin, norepinephrine and dopamine levels in humans: a meta-analysis of monoamine depletion studies. Mol Psychiatry. 2007;12(4):331-359.