A Reader Comments on CBT and Hypnotics

In his Putting Research Into Practice column, “Practical Implications of a Study on Treating Chronic Insomnia" (Psychiatric Times, Dec 2009, Vol XXVI, No 12, p 8), Dr Rajnish Mago described a study of cognitive-behavioral therapy (CBT) and hypnotics in 160 subjects with chronic insomnia, 15% with a comorbid psychiatric diagnosis. Subjects were divided into 4 groups for analysis, and 80% completed the study. This study had the methodologic problems of (1) unblinded for both medication and CBT; (2) no psychotherapy control arm (ie, supportive psychotherapy); (3) no placebo control for drugs; (4) over-stratifcation into 4 analysis groups—assuming 80% completion for each group—leaving 32 subjects per group; (5) psychiatric illness in 15% (ie, not all had primary insomnia) further confounding the already stratified groups.

Dr Mago reported the major analyses of the study and whether they were “statistically significant.” He then went on to mention some of the problems with the study: no medication placebo, possible recruitment bias in ads for subjects, not enough subjects for statistical confidence, and the dose and type of hypnotic were not flexible. At the end, he states, “Despite convincing data, CBT for insomnia is used very infrequently.” It is not clear if he is referring to the study in his article as “convincing data,” or to the other 21 “small” studies he mentions (suggesting they are not powered properly, and likely to be unblinded and uncontrolled).

Dr Mago also omitted the problem of lack of a psychotherapy control that would help elucidate whether CBT has specific effects on insomnia or whether the general aspects of psychotherapy (eg, human connection, support, and advice) are at work in helping patients with insomnia.

The FDA would never allow an “unblinded,” non-placebo controlled, over-stratified, and non-statistically confident (powered) low subject study to be a pivotal (phase 3) study for psychiatric drug approval. Most drug pivotal studies have hundreds of subjects in them. (For example, in another article in the December issue of Psychiatric Times (Bender, Vol XXVI, No 12, p 24), a phase 2 study (eg, a non-pivotal pre-phase 3 study) of an antidepressant was described as placebo-controlled and with 400 subjects.

Although Dr Mago rightly encourages us to use “nuanced” interpretation in our analysis, it is unclear why he is even considering a discussion of this seriously flawed study. This article would be misleading to a reader not familiar with clinical trials—not to mention that some of the results were hard to interpret, further questioning the rigor of the study—but this requires more space than available here.

Psychiatrists must insist that psychotherapy efficacy studies be held to the same methodologic rigor as pivotal drug studies if they are to be used to make any conclusions other than that the study was seriously flawed. Looking at the analysis of a flawed study to see if it fits with past studies or clinical experience is itself a flawed and misleading way to evaluate the results of a poor study.

Doug Berger MD, PhD
Dr Berger is in private practice in Japan.


DR MAGO RESPONDS:
I would like to thank Dr Berger for his careful reading of my column and his comments regarding methodological issues in interpreting the study discussed. I appreciate this opportunity to elaborate on and add to my discussion in the column.

I propose that the productive approach is to ask what, if anything, can be concluded from a study, given that it was conducted as it was. In thinking about any study design issues and about the significance of any so-called “bias” in the study design, I suggest that our task is made easier by thinking about these in the context of the research question(s) being asked. For example, in the first part of the study, patients with chronic insomnia were divided into 2 treatment groups—CBT only and CBT with hypnotic—to answer the question of whether there was any additional benefit to adding a hypnotic to the CBT in the acute phase (or later). Does absence of a control group for the CBT significantly affect our interpretation of the results? Since both groups received CBT and since the study did not address whether CBT works due to specific or non-specific factors, I would argue that it does not.

Does the absence of a double-blind placebo corresponding to the hypnotic significantly affect our interpretation of the results? Well, why would we need blinding or a placebo group to compare the efficacy of the hypnotic to? We would need it if addition of the hypnotic had been found to be significantly effective, since without a placebo group to compare the hypnotic group to we wouldn’t know if the improvement was simply due to a placebo effect. In this study, addition of the hypnotic was not found to be helpful in the acute phase; the absence of a placebo group does not prevent us from drawing that conclusion.

The problem with dividing any study sample into small subgroups for analysis is that, as Dr Berger rightly points out, each subgroup may have too few patients. I have no problem with the common practice of having enough patients to answer the primary questions of a study with statistical confidence, but to compare some subgroups for further exploring the data for possible subsequent studies. However, 3 of the 4 main questions addressed by the study involved subgroup analyses. Two comparisons were statistical, and if a comparison between subgroups leads to a statistically and clinically significant finding, the smaller numbers don’t matter. I will remind Dr Berger that the “power” of a study is only meaningful before as a study is done.

A third comparison was not statistically significant, but the proportions (85% vs 88%) were so close together that it would have not been clinically significant, even if it had been statistically significant. This left the fourth comparison that was not statistically significant though the proportions looked different (68% vs 42%). If the difference between 2 subgroups is found not to be significantly different from a statistical point of view, then (as I had noted in the column) “…an alternative explanation could be that this is a real difference, but this particular study did not have enough patients to demonstrate it with statistical confidence.”

Regarding the 15% comorbid psychiatric disorder to which Dr Berger refers, this was in the past; he is not correct in saying that not all had primary insomnia. As noted in the column, “Patients whose insomnia was secondary to another specific illness (eg, progressive medical illness, a medication adverse effect, current major depressive disorder, sleep apnea, restless legs syndrome) were excluded.” Thus, I don’t see how this significantly affects our interpretation of the study. In fact, given that past history of depressive disorder or anxiety disorder is common in the population, excluding even patients with past history of such problems would unnecessarily reduce the generalizability of the study.

Space prevents me from going into a fuller discussion of how clinical trials of psychotherapy may have to differ from drug trials (I hope to do so in one of my columns in the near future), but here let me first say that this study did not either aim to assess the efficacy of CBT for chronic insomnia, let alone assess whether such efficacy is due to “specific” effects or non-specific effects. Thus, a psychotherapy control was definitely not required. Secondly, as clearly stated in the column, the meta-analysis of 21 small studies addressed not the question of the efficacy of CBT for chronic insomnia per se (as Dr Berger seems to think) but whether or not a hypnotic has any advantage over CBT. For reviews of studies demonstrating the efficacy of CBT and other psychotherapeutic techniques in the treatment of insomnia, please see the 2 reviews commissioned by the American Academy of Sleep Medicine which reviewed studies with a total of about 2000 patients per review.^1,2

For the reasons stated above, I cannot agree that this study was “seriously flawed.” It did answer several of its questions with statistical confidence and draws the attention of readers to a potentially important and largely neglected form of treatment. It is not true that only very large sample sized, double-blind, and placebo-controlled studies are worth reading. Like most things in life—it all depends.

I hope that readers of this response will find that it helps them to some extent in furthering their understanding of how clinicians should read and interpret research studies.

Rajnish Mago, MD
Dr Mago is director of the Mood Disorders Program at Thomas Jefferson University in Philadelphia.