Measuring Outcome in Psychiatric Private Practice Using Outpatient Self-Reports

Outcome assessment in a private practice

APS consists of 36 adult and pediatric psychiatrists, 34 therapists, and 45 nurses in 11 psychiatric offices, situated in rural, suburban, and urban settings across Alabama. Care is provided for patients from a pool of 1 million covered individuals, about 22% of the population of Alabama. Services include office care, partial hospitalization and intensive outpatient treatment programs, and inpatient services.

All APS patients over 18 years were asked to complete the assessments; no exclusion criteria were applied. The mean age was 36.9 and just 2% were older than 60 years. Women made up nearly two thirds (64.4%) of the sample, a proportion that held roughly true within all diagnostic categories except substance abuse disorders, in which 69.6% were men. Insurance was provided through either the patient’s or a family member’s employer and many were self-referred. A depressive disorder was diagnosed in more than 51% of patients and anxiety disorders were diagnosed in an additional 14.7%. The complete distribution of psychiatric disorders is shown in the Table.

Our decision about what to measure was guided by the facts that: (1) selfreferred adults present most often because of subjective distress; and (2) anxiety and depression represent the most frequent symptoms across disorders. Therefore, anxiety and depression change scores were chosen to assess the overall improvements of all patients. Other clinicians/practices will make different choices about what to measure, based on philosophical/practical differences (eg, clinicians working with a population of the seriously mentally ill might choose parameters such as quality of life, hope, or functioning⁹).

We elected to collect data directly from patients, rather than use clinician rating instruments such as the Hamilton Rating Scale for Depression.¹³ This decision was driven partly by practical considerations because clinician rating scales are generally time-intensive and not useful for routine repeated assessments. ¹⁴ Seligman¹⁵ supported the use of a survey based on self-assessment, arguing that: (1) self-report is the basis of clinical diagnosis to begin with, especially in mental health; and (2) the correlations between self-report and diagnosis are usually quite high. Moreover, in the era of evidence-based medicine, especially given our consumer-driven health environment, increasing emphasis is put on the patient's own impressions of improvement, choice of treatment, and satisfaction with treatment.

It was then crucial to find a self-rating instrument that could be seamlessly integrated into the clinical routine, while being sensitive, reliable, brief, and well tolerated. The Quick PsychoDiagnostics (QPD) Panel¹⁶ is a fully automated true/false test with good psychometric properties that is given on a handheld electronic unit and takes an average of 6.2 minutes to complete. It is used at Kaiser Permanente and other primary care environments.

Internal logic allows this test to customize questions based on the response to previous questions (ie, healthier patients are spared irrelevant questions and more ill patients are further assessed). Thus, the QPD is both a categoric diagnostic instrument that gives diagnostic suggestions based on DSMIV criteria, as well as a dimensional one that measures symptom severity.¹⁶

The test screens for 7 mental disorders, with sensitivity ranging from 69% to 98%, and specificity ranging from 90% to 97%, when compared with the Structured Clinical Interview for DSMIV- TR.¹⁶ When the test unit is docked to a computer, a report provides numeric scores with a graphic illustration indicating the severity of depression and anxiety. As with the depression scale, change scores of 5 points on the anxiety scale on retest are considered clinically significant.

Our group purchased the QPD (including the program with an annual licensure fee and the computer boxes) with our own funds, enabling us to use it independently. Our information systems technicians programmed a spreadsheet database used for internal trending reports and for the outcome study. In our practice, we use the QPD as a measure of severity and diagnostic suggestion; actual diagnoses are made by board-certified psychiatrists. Patients complete the assessment in the waiting room, and change scores are routinely reported to the clinician before the actual visit.

Outcome

First, we focused on the depression selfrating scores over time to illustrate the outcome data that emerged from this approach. Results were obtained for 10,648 adult patients seen in the practice from July 2001 through December 2004, who completed at least one follow-up assessment; 6432 of these patients completed a second follow-up assessment. To give an idea of the results, we report here on patients whose condition was diagnosed as “major depression, single episode.” The Figure shows outcome (both improvement and recovery) at 2 time points for these patients—55.8% had improved to a clinically significant degree at Time 2 (after about 150 days), and a subset of those, 38.4% of the total sample recovered (depression scale in normal range in addition to clinically significant improvement). At Time 3 (after about 280 days), these figures increase to 65.3% and 47.7%, respectively.

Results for diagnoses not shown in the Figure show relatively minor deviations. Nevertheless, these deviations increase confidence in the validity of our results, since they appear to follow clinically meaningful patterns. For example, persons with a diagnosis of “major depression, recurrent” recover less frequently than the entire sample, and patients with adjustment disorders recover more frequently.

Comparing our results to others is not straightforward because of significant differences in methodology, case mix, and improvement criteria. These variables may account for the wide range of reported rates of recovery from major depression in outpatients (sustained improvement in 39% to improvement in 76%.^17-19 Yet Posternak and associates²⁰ observed a striking similarity in recovery rates between the Collaborative Depression study²¹ (41% at 13 weeks and 54% at 26 weeks) and their small sample private practice study (38% at 13 weeks and 57% at 26 weeks). Our own results (we report on both improvement and recovery) are similar to these findings, and show that outcome measurement in a large psychiatric private practice is both possible and meaningful.

Limitations

Although our data show that it is indeed possible to routinely assess and monitor patients' well-being and treatment effectiveness in a large practice using a computer-based patient self-rating instrument, more information (eg, which treatment modality, medication, psychotherapy, or combination was most successful for which type of patient; control groups; correlation with external ratings) would clarify the results on improvement and remission. Nevertheless, these effectiveness data support the assumption that real-world patients with psychiatric symptoms of depression and anxiety do in fact improve, similar to results in efficacy studies. This is encouraging, since more negative findings have also been reported: a recent review of the effectiveness of community-based psychotherapy services for children found only 9 scientifically sound studies, with an average effect size close to zero.²²

Another important aspect is that our practice group functions as a full-risk capitation model, so our reimbursement is not affected by an individual physician's performance. Indeed, we only share group outcome statistics with insurance companies; no individual physician’s results are reported externally. Internally, the goal is to improve the practice as a whole, not to use the information to penalize an individual physician. However, in a fee-for-service environment (perhaps even with the pay-for-performance model), there may be relevant ethical concerns about how much detail of such outcome information is shared, even when this information appears to be highly valid.

Conclusions

We believe the mandate to evaluate outcome will eventually reach all psychiatrists who are not paid exclusively from a patient's own funds. Our protocol may serve as a model or jumping-off point for psychiatrists wishing to add outcome evaluation to their practice. Our experience shows that we can do this at reasonable cost, we encourage others to forge ahead now and reap the clinical benefits, while potentially contributing to policy (how pay-for-performance will evolve) and scientific knowledge.

Dr Oepen is assistant medical director for clinical services and medical education at Alabama Psychiatric Services, PC; clinical professor of psychiatry at the University of Alabama, Birmingham; and research affiliate for the consolidated department of psychiatry at McLean Hospital of Harvard Medical School.

Dr Federman is a psychologist in private practice and on the faculty of Boston University School of Medicine; he is also a consultant on statistics and outcome evaluation at Alabama Psychiatric Services.

Dr Akins is President and Medical Director of Managed Healthcare Administration, Inc in Birmingham, Alabama, and Medical Director and CEO of Alabama Psychiatric Services, PC.

The authors report no conflicts of interest concerning the subject matter of this article.