Measuring Outcome in Psychiatric Private Practice Using Outpatient Self-Reports

Increased demand for accountability is requiring more clinicians to supplement their judgments of patient outcome with standardized and objective protocols. In 2004, Massachusetts required its behavioral health contractor to have all its providers conduct outcome assessments.¹ At the federal level, the Health Care Finance Authority mandated in 1998 that contractors include outcome evaluations² and now Medicare is moving rapidly toward differential payments based on performance and outcome.³

Although the pressure for standardized assessment comes from external forces, clinical and scientific benefits will result. Repeated assessments will enable dynamic adjustment of treatment, using either formal algorithms of evidence-based practice or individualized clinical pathways; and such assessments will help answer the key question in all health interventions are our patients getting better?

This question is not easy to answer in real world (ie, nonacademic) psychiatry, ^4,5 where patients differ greatly in compliance, comorbidity, and other parameters that are well controlled in academic research. Moreover, dissemination is imperfect. Even when evidence-based educational interventions (EBEI) are delivered, the impact on clinician behavior varies greatly.^6,7 For example, a recent study showed that only half of the clinicians who received EBEI training applied the knowledge, while half had a “knowledge-behavior gap.”⁸ Thus, truly unfiltered translation of best practices from the laboratory to the field is only rarely possible.⁴

Since treatment and patient factors in real-life practice environments vary from those in laboratory studies,⁹ knowing whether real-world patients get better requires assessment in the realworld environment. Knowledge about what works in the real world (effectiveness) would complement the increasing knowledge about what works in the laboratory (efficacy).

Outcome evaluation in the real world

Outcome evaluation has not been widespread in psychiatric practices in the United States, because the costs have generally been viewed as outweighing the benefits. Even when outcome evaluation has been instituted by public agencies, it has sometimes been curtailed because of fiscal constraints.¹⁰ When such evaluation becomes required, providers have objected to the increased demands, since they are unaccompanied by sufficient fees.¹ As psychiatrists face increasing pressures ranging from reduced reimbursement (with compensatory increase in the number of patients seen) to increased paperwork and regulations, the time to measure outcome becomes ever harder to carve out. Moreover, the decision to evaluate outcome subsumes more demanding detailed questions,^11,12 including:

Methodologic

• What variables should be measured level of distress (level of anxiety or depression), functioning (work, school, or social situations), or quality of life?
• Should the same data be collected from all patients or individualized (are the data from patients with agoraphobia the same as from patients with major depression)?
• Data source—patient (self-report instrument), significant other, or clinician (structured interview, clinician rating scale).
• When should the data be collected pretreatment, frequency thereafter, termination, follow-up.
• Properties of instrument—reliability, validity, existence of appropriate norms, sensitivity, specificity, ability to measure change.

Practical

• Costs of instrument, copyright, licensing fees.
• Ease and time to complete—how well tolerated is the instrument and how will this contribute to compliance and accuracy?
• Method of data collection, storage and analysis (all electronic vs all paper and pencil vs combination).

Finally, as Klein and Smith⁴ point out, “if patient evaluation is not unobtrusively incorporated into normal clinical activities, it is impossible to know if outcome findings are an artifact of the supervening assessment process.”

It is no surprise, then, that psychiatrists and the mental health profession in general have not rushed to embrace outcome evaluation. Presently, the fragmented system of health care delivery and reimbursement in America (fee–for- service, HMOs, capitated models, public sector) provides varying incentives and disincentives to conduct outcome evaluation. Capitated practices that compete for contracts, based on service, outcome, and price, provide a strong incentive to assess outcome.

Alabama Psychiatric Services (APS), a multioffice private practice providing psychiatric care for more than 1 million covered individuals with a capitated model has been routinely doing outcome evaluations for 5 years. To illustrate one way to consistently integrate meaningful outcome evaluation into clinical care, we show how APS addressed the methodologic and practical questions outlined above and describe our protocol for data collection.