Jeremy Matuszak, MD

Nearly 50 years ago, psychiatric diagnosis was described as the “soft underbelly” of psychiatry because of poor inter-rater agreement.

 Historically, psychiatric diagnoses were interpretive and somewhat subjective. Relative to medicine as a whole, psychiatry has few laboratory markers or imaging data to confirm a preliminary diagnosis. However, subsequent iterations of DSM have aspired to more accurate, reliable diagnoses. Despite this, the inter-rater reliability of psychiatric diagnosis remains a challenge as American psychiatry prepares to welcome DSM-5.

Inter-rater (or intercoder) reliability is a measure of how often 2 or more people arrive at the same diagnosis given an identical set of data. While diagnostic criteria help establish reliable diagnoses, the methods of gathering and interpreting patient data have a tremendous effect on how likely it is that 2 examiners will come to the same diagnostic conclusion for a given patient. Here we review inter-rater reliability research, identify factors that threaten diagnostic reliability, and offer examples of assessment strategies that may provide increased diagnostic reliability in general clinical settings.

Reliability of psychiatric diagnoses today

In diagnostic assessment, perfect inter-rater reliability would occur when psychiatric practitioners could always arrive at the same diagnosis for a given patient. When evaluating and interpreting inter-rater reliability, the measurement statistic used is kappa; the higher the kappa ranking, the stronger the degree of agreement is between raters. Indeed, no field of medicine is able to achieve perfect agreement-there are degrees of variance among diagnosticians in other specialties.

We are not saying that psychiatric diagnoses wholly lack objectivity, nor do we dismiss the “art” of psychiatry. Dependent on the skill of the rater and category of disorder, the inter-rater reliability of psychiatric diagnoses can be quite high and comparable to the kappas obtained for several diagnoses in other medical specialties. For example, Ruskin and colleagues

 demonstrated a kappa of 0.83 for 4 common psychiatric diagnoses. However, lacking some of the more objective measures enjoyed by other specialties, psychiatry faces additional challenges with inter-rater reliability.

Similar to goals of other fields of medicine, a central goal of psychiatric nosology is diagnostic reliability and consistency. Despite this goal, however, diagnostic reliability for some psychiatric disorders remains low. For example, studies have revealed a strong discordance among the diagnoses of psychotic disorders. McGorry and colleagues

documented 66% to 76% inter-rater reliability for this class of disorders and a corresponding discordance of 24% to 34%. Likewise, Maj and colleagues

 demonstrated a low kappa of 0.22 for the diagnosis of schizoaffective disorder.

Despite the ongoing revision of diagnostic criteria with each subsequent iteration of DSM, clinicians report concerns that diagnostic reliability generally remains poor. In a small-survey study in 2007, Aboraya 

 asked how clinicians’ attitudes and beliefs might explain low concordance in psychiatric diagnosis. In contrast to a similar study done in 1962, clinicians in the 2007 study attributed low inter-rater reliability more to individual clinician factors than to problems with nosology.

 Clinician-related factors that affect psychiatric diagnosis are heterogeneous, and the following is by no means a complete list.

Differences in clinical experience may affect knowledge about psychiatric phenomenology or alter the landscape of the clinical interview. For example, comfort and familiarity with diagnostic criteria, confidence in interviewing, and time management can each affect diagnostic conclusions. A loosening of the diagnostic criteria may occur if the clinician only asks about some of the diagnostic criteria. Availability of data (eg, collateral information) may also affect the ultimate diagnosis. Other important sources of inter-rater divergence are external pressures, such as different reimbursement practices by insurance companies, exposure to popular diagnoses through the media or pharmaceutical marketing, and the cognitive biases and heuristics that affect the practice of all medical specialties.

Clearly, interview styles vary widely and no single approach is the right one. Clinical interviews are often informal with variable structure. The aid of rating scales or structured instruments, therefore, can limit omissions or biases during history taking. While some validated structured instruments (eg, the Structured Clinical Interview for DSM Disorders-Clinician Version [SCID-CV]) require too much time for administration to be practical in everyday clinical work, others (eg, the Diagnostic Interview Schedule for Children [DISC]) may be successfully administered by lay providers or by computer, allowing the clinician to subsequently focus on one or more identified areas of concern.

 Similarly, systematic screening instruments are available for all major categories of psychiatric illness.

In light of these data, would an approach that offers more structure and objectivity increase the reliability of diagnoses? We offer 3 examples of approaches that may improve inter-rater reliability:

• A structured approach to collateral data

• The use of validated instruments to collect patient information

• A cognitive approach that anticipates critical challenges to diagnosis

The general clinician may not routinely seek outside information to test the accuracy and validity of a patient’s self-report, especially in outpatient settings. Yet clinicians should be cautious about obtaining most or all of the diagnostic data directly from the patient interview. Clinicians may empathize with patients even when they find a patient’s self-reported history to be implausible. In contrast, the regular, careful use of collateral information may reduce or limit biases. Opinions and diagnoses based on reviews of relevant outside material, interviews with people who know the patient and, at times, measures to screen for malingering or biased response style may provide a broader, more relevant and reliable history. If important data are missing, such as a drug screen, specifically noting diagnostic uncertainty created by the missing information will help shape the differential diagnoses.

Of course, patients who seek clinical care do not present with a box brimming with past records, and clinicians typically do not have time to sort through volumes of collateral data. Yet, in our experience, collateral data improve the reliability of diagnoses because they offset some of the problems associated with a patient’s recall and report of past events. For example, a patient may describe a past diagnosis of bipolar disorder from a recent hospitalization, but a review of past records might indicate that the patient’s discharge diagnosis was related to substances and that mood problems were characterized as a “rule out bipolar disorder versus substance-induced mood disorder.”

One practical solution is to structure the method for obtaining collateral information. Intake questionnaires are an opportunity to inquire comprehensively about current symptoms, obtain information on past treatment and hospitalizations, and obtain the names of a few people who know the patient well. Release of information forms and laboratory orders for screening tests may be paired with the intake questionnaires. Including past treatment records and laboratory results in the chart allows for a review before the follow-up appointment and critical reflection on the initial impressions from the patient’s self-report. Phone calls to the personal contacts, when consent is given, allow for further clarification, but always keep in mind the contacts’ own agendas or opportunities for secondary gain. If structured into the clinical routine, collateral information will increase the likelihood that a diagnosis is based on a full set of data.

The third approach is the mindset that anticipates a challenge to the diagnosis or scrutiny of review by an outside party. Child psychiatrists prepare reports that are scrutinized by parents, social services, juvenile justice professionals, and payers. Forensic psychiatrists prepare opinions with cross-examination in mind. As a diagnostic opinion is formulated, a number of questions arise:

• How good is the evidence that supports this diagnosis? What are the limits of the evidence?

• Is there another legitimate interpretation of the same data?

• Could the individual’s psychiatric symptoms be better accounted for by the use of substances?

• Was the individual (attorney or parent) invested in a particular diagnosis and could the self-report be biased?

• How does payer source affect the ways in which data were obtained and weighed?

Anticipation of these challenges counteracts bias and likely improves the probability of a reliable diagnosis.

DSM-5 presents psychiatry with a potential “reset button” for diagnostic reliability. Our field has an opportunity not only to revisit diagnostic criteria but also to revisit the ways clinicians can apply these criteria in the most valid and reliable fashion. Checklists have improved clinical outcomes in many other fields, but to date, psychiatry has not established a widely accepted or implemented checklist for screening and diagnosis.

 Perhaps the advent of DSM-5 and the studies of field reliability will lead to a new set of tools for the general psychiatrist’s toolbox. In the meantime, increasing collateral data, using validated screens, and using critical reflection on diagnostic evidence and influences may allow for increased confidence in diagnostic reliability and validity.

 Giffen MB, Kenny JA, Kahn TC. Psychic ingredients of various personality types. 

 Pies R. How “objective” are psychiatric diagnoses?: (guess again). 

 Ruskin PE, Reed S, Kumar R, et al. Reliability and acceptability of psychiatric diagnosis via telecommunication and audiovisual technology. 

 McGorry PD, Mihalopoulos C, Henry L, et al. Spurious precision: procedural validity of diagnostic assessment in psychotic disorders. 

 Maj M, Pirozza R, Formicola AM, et al. Reliability and validity of the DSM-IV diagnostic category of schizoaffective disorder: preliminary data. 

 Aboraya A. Clinicians’ opinions on the reliability of psychiatric diagnoses in clinical settings. 

 Ward CH, Beck AT, Mendelson M, et al. The psychiatric nomenclature. Reasons for diagnostic disagreement. 

 Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. 

 First MB, Spitzer RL, Gibbon M, Williams JB. 

Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-I), Clinician Version, User’s Guide

. Arlington, VA: American Psychiatric Press Inc; 1997.


 Shaffer D, Fisher P, Lucas CP, et al. NIMH Diagnostic Interview Schedule for Children Version IV (NIMH DISC-IV): description, differences from previous versions, and reliability of some common diagnoses. 

Articles

DSM-5 presents psychiatry with a potential “reset button” for diagnostic reliability.

Inter-Rater Reliability in Psychiatric Diagnosis

Mary, aged 40 years, was referred for psychiatric evaluation out of concern that a mental health diagnosis was interfering with her ability to appropriately and safely care for her child. The patient had stated on numerous occasions that her 9-year-old daughter, Sarah, had been placed in the custody of Child Protective Services and replaced by an imposter. Mary also reported that she had given birth to twins but that the hospital record only documented the birth of Sarah. On occasion, she had shown up at her daughter’s school, refusing to pick her up, screaming, “Give me my real daughter; I know what you’ve done.”

Despite reassurance from multiple health care providers and relatives, Mary continued to express concern that her daughter was not, in fact, her own. Furthermore, she related several episodes in which her daughter was “whisked away before I could talk to her” while going about her daily business. For example, Mary related that a car driven by an unknown person, with Sarah in the passenger seat, passed by her while she was running errands but sped away once she was noticed.

Following Mary’s initial evaluation, a delusional disorder-Capgras syndrome-was diagnosed.

The results of this evaluation indicated that Mary’s condition was indeed a delusion, because it was a fixed false belief that persisted over time. Attempts to redirect or foster the development of insight were in vain. Given the patient’s suspiciousness regarding her daughter’s true identity, social services questioned Mary’s ability to parent effectively.

Although medication was prescribed (risperidone 2 mg by mouth at bedtime), Mary responded poorly, and her delusion persisted. Her medication was subsequently titrated to a total dose of 4 mg/d, also with poor response. Mary’s adherence with the prescribed medication regimen was questioned, but the intervals between prescription refills and pill counts failed to suggest that she was nonadherent.

After 2 months of regular follow-ups and medication management, she continued to display poor insight into the nature of her illness. She refused to entertain the possibility that Sarah was truly hers or the possibility that her thought processes were distorted.

At that time, the Department of Social Services sought guardianship of her daughter until Mary could be deemed fit to parent. Following the hearing, her daughter was placed into the custody of the Department of Social Services. In response, her daughter replied, “I love my mother except for when she doesn’t believe I’m me. . . .” During visitations supervised by Social Services, Mary refused to talk with her daughter, maintaining that she was not Sarah and that the real Sarah had been hidden away from her.

Mary was believed to be suffering from a specific delusion identified as Capgras syndrome. This condition was first described by Joseph Capgras and Reboul-Lachaux in 1923 and was later named for Joseph Capgras. The syndrome describes a delusion (or fixed false belief) in which the affected individual believes that another person, generally a family member or close acquaintance, has been replaced by a look-alike imposter. It is important to note that Capgras syndrome is not a DSM-IV-TR diagnosis, but it is most often found as part of another underlying disorder, such as schizophrenia or dementia of the Alzheimer type.

The specific cause of Capgras syndrome has been hypothesized from neuropsychological and psychodynamic views. Current neurological studies have focused on similarities between Capgras syndrome and prosopagnosia, a neurological disorder in which damage to the right ventromedial occipitotemporal areas in the brain causes impairments in the ability to recognize faces.

 Some psychodynamic approaches have looked at ambivalence and hostility the delusional person may have toward the imposter, either directly or indirectly as the cause of the delusion. Severe anger toward the imposter can lead to the use of splitting as a defense mechanism. In this way, the patient can preserve some of his love for the close acquaintance while still expressing anger by saying that he loves the true person and is only mad at this “new faker.”

A case has been described in which a mother believed her adult daughter had been replaced with a look-alike imposter. On delving into the patient’s history, it was found that the patient had to leave a very serious relationship with a man she loved because she was pregnant with another man’s child. The baby eventually grew into the adult daughter who the patient believed had been replaced. The patient’s resentment for having to leave her lover when she was pregnant was believed to be the psychodynamic source for the current delusion.

With Capgras syndrome, the family member who is believed to have been replaced is most often a spouse, parent, or sibling. For unknown reasons, the “replaced” family member is rarely the child of the delusional person and even more rarely is the child younger than 20 years.

 Although violence aimed at any person is a significant threat, violence aimed at children is particularly worrisome. Mary’s case is unique because the increased potential for violence in her relationship with her daughter needs to be taken into consideration when assessing the patient’s ability to be a safe and effective parent.

Although violence can be seen in all psychiatric disorders, there is a higher incidence of severe violence in patients with delusional disorders.

 In patients with Capgras syndrome, the violence is often directed at the imposter or, in some cases, the people the patient believes replaced the loved one with the imposter.

 found several demographic features that increase the likelihood of violence in persons with Capgras syndrome. Specifically, if the imposter lives with the patient or if the delusional person is male, has a persistent and long-term delusion, or has a history of violence or substance abuse, the risk of violence is increased. The sources of violence can be frustration or fear of the imposter, but it can also be cultural.

 found that some folklore and regional legends suggest that if a child is thought to have been replaced by another person or even by a demon, battering and being physically aggressive toward the imposter might bring the “real” child back. One Swedish fairy tale recounts the story of a woman who believed her child was an imposter. In the story, she is advised to put her baby into a hot oven; when she does this, her “true” child is returned.

 This is a severe case that is not necessarily the norm for patients with misidentification delusions; however, it is evidence that violence in delusional persons can happen.

When the patient believes that his or her child has been replaced and when violence is a possibility, the termination of parental rights and the removal of the child from the patient’s care must be considered. The clinician should consult his local state statutes and laws to determine the rationales and processes for termination of parental rights specific to the case and area. Generally, a local court must find that removing parental rights would be in the best interest of the child. Furthermore, this decision must be based on evidence of abandonment or neglect of the child or a risk of serious physical, mental, or emotional injury to the child while in the custody of the parent. Minimal effort by the parent to support the child, prevent neglect, and avoid being an unfit parent, and to remove risks of injury (physical, mental, or emotional) to the child are also grounds for termination of parental rights.

The dangers a delusional parent poses to his children are significant and must be taken seriously. In Mary’s case, the court system granted temporary guardianship of her daughter to Social Services, because she was seen as unfit to parent. Some insight into the psychosocial stress leading to the development of this disorder would provide an area to begin psychotherapeutic treatment, but her history failed to reveal any acute stressors. Unfortunately, treatment with pharmacotherapy alone was minimally effective, underlining the fact that 50% or fewer patients with a delusional disorder improve.

 Given the very real risks associated with this case, Sarah was placed into a safe environment until her mother responds to treatment, if ever.

 Talley BJ, Michels L. Two patients who think their family members have been replaced by “imposters.” 

Fishbain DA, Schiffman J. The daughter as the principal “double” in Capgras’ syndrome: psychodynamic correlates. 

Kaplan and Sadock’s Synopsis of Psychiatry: Behavioral Sciences/Clinical Psychiatry

. Philadelphia: Lippincott Williams & Wilkins; 2003:511-520.


 Bourget D, Whitehurst L. Capgras syndrome: a review of the neurophysiological correlates and presenting clinical features in cases involving physical violence. 

 Silva JA, Sharma KK, Leong GB, Weinstock R. Dangerousness of the delusional misidentification of children. 

 The specific cause of Capgras syndrome has been hypothesized from neuropsychological and psychodynamic views. 

Author | Jeremy Matuszak, MD

Articles

Inter-Rater Reliability in Psychiatric Diagnosis

That’s Not My Child: A Case of Capgras Syndrome