The DSM5 Field Trial Proposal-An Expensive Waste of Time
Time is running out on DSM5 and the mistakes keep piling up. The latest puzzling misstep is the design for the DSM5 field trials. The APA will conduct a remarkably complex and expensive reliability study to determine whether 2 raters can agree on a diagnosis. It will devote enormous resources to answer a question that once mattered greatly but is now of quite limited interest. Meanwhile, DSM5 will perversely avoid the one question that does really count: ie, what will be its likely impact on the rates of psychiatric diagnosis? At least $2.5 million and 1 year later (or possibly 2, if things get delayed as I expect they will), DSM5 will still be flying completely blind on the safety of its proposals.
Time is running out on DSM5 and the mistakes keep piling up. The latest puzzling misstep is the design for the DSM5 field trials. The APA will conduct a remarkably complex and expensive reliability study to determine whether 2 raters can agree on a diagnosis. It will devote enormous resources to answer a question that once mattered greatly but is now of quite limited interest. Meanwhile, DSM5 will perversely avoid the one question that does really count: ie, what will be its likely impact on the rates of psychiatric diagnosis? At least $2.5 million and 1 year later (or possibly 2, if things get delayed as I expect they will), DSM5 will still be flying completely blind on the safety of its proposals. I will list the 6 seriously disabling limitations of the DSM5 field trials-saving the absolutely fatal flaw for last:
1. The current criteria sets are so sloppily written that their reliability will be far below what could be achieved with clear and consistent criteria.
2. It is already well established that good to excellent diagnostic reliability can be achieved by trained raters using a semistructured interview to assess well written criteria sets, but that reliability is generally much lower under average clinical conditions. Determining the reliability of the DSM5 proposals will at best confirm this and is therefore of very limited value. At worst, (done poorly as is likely given the many problems with the study design) the field trials will achieve unnecessarily low reliabilities that will undercut the credibility of psychiatric diagnosis.
3. Because the DSM5 field trial will be conducted in an unfocused, scatter shot fashion covering a large array of both new and existing diagnoses, it will enroll relatively few patients with the new diagnoses that most require testing. This creates a huge signal to noise problem. Despite all the work and expense, the field trial will wind up lacking the power to provide much information about the changes that matter most.
4. Since the reliability studies will be conducted in just a few highly specialized academic centers, the results will not generalize well to the real world.
5. The design calls for remarkably complex and unweildy logistics (eg, approval by 10 IRB’s, recruiting thousands of patients, 3 visits per patient with likely high drop out rates, coordinating patient visits with clinician schedules, videotaping etc) and cannot possibly be done in the time allotted. My first prediction made 10 months ago was that DSM5 couldn’t possibly meet its then May, 2012 publication date. This was proven true when the Task Force was forced to ask for an additional year. The extra time has been used very inefficiently. I predict now that the current 2013 deadline will likely also slip away under the burden of this complex and essentially useless field trial.
6. Which brings us to the fatal flaw that makes this field trial such an enormous waste of time, effort, and money. Its design completely evades the one real question that needs to be answered before any suggested change can be safely accepted into DSM5: ie, will this cause a large wave of overdiagnosis? The major suggestions for DSM5 are the introduction of new disorders and the lowering of diagnostic thresholds for a number of most common existing disorders. These changes could potentially misidentify tens of millions of false positive “patients,” many of whom would receive unnecessary treatment. It would be irresponsible to conduct a field trial that makes absolutely no attempt to measure the likely DSM5 impact on prevalence and overdiagnosis.
How could DSM5 have gotten the field trial design so wrong? The reasons are familiar- the same fundamental problems that resulted in the flawed DSM5 drafts have also contaminated the field trial design. All the work on DSM5 suffers from a combination of pervasive secrecy and always being late to the gate. Early plans for the field trial design should have been prepared and posted one year ago in parallel with the work of drafting criteria. This would have allowed wide scrutiny from the field and the improvements that always come from iterative revision. Instead, we have a back room proposal thrown together at the last minute that really misses the point and will be a $2.5 million white elephant.
What should DSM5 do now? Ironically, a more appropriate field trial method would be much less expensive and time consuming. It would consist of a telephone survey sampling the general population to determine the impact of suggested changes on prevalence. Any questions that require testing in a clinical setting should be focused exclusively on new suggestions, not on the existing system. Before any testing is done at all, the criteria sets must be cleaned so that they are worthy of testing. Anything less and I doubt that a usable DSM5 will ever really make it to the gate.
