Risk Assessment Promise and Peril: The Colorado Experience

By David S. Prescott, LICSW

Background

The state of Colorado has long been at the forefront of attempts to develop effective methods for coming to terms with the risks posed by people who have sexually abused. In the days when each county or jurisdiction seemed to have different approaches, Colorado implemented the Containment Model. When there were no actuarial measures or other tools for structured professional judgment for grounding assessments, the Colorado Sex Offender Management Board (SOMB) assembled a list of 17 factors, which are a focus of the study below. For purposes of the study, these 17 items are treated as an alternative assessment measure, which was apparently not the original intention, although many evaluators have doubtless treated them as such. For its part, the SOMB is well aware that these 17 items are no longer the final word in assessments, even as they still receive consideration.

It seems important to note this background context, as Colorado’s efforts have indeed been pioneering over the years. In retrospect, it can seem easy to criticize the pioneering developments of groups of professionals. However, it should not be forgotten that when knowledge was scarce and approaches to sex crime resembled the Tower of Babel across the US, Colorado was among the first to develop approaches that numerous other states have emulated. Just the same, there is much we can learn from the study of these approaches, which is the subject of this blog.

The Research

An Online-First study by Katharine McCallum, Marcus Boccaccini, and Claire Bryson in the journal Criminal Justice and Behavior, offers fresh insight into the practical application of risk assessment research. The abstract describes their findings succinctly:

In Colorado, evaluators conducting sex offender risk assessments are required to assess 17 risk factors specified by the state’s Sex Offender Management Board (SOMB), in addition to scoring actuarial risk assessment instruments. This study examined the association between instrument scores, the 17 SOMB risk factors, and evaluator opinions concerning risk and need for containment in 302 Colorado cases. Evaluators’ ratings of risk indicated by noninstrument factors were often higher than their ratings of risk indicated by instrument results, but only their ratings of noninstrument factors were independently predictive of containment recommendations. Several of the most influential noninstrument factors (e.g., denial, treatment motivation) have been described by researchers as potentially misleading because they are not predictive of future offending. Findings highlight the need for more studies examining the validity of what risk assessment evaluators actually do, as opposed to what researchers think they should do.

This is not the first study finding that professionals often over-estimate risk across a range of conditions. The authors provide an eye-opening literature review, and Dr. Boccaccini has elsewhere found that the results of evaluations are often swayed according to who is paying for the service. For a context in which evaluators consider 17 items originally developed by the SOMB as a part of their evaluations over and above the far more scientifically proven actuarial measures, it is not surprising that evaluators would give extra weight to the SOMB measure and the items within it. In reading the study, several points become clear:

First, the evaluators in Colorado seem to face a difficult assignment, having historically assessed risk using items shown in research to have no predictive utility. What is the evidence-based assessor to do? Among the most heavily weighted items in the SOMB measure are defensiveness, psychopathology, and level of empathy, which are famously not associated with risk (and therefore with summary risk ratings), but are very likely strong responsivity factors to consider. This leads to questions as to what kinds of risk is actually being assessed, risk for sexual re-offense or risk for problematic adjustment to the conditions of community supervision. If it is the latter, perhaps the findings in this study might be more understandable – even appropriate – if the SOMB tool became more of a measure of risk, need, and responsivity? In this way, risk for sexual re-offense would be evaluated as a first hurdle, with treatment needs and the ability of the examinee to respond to treatment as the second and third hurdles of a more comprehensive assessment.  Whatever the case, this study suggests that many evaluators were not pursuing evidence-based approaches in making recommendations related to detention; this should be of concern to anyone interested in effective policy and human rights.

Adding to the complexity of the task, many of the SOMB items most considered in evaluations seem to overlap with items in actuarial scales such as Static-99r, the VRAG, and SORAG. Examples include criminal history, offense history and victim choice, and the nature of the person’s social support system. All of these lead to questions about conceptual double-dipping; how many times does one review criminal history before assessment results become skewed?

At the risk of appearing to be a Pollyanna, it is at least encouraging to see as much use of empirically validated measures as there is. It wasn’t that long ago that risk was assessed with little structure in the process and low accountability for the examiner (e.g., even including the physical attractiveness of the examinee). Although these findings point to much hard work ahead for professionals and policymakers alike, we can at least take heart that our methods have improved in many jurisdictions.

Just the same, the apparent conflation of responsivity and risk factors should cause any professional or lawmaker to be concerned. This comes along with the persistent overestimation of risk, and the means by which conclusions take shape. Further, as Boccaccini’s other research has shown, biases can enter the assessment process through any number of ways, whether explicitly or beyond the awareness of the examiner. This study reminds us that, for all of the rich scientific evidence at our disposal, we are still human beings, subject to being judgmental, opinionated, and biased.

Extending this last point further, one of the most interesting findings in this study was also one of the least explored. In the authors’ words: “In the current study, evaluator differences accounted for 8% of the variance in SOMB summarized risk ratings and 21% of the variance in summarized actuarial risk ratings” (p. 13). In other words, who the evaluator is can be a highly variable part of the equation. For all of our attempts to – and bluster about – the importance of impartiality, we have yet to reach the goal of remaining objective. In some cases, this may be an artifact of using relatively vague items. In other words, evaluator bias may even be akin to the famous country song: “Ya gotta dance with the one that brung ya.”

Final Implications

Reviewing both the study and the Colorado experience itself brought to mind a number of important reminders:

-          First, although these findings echo related findings elsewhere, it is still a single study

 

-         Second, it is always important to keep in mind that our best measures and best policies are always subject to bias at the hands of the individuals involved. Our ultimate work should be in the direction of professional self-development and consistency across groups of professionals.