What Different Research Methodologies Tell Us

By Jeff Sandler, Research Scientist New York State office of Mental Health & ATSA research committee

This is a blog posting by the ATSA research committee. Kieran

 

A few weeks ago, an entry was posted to the ATSA blog entitled “Hearing the Narrative, Seeing the Person: Considering the Appropriate Research Methodology.” The piece discussed the importance of using various research methodologies to investigate questions surrounding sexual offending and treatments for sexual offending. The central thesis of the piece was that no single research methodology, or style of research, should be used for all studies on topics related to our field. The ATSA Research Committee would like to support the importance of using different research methodologies when studying a topic, but also offers a note of clarification and caution.

The ATSA Research Committee is, not surprisingly, very pro-research. As the blog from a few weeks ago correctly points out, there are many different types of research and research designs: large-sample, small-sample, cross-sectional, longitudinal, experimental, archival, prospective, retrospective, quantitative, qualitative, and many more. Each different type of study allows for different conclusions to be drawn from its results.

 

The blog from a few weeks ago argued that case studies and qualitative research have an important role to play in the study of sexual offending and the treatment of sexual offending. The Research Committee would like to echo that sentiment. There is no question that case studies and qualitative research play an important role in the research process.

 

A note of caution needs to be struck, however, about what the results of case studies and small-sample studies can tell us and how the results of such studies can be used. Specifically, the results of case studies and small-sample studies cannot be assumed to generalize to other people or populations.

 

That is, showing that a particular treatment model or intervention was effective for one, or five, or ten people is just that: A finding that one, or five, or ten people benefitted from a particular treatment model or intervention. It is not an indication that the treatment model or intervention is likely to benefit larger groups of people (particularly groups with different cultural backgrounds or from different geographic settings). To be able to make statements about the likelihood of an intervention being effective with other groups, quantitative, inferential studies are required.

 

Furthermore, the results of case studies and small-sample studies almost never have confirmatory or disconfirmatory value. That is, the data from such studies, as usually conducted in our field, cannot be used by themselves to either validate or refute a particular belief. Not only is it incorrect to cite a single case or a handful of cases as proof that a certain treatment model or intervention works, but it can also actually be dangerous to do so. This is something that we, who work in the field of sexual offending, have seen firsthand and all too often. Time and again, we have seen legislatures draw up and enact well-intentioned, but ultimately failed, laws based on a single case or a handful of cases. This is how we ended up with public policies such as community notification and residence restrictions that likely would never have been enacted had lawmakers consulted and heeded the findings of empirical quantitative studies.

 

With so much research being conducted and published these days, it is important to understand what we can and cannot learn from various types of studies. Case studies and qualitative studies are most important in the context of discovery: exploring a new line of inquiry or further understanding the lived experiences of participants. For those purposes, case studies and qualitative studies are invaluable. For clinicians, they offer possible avenues of treatment to explore once all evidence-based treatment options have been exhausted. For researchers, they offer a hypothetical roadmap, indicating places where further quantitative research is both warranted and needed. Case studies and qualitative studies should not be used, however, to assess either the effectiveness of treatment models or interventions that have been shown by quantitative research to be effective or ineffective, or to test a particular theoretical model.

 

One final cautionary note the Research Committee would like to offer relates to confirmation bias, the tendency to seek out and favorably view research that supports our beliefs and discount research that refutes our beliefs. Confirmation bias is a natural human tendency. We all want to believe that what we believe is correct, that what we are doing is good. As such, it is easy to look for findings that support our convictions and to search for reasons to discount the findings of studies that do not support our beliefs or actions. It is problematic, however, to hold up a case or a handful of cases as evidence that a particular empirical finding is wrong or that the finding does not apply to us and our unique circumstances. Case studies are valuable and can stimulate empirical research studies that researchers can and should conduct, but they are a starting point and cannot replace the findings of rigorous empirical research.

 

Being evidence-based means being open to the results of all rigorously-conducted research, even research that contradicts our beliefs or tells us that we should be doing things differently. It means being vigilant and explicit about the certainty and uncertainty of our claims, interpretations, conclusions, and recommendations. It means recognizing that although almost all research methodologies can contribute some insight, that does not mean that all research should be considered equal in quality or in the weight given to it. It means recognizing that informed speculation is valuable and necessary to bridge the gap between what has been demonstrated by available evidence and what has not yet been rigorously tested, but that speculation should not be used to circumvent available evidence.

Being evidence-based is tough and occasionally unpleasant. It means being open to the idea that we are wrong, that what we believe is not accurate, and that we should adjust our actions accordingly.