Critical Review of Evidence-Based Program Repositories/Registers for Behavioral Health Treatment
Sponsor: U.S. Department of Health and Human Services | National Institutes of Health
Principal Investigator: Dr. Stephen Magura
Co-Principal Investigator(s): Dr. Daniela Schroeter, Dr. Chris Coryn
July 1, 2011 – June 30, 2014
Identifying “what works” in the behavioral health disciplines is crucial for making well-informed defensible decisions when selecting and implementing behavioral health-related interventions. As the need for relevant, accessible, and systematically derived information that supports policy decisions has increased, so has the demand for comprehensive sources of information about effective programs and intervention modalities. To this end, a number of “evidence based” repositories and registers have been developed by various federal and state organizations, NGO’s, and universities.
Unfortunately, the proliferation of potentially useful repositories/registers has presented policy-makers and practitioners with ambiguous, inconsistent, and incomplete recommendations for putative evidence-based interventions, since these databases often have dramatically different purposes, criteria for inclusion, definitions of acceptable “evidence,” and standards for designating interventions as “effective.” In order to provide policy-makers and practitioners with information on how best to use evidence-based repositories/registers in the decision-making process, this study comprehensively and critically examines the strengths, weaknesses, and limitations of these evidence-based repositories/registers.
2. Classify those EBPRS according to their purposes, methodologies, acceptable types and standards of evidence, and other factors that are used to include and certify “effective” interventions.
3. Determine the practical consequences of using different types and standards of evidence for including and certifying interventions as effective.
Potential study outcomes
This study seeks to accomplish four main goals:
1) Identify the varying definitions of program efficacy, effectiveness, and readiness for dissemination used in evidence based practice.
2) Gain understanding of assumptions about what constitutes sufficient empirical evidence related to “what works” in the behavioral health disciplines.
3) Understand the criteria used to assess the credibility of the evaluation or research studies that support inclusion in evidence based repositories.
4) Understand and decompose the standards used to judge the absolute or relative effectiveness of the constituent interventions and modalities of intervention contained within evidence based repositories.
Our progress and lessons learned so far
At this time, we have completed all phases of the study, and are preparing our manuscripts for submission to the Journal of Evaluation and Program Planning, edited by Jonny Morrell.
Presently we have identified several key findings:
- The evidence-based registers we have found varied somewhat in their structure, funding sources, content areas, and purposes.
- Typically the registers focused on several key user groups: decision makers, practitioners, and researchers
- Maintaining current reviews tended to be a time and resource challenge cited by most of the registers.
- The registers tended to agree that a “Campbellian heirarchy” (Gugiu, in press) was the best way to frame evidence> This type of model privileges evidence developed with well controlled RCTs over evidence derived from other means.
- Although the registers tended to accept the Campbellian approach, they often differed in the ways they applied criteria for evidence quality, leading to discrepancies in program ratings across registers.
- Most registers tended to use word of mouth marketing, and could use a broader dissemination strategy.
We will be publishing two articles, cited here:
Schröter, D.C., Burkhardt, J.T., Magura, S., Means, S.N., Coryn, C.L.S. (2014). A Comparison of Evidence-Based Registers for Behavioral Health. Journal of Evaluation and Program Planning. Manuscript accepted for publication.
Means, S.N., Magura, S., Burkhardt, J.T., Schröter, D.C., Coryn, C.L.S. (2014) Comparing Rating Paradigms for Evidence-Based Practice Registers in Behavioral Health. Journal of Evaluation and Program Planning. Manuscript accepted for publication.