4: Capacity to Implement

4: Capacity to Implement web_admin

Activity 4:
Understand and have the capacity to implement evidence-based practices

For evidence-based decision making and practices to take hold, policymakers and professionals across the system must

What Do We Mean By "Research Evidence"?

In the justice system, the term "evidence" is used in a variety of ways. It can refer to items collected at a crime scene, eyewitness accounts, or security camera footage. These types of evidence are referred to as "legal evidence."

For the purposes of the EBDM initiative, however, the term "evidence" is used to describe findings from empirically sound social science research. The initiative refers to the results of this research as "evidence-based policy and practice."

  • share a common understanding of the key research findings underlying harm and risk reduction; and
  • possess a core set of skills, or competencies, to effectively apply these empirically based research findings to their day-to-day practices.

Through the assessment and bolstering of professionals' level of knowledge and skills, changes in attitudes, activities, and adaptation to new technologies occur, and pathways to constructive modifications in policy and procedure are opened—even in the face of unprecedented work demands and ever-tightening resources.

Elements of an EBDM justice system include

  • a common understanding of the research (and its limitations) across all relevant agencies/staff;
  • an understanding of the implications of these findings; and
  • a workforce skilled in effectively carrying out evidence-based practices.

4a: Understanding Your Agency: Conducting an EBP Knowledge Survey

4a: Understanding Your Agency: Conducting an EBP Knowledge Survey web_admin

Introduction

For a jurisdiction to be truly evidence-based, the professionals managing its justice system agencies[1]—and those working directly with the defendants and offenders in that system—should all share a common base of knowledge about key findings in the correctional literature. A knowledge survey can serve as a diagnostic tool to determine the level of current knowledge among these parties and potentially identify needs for knowledge enhancement. If it is determined that further knowledge development is needed, steps should be included in the jurisdiction’s strategic plan to enhance knowledge among key stakeholders and agency staff. This document provides a process for administering a knowledge survey and considering the implications of its results for the team’s future work.

Purpose

  • To assess the level of evidence-based knowledge among policy team members and of select groups of staff within their agencies.
  • To identify action steps to address gaps in knowledge as identified through the survey.

Participants

While the policy team will ultimately decide who will take the survey, it is recommended that knowledge survey participants include a sample of staff from all agencies represented on the team.

Instructions

Measuring the Impact of an Event

In Yamhill County, Oregon, the policy team used the EBP knowledge survey to measure changes in knowledge regarding EBP after a day-long awareness-building event for system stakeholders.

A greater percentage of respondents answered 17 out of 18 questions correctly following the event than prior to the event. These findings suggest that participation in this event had a positive impact on participants’ level of knowledge about various evidence-based practices.

  1. During a policy team meeting, discuss the reason(s) why a knowledge survey is important to the team’s work and how it will aid in the development of the team’s strategic plan.

    1. Determine what kind of knowledge (i.e., information that has been demonstrated to rise to the level of “evidence” and therefore be used to inform decision making) the team wants its members to know. From this feedback, either develop a knowledge survey or use the one in Appendix 1.
    2. If the policy team decides to develop its own knowledge survey, it is recommended that the survey be reviewed by a respected researcher and pilot tested with practitioners who are knowledgeable about the research before it is administered.
    3. Distribute the knowledge survey to policy team members.
    4. Survey respondents should not be coached in advance as to the specifics of the survey or possible answers. Results will only reflect accurate indicators of existing agency knowledge if the survey is administered without foreknowledge of content.
    5. Explain that not all agency staff will be surveyed. Instead, a selected group of policy and line level staff will be anonymously surveyed.
  2. Secure a commitment from each policy team member to fully participate in the survey process.
  3. Following the team meeting, discuss with each agency’s lead policymaker[2] which of their staff members should participate in the survey. It is recommended that only those who are in charge of policy decisions (i.e., directors/managers/supervisors) and those with direct contact with defendants, offenders, and victims be included in the survey.
  4. Calculate the number of staff persons who will participate in the survey, based on the criteria agreed to in #3. This will be the total survey “pool” for each agency. If possible, the survey should be sent to all individuals in this survey pool.
  5. The survey may be distributed in hard copy or through an automated online survey mechanism such as Survey Monkey.
  6. Communicate with agency staff in advance about the purpose of the survey. Some recommended email scripts are provided in Appendix 2.
  7. Once the survey results are compiled, review them during a policy team meeting.

    1. Determine how confident you are that the survey results reflect the knowledge of the pool of staff in each agency. Keep this in mind as you discuss the results. (For instructions on how to do this, see Appendix 3.
    2. Determine as a team the threshold for a satisfactory and unsatisfactory knowledge level.
    3. Determine whether the survey results were satisfactory (as determined by the team). If not, for which agencies were the results unsatisfactory?
    4. For those agencies with unsatisfactory knowledge survey scores, determine what action steps the team wants to take.
    5. If one or more action items are identified to enhance knowledge, consider, as part of your action plan, re-administering the knowledge survey after these efforts are completed to determine whether the knowledge gaps have narrowed. In this case, use the same process to determine the survey sample for re-administration as you used for the initial survey. The same individuals do not need to be surveyed; however, the same protocol for identifying the survey group must be followed.

Tips:

Using an EBP Survey with Agency Leaders

In Charlottesville-Albemarle County, Virginia, the director of the local community corrections agency used the knowledge survey with her statewide agency heads to test their knowledge of EBP principles and practices, and gave out prizes for the highest scores.

  • Make sure that survey participants understand the purpose of the survey. (See the email template below.)
  • Allow, at a minimum, two weeks (10 business days) for participants to complete the survey.
  • Send an email to the survey participants at least two days before the beginning of the two-week survey period to let them know that they are being asked to participate in the survey. (Make sure the survey is up and running early, in case participants want to fill out the survey immediately!) It is preferable that this email be sent by the agency policymaker or by another person in a leadership position. See Appendix 2 for ideas about what this email should contain.
  • Send out one reminder during the two-week survey period.
  • If, after ten business days, all of the surveys are not returned, send out another reminder indicating that, although the deadline has passed, it is important that each person participate and, therefore, the deadline has been extended for three more business days.

Additional Resources/Readings

The knowledge survey contains information collected from numerous research studies. See NIC’s Evidence Based Decision Making Framework[3] for research studies that include some of the key findings.

Appendix 1:
EBP Survey with Answer Key

 

Evidence-Based Practices Survey

Answer

1

True or false? Most offenders don’t handle stress well, so anxiety and stress reduction programs such as yoga and meditation are helpful in reducing the likelihood of rearrest.

F

2

Of the following, which is the most effective method to determine the likelihood that an individual will be rearrested?

  1. A validated assessment instrument
  2. Professional judgment by an individual with extensive experience
  3. None of the above; there is little evidence that an instrument or professional judgment predicts arrest patterns

A

3

Which of the following interventions best reduces recidivism in the long term?

  1. Boot camps
  2. AA
  3. Sanctions for non-compliance
  4. Programs that build thinking skills
  5. Victim empathy classes

D

4

To best achieve offender behavioral change, professionals should use social learning techniques that reduce recidivism. Which of the following is not a social learning technique?

  1. Modeling: demonstrating desired behavior
  2. Reinforcement: rewarding behaviors you want repeated
  3. Redirection: confronting and redirecting antisocial thinking
  4. Relationship: developing a meaningful working alliance with the offender
  5. Leadership: enhancing character through opportunities to demonstrate leadership
  6. Practice: teaching concrete problem solving skills by having the offender practice the desired skills

E

5

Which of the following statements is true?

  1. Validated risk assessment instruments predict the likelihood of reoffense better than professional judgment.
  2. Validated risk assessment instruments do not work as well as the judgment of an experienced, veteran professional.
  3. Validated risk assessment instruments are so consistently reliable that professional judgment should not be used to override the results of the instrument.

A

6

Which individual trait or circumstance, when present in a person’s life, does not likely contribute to whether that person commits a crime?

  1. Depression
  2. Friends who have low regard for the law
  3. Low victim empathy
  4. Unemployment

A

7

True or false? Placing offenders with low self-esteem in programs that increase their confidence does not reduce the likelihood of rearrest.

T

8

True or false? It is generally true that treatment does not work in reducing rearrest rates.

F

9

True or false? Lack of employment is among the top four influences that can lead an individual to commit a subsequent crime.

F

10

True or false? Lack of education is among the top four influences that can lead an individual to commit a subsequent crime.

F

11

Which statement is most accurate when trying to maximize the effectiveness of programming for medium and high risk offenders?

  1. Treatment should be intense over approximately a four-week period.
  2. Treatment is largely ineffective for this group of individuals.
  3. Treatment should last as long as possible—preferably 24 months or more.
  4. Approximately 200 hours of treatment time gets maximum effect.

D

12

True or false? Failure of a sex offender to register has a direct correlation to convictions for new sex crimes.

F

13

Which one of the following programs reduces recidivism over the long term?

  1. Gardening and horticulture
  2. Yoga
  3. Drum circles
  4. Lectures designed to give insight
  5. AA
  6. None of the above

F

14

True or false? It is better to invest in interventions for low risk offenders than for high risk offenders because their criminal tendencies are less hardened, or “fixed.”

F

15

True or false? Programs like “Scared Straight” and boot camps are particularly effective for youthful offenders between the ages of 16 and 25.

F

16

True or false? Jails and prisons are effective in changing future offender behavior after release if the conditions are severe enough that the offenders don’t want to return.

F

17

True or false? Giving offenders positive reinforcement and feedback when they exhibit prosocial behaviors supports positive changes in the future.

T

18

True or false? Of those probationers who are revoked from supervision and incarcerated, most are not revoked from supervision for new crime behavior.

T

19

True or false? Medium and high risk offenders benefit more from punishment than treatment.

F

20

Research suggests that the onset and use of drugs is different for female offenders than for male offenders. Which one of the following statements is not true about female offenders when compared to male offenders?

  1. Women describe the onset of drug use as sudden, rather than gradual.
  2. Women tend to report that their drug use begins as a result of a specific emotion or situation, such as depression or family problems.
  3. Women experience the adverse physical effects of alcohol more slowly than men.
  4. Women are more likely than men to be introduced to drugs by a sexual partner and to continue using drugs to maintain the relationship.
  5. Women who abuse drugs have higher rates of childhood physical and sexual abuse than men and non-substance-abusing women.

C

Appendix 2:
Email Templates for Conducting the Knowledge Survey


  • Email 1

    Send this email a few days before the survey period. (Make sure that the survey is up and running before you send this email!)

    Dear Colleague,

    We would like to ask you to complete a short, online survey. It should take no more than 10 minutes of your time. The purpose of this survey is to determine the extent to which staff in our organization share common knowledge about key justice practices that are supported by research.

    A number of policy level and line staff members have been selected to participate in this survey, including managers and those who work directly with defendants/offenders. Your answers will remain completely anonymous.

    To take the survey, please click on the following link: xxxx.

    The survey will be open for the next two weeks. If you have questions about how to take the survey, please contact Name and Email.

    Thanks in advance for your cooperation.

    Agency Leader Name

  • Email 2

    Send this reminder email about halfway through the survey period.

    Dear Colleague,

    If you have not completed the Evidence-Based Practices survey, please do so as soon as possible.

    To take the survey, please click on the following link: xxxx. Please complete this survey by: date.

    Thanks,

    Agency Leader Name

  • Email 3

    Send this reminder the day after the survey period ends.

    Dear Colleague,

    Although the deadline has passed, we feel it is important that each person participate in the Evidence-Based Practices survey and therefore the deadline has been extended for three more business days.

    If you have not already completed the survey, please do so by date by clicking here: xxxx.

    Thanks,

    Agency Leader Name


Appendix 3:
Determining the Minimum Number of Responses Desired for Accurate Inferences

In order to make more accurate inferences about the larger population, it is necessary that a certain percentage of the individuals in the target pool complete the survey. To determine the minimum number of responses necessary for confidence in the results, visit the following website: https://www.surveysystem.com/sscalc.htm#one. (This website provides a tool for determining survey sample sizes.)

Follow these instructions:

  1. Only use the box entitled “Determine Sample Size.”
  2. Click on the button marked “Confidence Level 95%.”
  3. Enter the number 10 in the “Confidence Interval” box.
  4. Enter the number of persons in the survey “pool” (as calculated in #4 above) in the “Population” box.
  5. Click “Calculate.” The number displayed is the minimum number of respondents that are necessary to get results that reflect the target population.

 

Once this process is completed for each agency, make a list of the survey targets for the policy team using a chart similar to the one below. This will help document the survey pool, the number of surveys received, and level of confidence in the results.

 

Agency

Total # of Staff in Agency

Total # of Staff in the Pool for Participation in the Survey

(from #4 above)

Minimum # of Responses Desired

# of Responses Received

Received More than the Minimum and Therefore Confident in Results?

Court

12

10

9

4

No

Prosecution

53

46

31

37

Yes

Police

324

217

67

121

Yes

 


[1] Throughout this document, the term “agency” is used to represent independent entities such as a police department, court, organization of public defenders, community corrections agency, etc.

[2] For the purposes of this document, the “lead policymaker” is the person on the policy team representing his/her agency (e.g., chief of police, chief public defender, chief judge).

[3] NIC's Evidence Based Decision Making Framework.

4b: Equipping Stakeholders to Apply Research Evidence

4b: Equipping Stakeholders to Apply Research Evidence web_admin

Introduction

A justice system that seeks to use evidence to guide decisions must have as its foundation a group of professionals (both policymakers and practitioners) who share a core base of knowledge about evidence-based practices and an appreciation for the usefulness of research to system, agency, and case-level decision making.

While agencies/individual professional groups necessarily promote knowledge and skill development that is unique to their disciplines (e.g., firearm instruction for law enforcement, interpretation of law for lawyers, safety practices for correctional staff), the research around evidence-based practices as it relates to risk and harm reduction is applicable to, and crosses, all disciplines. So too is the practice of using research to inform decisions—regardless of field or specific decision point.

These common areas of knowledge and the practice of applying research to decision making comprise a core EBP training curriculum and strategy that is appropriate for all justice system professionals. A core, systemwide evidence-based training strategy has several benefits. It

Developing a Learning Culture

Justice systems become learning cultures when they grow dependent on information, particularly empirically derived information. Policy debates that once centered on philosophical beliefs or one’s personal experiences are quite different when the core of the discussion is based on what the evidence does or does not suggest. Individual case decisions that are guided by science rather than intuition or past experience often result in different actions and, fortunately, better outcomes. Learning cultures begin with a common base of knowledge and an appreciation for science.

  • promotes the notion that criminal justice professionals are each an important part of a system, rather than operating in individual “silos” that have little bearing on one another;
  • opens lines of communication by creating a common language across discipline groups;
  • provides an opportunity for an open exchange of ideas, airing of concerns, and joint problem solving; and
  • creates a culture of knowledge development and growth—a learning system—whereby knowledge acquisition and its application are encouraged and reinforced. To this end, training and knowledge development is a dynamic process. It does not occur once or for one period of time and then come to an end. As systems grow and evolve, as new staff are hired, as experiences are acquired, and as new research emerges, learning cultures grow and evolve too.

Purpose

To equip professionals across the justice system with a core base of evidence-based knowledge and an appreciation for the application of research evidence to their decision making.

Participants

The policy team will ultimately take the lead in developing and delivering a core EBP training curriculum to local stakeholders. However, other staff may be brought in as needed (to assist with surveying, curriculum development, training, etc.).

Instructions

  1. Assess training needs and develop a knowledge baseline. The first step in developing and delivering a core EBP training curriculum is to assess knowledge regarding evidence-based practices across professional staff in all justice system agencies. This can be done fairly efficiently through the use of a brief survey or some other structured or semi-structured method.[1] The survey results will provide aggregated baseline data for each responding agency.[2]
  2. Review the knowledge survey results, identify information gaps, and build training plans. The results of the knowledge survey will provide clear and specific information about training needs across the system and within each discipline. This information should be used to guide the development of a systemwide training strategy and to inform agency-specific knowledge development efforts.
    1. In building a training strategy, keep in mind that although professionals across the system have a common need for core information, some will require more in-depth knowledge of some subjects while others may not.
    2. Also keep in mind that, in many areas, understanding the research and applying it requires the development of new skills. Skill-development training is quite different from knowledge-development training. It typically requires more time and always requires the opportunity to practice new strategies and techniques and receive feedback on the application of the new skills. The appendix illustrates some of the core EBP research topics and the stakeholder groups for whom each topic might be most relevant. Items in red are knowledge-based training objectives, while items in gray are skill-based training objectives.
  3. Customize the training to meet individual agency, as well as jurisdictional, needs. Development of a training strategy should take into account several questions:
    1. Should training delivery be directed to a single audience (e.g., judges), should it be directed to partially mixed audiences (e.g., judges, prosecutors, defenders), or should it be fully multidisciplinary? There are pros and cons to each choice. Keep in mind that when a core set of messages or base of knowledge is delivered, it is often beneficial for all stakeholders to receive this information at the same time. This helps support the goal of cross-agency collaboration.
    2. Should training be delivered by local staff/experts, external experts, or some combination of these? There are also pros and cons to each of these options. Regardless of the choice made, ensure that the selected trainers have sufficient knowledge and credibility with respect to the training topics that they will be received favorably by the audience.
    3. Should training cover basic information, advanced information, or both? The training needs assessment is likely to uncover some “unevenness” in the needs of cross-agency groups and within single-agency groups. Determining how to address individual needs within the context of a large number of potential trainees will be an important point to consider. Regardless of the final methods selected, be careful to avoid a “one size fits all” approach by assuming that what is right for one group will be right for all. It may be advisable to develop a customized needs approach—within the overall context of what is possible—by engaging the various stakeholder groups in discussions around how best to meet their needs.
  4. Conduct the training and then elicit feedback. Plan for the collection of post-training feedback prior to the training event. This could be in the form of re-administering the knowledge survey to determine if knowledge gaps were lessened (see below), an open discussion about the gaps filled by the training or lingering questions, and/or a written follow-up survey to determine if participants believe that the training goals were met and/or whether they identify additional training needs and, if so, which ones. Regardless of the method selected, feedback from the training will help determine where additional or more targeted training is needed. These needs should be incorporated into the systemwide training strategy.
    1. Re-administer the knowledge survey following the delivery of core EBP training material and compare the results with those of the initial survey to determine if knowledge levels improved sufficiently or if additional training is necessary. Then develop a “next stage” training plan, as needed.
    2. Identify methods to assess evidence-based knowledge over time and continue to build an ongoing, systemwide training strategy.

Example:
Knowledge Survey Summary Report from One County

The following chart is an excerpt from a larger report that summarizes the percentage of respondents in a select number of agencies who provided the correct answer to each question in an EBP knowledge survey.[3] This report highlights the differences between four agencies.

Question

Agency 1

Agency 2

Agency 3

Agency 4

All Respondents

 

N=20

N=16

N=22

N=106

N=228

  1. Most offenders don’t handle stress well, so anxiety and stress reduction programs such as yoga and meditation are helpful in reducing the likelihood of rearrest.

25%

88%

45%

59%

58%

  1. Of the following, which is the most effective method to determine the likelihood that an individual will be rearrested?

85%

100%

55%

39%

50%

  1. Which of the following interventions best reduces recidivism in the long term?

100%

94%

45%

67%

65%

  1. To best achieve offender behavioral change, professionals should use social learning techniques. Which of the following is not a social learning technique that helps reduce recidivism?

45%

38%

40%

13%

20%

  1. Which of the following statements is true about the use of validated risk assessment instruments to determine risk of reoffense?

90%

75%

62%

52%

59%

  1. Which individual trait or circumstance, when present in a person’s life, does not likely contribute to whether that person commits a crime?

65%

69%

43%

35%

45%

  1. Placing offenders with low self-esteem in programs that increase their confidence does not reduce the likelihood of rearrest.

5%

56%

33%

35%

34%

  1. It is generally true that treatment does not work in reducing rearrest rates.

95%

69%

81%

73%

75%

  1. Lack of employment is among the top four influences that result in whether an individual commits a subsequent crime.

10%

19%

30%

13%

18%

  1. Lack of education is among the top four influences that result in whether an individual commits a subsequent crime.

15%

63%

14%

21%

23%

  1. Which statement is most accurate when trying to maximize the effectiveness of programming for medium and high risk offenders?

21%

7%

10%

10%

10%

  1. Failure of a sex offender to register has a direct correlation to convictions for new sex crimes.

85%

38%

91%

31%

44%

  1. Which one of the following programs reduces recidivism over the long term?

55%

69%

38%

61%

57%

  1. It is better to invest in interventions for low risk offenders than for high risk offenders because their criminal tendencies are less hardened, or “fixed.”

90%

87%

60%

53%

56%

  1. Programs like “Scared Straight” and boot camps are particularly effective for youthful offenders between the ages of 16 and 25.

80%

81%

48%

47%

51%

  1. Jails and prisons are effective in changing future offender behavior after release if the conditions are severe enough that the offenders don’t want to return.

95%

94%

86%

62%

72%

  1. Giving offenders positive reinforcement and feedback when they exhibit prosocial behaviors supports positive changes in the future.

100%

100%

100%

85%

90%

  1. Of those probationers who are revoked from supervision and incarcerated, most are not revoked from supervision for new crime behavior.

75%

63%

71%

66%

64%

  1. Medium and high risk offenders benefit more from punishment than treatment.

95%

100%

91%

65%

71%

  1. Research suggests that the onset and use of drugs is different for female offenders than for male offenders. Which one of the following statements is not true about female offenders when compared to male offenders?

40%

31%

33%

20%

27%

Appendix:
Targeting Core EBP Knowledge Areas to Individual Stakeholder Groups

Targeting core ebp knowledge areas


[1] See 4a: Understanding Your Agency: Conducting an EBP Knowledge Survey for one example of a tool that can be used.

[2] See the Appendix for an illustration.

[3] For more information, see 4a: Understanding Your Agency: Conducting an EBP Knowledge Survey.

4c. Becoming a Better Consumer of Research

4c. Becoming a Better Consumer of Research web_admin

Introduction

The EBDM Initiative seeks to help local policy teams find and understand evidence-based knowledge about effective justice practices and to design more effective responses to defendants and offenders.[1] Many stakeholders already know how to find and use research; others will appreciate these tips regarding how to quickly access reliable research and how to review and understand the findings and their applications. The evidence or empirical studies will be drawn from many fields: evidence-based practices in criminal justice, behavioral health interventions, organizational development, leadership and management, effective collaboration processes, and cost–benefit analyses.

Purpose

Defining EBP

"Evidence-based practice is the use of direct, current empirical evidence to guide and inform effective and efficient decision making and supervision practices."

Broadly speaking, the goal of this document is to increase policy officials’ and practitioners’ skills in finding the research that matters and in understanding and translating empirical findings for their use in improving policy and practice. Specifically, this document offers

  • tips for finding research relevant to critical questions about evidence-based practice;
  • a list of searchable databases on criminal justice topics; and
  • advice on how to review and assess the quality of the findings in academic articles and the research literature.

Participants

This document was developed for EBDM policy teams, their work groups, and agency practitioners to enhance their ability to find and understand the best available research that may be applied to criminal justice problems and proposed solutions.

Instructions

Step 1: Look in the Right Places to Find the Evidence that Matters

Where should the discerning consumer begin the search for evidence-based policies and programs and answers to specific research questions? The answer is three-fold: the Web, written literature, and experienced colleagues from your local and state criminal justice systems and from national networks of professionals.

Websites that Filter the Information for You: Evidence-Based Program Databases

Websites designed specifically to summarize research in one or more criminal justice practice areas are an excellent place to begin the search for information on effective programs and policies. A growing number of government agencies, academic institutions, and professional groups maintain these databases as a service to criminal justice professionals and the public. These organizations

  • formulate evaluation criteria for assessing the strength of research findings;
  • employ experts to review multiple studies of research on programs in a single area; and
  • indicate which programs are shown to be effective (and at what level of rigor or confidence).

Some of these websites specialize in “systematic reviews” (also called meta-analytic reviews) of the literature regarding specific research questions and program areas. As the Center for Evidence-Based Crime Policy at George Mason University explains, systematic reviews “summarize the best available evidence on a specific topic using transparent, comprehensive search strategies to find a broad range of published and unpublished research, explicit criteria for including comparable studies, systematic coding and analysis, and often quantitative methods for producing an overall indicator of effectiveness.”[2]

A partial list of evidence-based program databases in criminal justice follows:[3]

A Website Caution

The consumer of website research summaries should be careful to not take the information at face value. Definitions as to what constitutes evidence, methodological soundness, and robust findings can vary significantly.

Furthermore, researchers do not always agree on what can be concluded from a research study. While some website authors make transparent attempts to give the user an accurate description of research findings, it is up to the user to exercise judgment. It is recommended that the user seek corroborating information to increase confidence in the relative strength of the research and its implications.

  • The Campbell Collaboration, The Crime and Justice Coordinating Group (CCJG) is an international network of researchers that prepares and disseminates systematic reviews of high-quality research on methods to reduce crime and delinquency and to improve the quality of justice. https://www.campbellcollaboration.org/
  • The Center for the Study of the Prevention of Violence, University of Colorado, maintains a website, Blueprints for Violence Prevention, on evaluated programs to prevent adolescent violence, aggression, and delinquency. https://www.ncjrs.gov/pdffiles1/ojjdp/187079.pdf
  • George Mason University’s Center for Evidence-based Crime Policy offers a number of services, including systematic reviews, research on crime and place, and a summary (matrix) of evidence-based policing practices.
  • Substance Abuse and Metal Health Services Administration’s (SAMSHA) National Registry of Evidence-based Programs and Practices (NREPP) provides a database of more than 190 interventions supporting mental health promotion, substance abuse prevention, and mental health and substance abuse treatment.
  • U.S. Department of Justice, Office of Justice Programs’ Crime Solutions’ website provides research on program effectiveness; easily understandable ratings (effective, promising, and no effects) that indicate whether a program achieves its goal; and key program information and research findings. https://crimesolutions.ojp.gov/

Websites that Provide Bibliographic Databases

These websites, which provide a listing of hundreds of studies, are often maintained by government agencies and universities. Prominent among these in the criminal justice field are the following:

Websites that Provide Summaries of Research and Practical Guidance

Some universities, state criminal justice agencies, and professional organizations also run websites that summarize the research on effective criminal justice practice and/or provide guidance to users. While not as extensive as bibliographic databases, these websites focus their publications on the critical issues of most concern to policymakers and practitioners. A partial list follows:

  • Center for Evidence-Based Crime Policy. https://cebcp.org/
  • Correctional Treatment Evaluations, Texas Christian University, Institute for Behavioral Research. This national research center for addiction treatment studies in community and correctional settings provides access to over 700 resources on its website. https://ibr.tcu.edu
  • National Implementation Research Network. This website contains research on the successful implementation of new processes within organizations and systems. http://nirn.fpg.unc.edu/
  • Stanford University, Evidence-Based Management. This website specializes in evidence directly related to the management of agencies. https://cebma.org/
  • University of Cincinnati School of Criminal Justice. This university-based site contains a number of research studies regarding the use of evidence in correctional interventions. https://cech.uc.edu/schools/criminaljustice.html
  • Washington State Institute for Public Policy. This website contains a number of helpful studies on what is or is not an effective intervention for reducing recidivism and costs. It is perhaps best known for its cost–benefit studies. http://www.wsipp.wa.gov/

Your Colleagues

Often an efficient way to check out the results of web-based and library searches is to ask experienced colleagues in your state and local jurisdiction and in national networks for recommendations regarding the latest and most reliable research. This strategy helps triangulate or hone in on the best studies.

Further, when identifying a journal article that appears useful but for which a subscription is required, contact colleagues at nearby colleges and universities and inquire about their ability to access the article from their library and provide a single copy for your review. (Be careful to not copy, distribute, or otherwise violate copyright laws.)

An increasing number of states support websites that summarize evidence-based research and practical guidance that is directly relevant to their criminal justice constituents and agencies. The websites may be hosted by a state criminal justice agency or university. Your colleagues will know how to access these sites.

Step 2: Evaluate Research Quality

What criteria should be used to decide if program evidence has been collected and analyzed according to high quality research standards? As Hess and Savadsky (2009) emphasize in their article “Evaluation Research for Policy Development,” all evidence is not created equally. Familiarity with a few key concepts can help policymakers wade through the growing body of information and make better-informed decisions about what is reliable. Following are a few tips about how to read the research literature and evaluate its quality:[4]

All Research Is Not Created Equal

"The golden rule here is to recognize that everything promoted as ‘research’ is not equally reliable or useful." – Hess & Savadsky, 2009

  • Understand the target population of the study and consider its relationship to the target population under consideration in your jurisdiction. Pay attention to sample size and sample selection. In general, larger samples provide more reliable data; however, there is no one hard and fast rule about sample size. The sample size may vary according to the purpose of the study, overall population, sampling error, and so forth.
  • Consider the context. What works in one place or for one population may not work for another (e.g., a study completed in a small, rural state with unique characteristics may not be applicable to a large, densely populated state with a different offender profile and justice system challenges). In addition, the context of one study cannot necessarily be transferred to other settings. An often-quoted study examined successful program results and found that 15% of the outcome was derived from the intervention itself (e.g., cognitive program, didactic intervention, or therapeutic community) and 30% from the working alliance with the individual providing the service.[5] However, the study was not carried out with correctional clients. The results could be valid across populations but until that hypothesis is tested, caution must be exercised about its applicability to the correctional population.
  • Be cautious about assertions of causality. Correlation does not mean causation; an intervention may be related to a certain outcome but may not be responsible for that outcome. For example, a significant portion of many communities’ offender population includes individuals with mental illness. A common assumption is that mental health treatment will reduce the likelihood of reoffense among this population. However, while a mental health condition should be treated, studies have shown that mental health treatment alone is unlikely to reduce recidivism.
  • Recognize that changes in implementation can change the outcomes of an intervention. For instance, an effective probation intervention that relies on officers proficient in motivational interviewing, case planning, and problem solving with clients may not work as well if delivered by staff who do not possess these skills.
  • Be sure the conclusions follow logically from the reported findings. The summaries or conclusions of some studies can be deceptive or take license in explaining the implications of findings. Consumers should look for research that “measures the impact of particular interventions on identifiable populations under controlled circumstances.”[6] These studies offer prescriptive guidance about actions that can be consistently replicated elsewhere.
  • The issue of confidence in results is important. The research consumer needs to know if the results of the intervention are “statistically significant.” This refers to the likelihood that a result is caused by something other than mere chance. In general, a 5% or power p-value is considered statistically significant. While policymakers may not want to dig through the statistical results’ section in great detail, it is useful to check whether the article mentions that the findings are statistically significant. Other issues such as whether the person(s) conducting the research study has a vested interest in the outcome of the study and whether the study was replicated elsewhere should also be considered.[7]

Additional Resources/Readings

Hess, F. M. & Savadsky, H. (2009). Evaluating research for policy development.

Fink, A. (2008). The research consumer as detective: Investigating program and bibliographic databases. Practicing Research: Discovering Evidence that Matters (pp. 33–64). Retrieved from https://www.sagepub.com/sites/default/files/upm-binaries/19270_Chapter_2...

Wampold, B. E. (2001). The great psychotherapy debate: Models, methods, and findings. Mahwah, NJ: Lawrence Erlbaum Associates.


[1] In Appendix 3 of the Framework for Evidence-Based Decision Making in Local Criminal Justice Systems, the Initiative provides a matrix of research findings on reducing pretrial misbehavior and offender recidivism. EBDM policy teams are encouraged to review this resource; however, the EBDM Research Matrix can only provide a snapshot of the research at one point in time, as new research is continually conducted. Therefore, this Starter Kit document is intended to provide EBDM policy teams with additional guidance on how to keep current with the research on EBDM.

[3] Adapted from Fink, 2008.

[4] Adapted from Hess & Savadsky, 2009.

[5] Wampold, 2001.

[6] Hess & Zavadsky, 2009.

[7] See Hess & Zavadsky (2009) for more information on how to be a good consumer of research.