Purpose

The purpose of this guide is to support the high-fidelity implementation of the Public Safety Assessment (PSA). The PSA is an actuarial assessment that estimates the likelihood that people will fail to appear in court pretrial, have a new criminal arrest while on pretrial release, and have a new violent criminal arrest while on pretrial release. Use of the PSA, in combination with other pretrial advancements, is associated with a variety of outcome improvements.

The benefits of the PSA are many, including providing pretrial decision makers with consistent, relevant, research-supported data that informs pretrial decision making. A growing body of evidence shows that release decisions based on research are fairer and more equitable. Importantly, in addition to these benefits, use of the PSA can promote consistency and transparency across decision makers and provide important data that contributes to understanding pretrial practices and outcomes, including those that inform the analyses of racial and ethnic disparities.

Audience

This manual was designed for use by jurisdictions that have previously implemented the PSA or that are in the process of doing so. It is consistent with one of APPR’s core principles: that pretrial assessment, policies, and practices that are grounded in rigorous research and refined through ongoing performance measurement offer the greatest promise for success. The manual was written for a broad group of people from governmental, community, and other organizations (hereafter, “stakeholders”) who, working through a collaborative, consensus-based process, agree to implement the PSA. The manual presupposes that a high-functioning policy team will oversee both the implementation and ongoing fidelity aspects of local PSA use, and that small groups of individuals—perhaps but not necessarily including policy team members—will undertake certain aspects of the planning, implementation, and fidelity assessment.

Contents

This manual serves as a companion to a variety of materials that Advancing Pretrial Policy and Research (APPR) developed to support successful and effective implementation of the PSA. It is assumed that readers are familiar with these materials, which are available on this website, including an overview of and detailed information about the seven-phase implementation process. Jurisdictions implementing the PSA are expected to comply with the guidance in all Implementation Guides, the terms of use, and other updated recommendations.

This guide addresses five of the seven phases and nine core aspects of PSA implementation:

Each section of the guide includes a brief description of the phase, the resources available to support implementation, potential threats to fidelity, and checklists designed for use by local stakeholders to self-assess their fidelity of implementation. [1] There are two checklists for each of the nine core aspects of PSA implementation that are being assessed: one for the initial fidelity assessment and a second for subsequent fidelity assessments, which can (and probably should) be conducted multiple times.

Jurisdictions may wish to complete fidelity checklists for each of the nine core aspects of PSA implementation or only for some of them. They might complete the checklists in the sequence shown or in a different sequence. Finally, checklists may be completed during the implementation process or afterward. There is flexibility in how this manual can be used.

Whichever fidelity checklists are completed and whenever they are completed, they are designed to highlight specific activities in implementation that, if missed, conducted in haste, or conducted ineffectively can lead to fidelity concerns. These concerns may fall into one (or more) general categories:

• Preparation and planning: Implementation science is the study of methods and strategies that promote the effective application of interventions. In their research on change efforts, Rogers et al. [2] have concluded that most change efforts fail not due to ill-conceived ideas or faulty tools or strategies but because of poor planning and execution. The checklists contained in this manual include questions that are intended to surface whether careful planning and preparation were conducted relative to each of the nine core aspects of PSA implementation contained in this guide.
• Stakeholder consensus: Both research and experience indicate that inclusive, collaborative processes have a greater likelihood of acceptance—and sustainability—over time. The checklists contained in this manual include questions intended to identify key decisions in the PSA implementation process that are best made through consensus by a multidisciplinary stakeholder group.
• Local policy development and adherence: Implementation efforts have a greater likelihood of being well understood and implemented as intended—as well as being more transparent—when they are documented in written policy and routinely reviewed for adherence and adaptation as practices evolve or conditions change. The checklists contained in this manual include questions to determine whether key implementation practices are documented in policy, followed, and reviewed and updated annually.
• Training and quality assurance: Effective implementation of the nine core aspects of the PSA involve awareness building, skill training, and specific strategies associated with quality assurance. The checklists contained in this manual include questions to determine whether critically important steps were—and are being—taken to ensure that stakeholders are fully familiar with the PSA and each of its components, PSA assessors are adequately trained and highly skilled in PSA scoring, decision makers understand the PSA’s purpose and appropriate use, and ongoing quality assurance measures are in place to ensure the highest possible degree of accuracy and fidelity at all times.

The Timing of Fidelity Assessments

Although the timing of fidelity assessments is ultimately a local decision, the following recommendations are offered:

• Initial fidelity assessment: The initial assessment will ideally occur as soon as the planning process steps are completed and prior to PSA implementation. However, some jurisdictions may have implemented the PSA some time ago. In these cases, the initial assessment should be completed as soon as reasonably possible.
• Subsequent fidelity assessments: Subsequent assessments will ideally become routine and occur on a periodic basis. As a general rule, the second assessment will be conducted six months following the initial assessment, and subsequent assessments will take place every 6–12 months thereafter.

Identifying the Fidelity Assessors

While some jurisdictions may have the benefit of one or more people tasked with conducting fidelity assessments for pretrial or other justice system operations, such capacity may not be universally available. Take heart! No specific expertise—other than familiarity with the pretrial system and perhaps with the people involved in the implementation of the PSA and the processes they used—is required. The fidelity assessments described in this manual will ideally be conducted by a fidelity designee or a small group of stakeholders representing the various entities invested in the success of (or even dubious about) the PSA. Each set of checklists can be completed by the same person(s) or by different people.

Maximizing the Benefit of a Fidelity Assessment

The purpose of a fidelity assessment is to identify deficiencies that could threaten the success of a jurisdiction’s PSA implementation efforts. A fidelity assessment necessitates an objective approach that seeks to reveal potential or real shortcomings and to affirm areas of strength; it is not a perfunctory effort or one that should be undertaken in an effort to “prove” that all activities were flawless. The value is in finding and addressing flaws for the purpose of continuous quality improvement. In this vein, wherever possible, look for substantiation that an activity was or is being conducted rather than merely asking people about their opinions about how work was (or is) accomplished, or relying on memories. This will mean reviewing work products (e.g., documents related to the local Release Conditions Matrix; a printed copy of the pretrial assessment report), meeting minutes (reflecting that a stakeholder presentation on the PSA was provided, or that consensus was achieved on the PSA Violent Offense List), and other written documentation (e.g., a quality assurance policy on PSA scoring, or a training curriculum for stakeholders on the PSA and its factors), or observing certain practices in action (e.g., a PSA assessor interrater reliability session).

We recommend that the fidelity process be conducted transparently: that the policy team (and others) be aware of and a part of the process; that results be documented, fully shared, and discussed; and that deficiencies be viewed as opportunities for improvement.

Instructions

The process for conducting the fidelity assessments is straightforward:

1. Identify your fidelity assessment team.
2. Become familiar with the contents of this manual, especially with the fidelity checklists.
3. Meet to discuss the purpose and benefit of fidelity assessments, and the contents of this manual.
4. Decide which core area(s) to assess.
5. For each core area that will be assessed, download the materials listed in “Resources to Support Jurisdictions.”)
6. Develop a plan for conducting the fidelity assessments you’ve chosen, download the worksheets (which contain appendixes), and document your decisions in Appendixes A–I, as appropriate.
   1. Identify the people responsible for each checklist/item on the checklist.
   2. Discuss the method(s) that will be used to determine the answer to each item on the checklist.
   3. Agree on the method(s) that will be used to document the results of the assessment.
   4. Agree on a timeline for completing each checklist.
   5. Agree on a regular schedule of reporting progress to the policy team.
   6. Discuss whether broader communication (i.e., beyond the policy team) about the fidelity assessment process is needed and, if so, how and when this will occur.
7. When the assessments are completed and documented, review the findings with the policy team and with any additional stakeholders who should be involved. Discuss possible strategies to address any deficiencies that were discovered. Then, use Appendix J to develop and document a fidelity action plan.

Purpose of the Automation Phase

Arnold Ventures (AV) requires the PSA to be automated to decrease the risk of human error that would likely occur in the manual calculation of weights, points, and scores. As an added benefit, automation allows for the efficient production of pretrial assessment reports.

Resources to Support Jurisdictions with the Automation Phase

AV and APPR have developed several resources to support the selection of a PSA automation method that best meets local needs.

• 7. Guide to PSA Automation: This document provides information about PSA automation options and requirements, reviews the PSA factors and their data sources, and describes functionality—beyond the scoring of the PSA—that an automated solution may provide.
• 7A. PSA Business Requirements: If you are automating the PSA on your own, this document defines business-level requirements, specifications, and general operational characteristics for automating the PSA. It also provides instructions for the proper weighting and scoring of the PSA. This document does not offer technical details on how the PSA should be implemented from a programming or system perspective.
• 7B. PSA Data Elements Worksheet: This worksheet identifies each data element necessary to score the PSA and is used to identify the source of these data in the local jurisdiction.
• 7C. PSA Test Scenarios: This resource provides a set of 200 test scenarios that must be used prior to deployment of a locally developed automated PSA to determine its accuracy.

Potential Fidelity Challenges

There are a number of threats to the appropriate and effective implementation of the PSA pertinent to the Automation Phase. Below we highlight the most critical:

• Incomplete consideration of the various options for automating the PSA, including underestimating or disregarding the requirements for building the software locally, may result in an unsuitable choice. Several options for automating the PSA are available, each with its own requirements and special considerations. Failure to carefully consider and weigh each option to determine which one will best suit the locality may result in an inappropriate selection and in an inefficient use of human and financial resources.
• Failure to thoroughly test locally built software may result in improper functioning/inaccurate scoring of the PSA. The benefits of building a local software application in terms of integration with pre-existing applications will be lost if information technology staff are not fully aware of and compliant with the PSA business requirements and/or do not thoroughly test the software to ensure it functions properly/scores the PSA accurately.
• Errors in establishing application programming interfaces (API) for purposes of auto-populating PSA factors will lead to inaccurate scoring. While auto-populating PSA factors can create efficiency, saving valuable assessor time, API errors can have dire scoring consequences (i.e., result in incorrect factor scores and potentially harmful pretrial release decisions).
• Using a software application that inaccurately scores the PSA could lead to inaccurate PSA results. AV maintains a list of software vendors that are approved to offer the PSA as standalone software or as a component of a larger case management software package. AV enters into limited license agreements with these vendors which require them to adhere to the terms of use. The use of a non-approved vendor is prohibited.
• Lack of access to the PSA software will make it challenging for assessors to complete the PSA. The PSA terms and conditions of use require that all PSAs are automatically scored. Assessors must have ready access to the PSA software when it is needed to score PSAs and generate pretrial assessment reports; therefore, decisions regarding where the software is hosted and how and when assessors will access it are critical.

The fidelity checklists are intended to determine whether any of these threats are present and, if they are, to direct implementation teams to the appropriate areas in need of improvement. Use Appendix A and Appendix J to document decisions regarding the process for completing the fidelity checklists and to record strategies that will be implemented to address any deficiencies identified through the assessment process.

Purpose of the Assistance Phase

The purpose of the Assistance Phase of the PSA implementation process is to examine state pretrial laws, study the research on pretrial assessment and services, decide where and how the PSA results will be used in a jurisdiction’s pretrial system, and develop a Release Conditions Matrix that matches PSA results to appropriate release conditions.

Resources to Support Jurisdictions with the Assistance Phase

Several resources have been developed to assist jurisdictions as they create a Release Conditions Matrix that accommodates their local needs.

• 9. Guide to the Release Conditions Matrix: The Release Conditions Matrix is an instrument that local policymakers develop to help match pretrial release conditions with a person’s scores on the PSA. Every jurisdiction that uses the PSA should create and use a Release Conditions Matrix. The Guide to the Release Conditions Matrix describes the components of the Release Conditions Matrix, the appropriate use of the Release Conditions Matrix, and the steps to developing a locally tailored Release Conditions Matrix.
• 9A. Examples of a Release Conditions Matrix: This document contains two examples of a Release Conditions Matrix: one for a jurisdiction with moderate resources allocated for pretrial assistance and the second for a jurisdiction with fewer resources allocated for pretrial assistance. It also includes a blank (but non-editable) Release Conditions Matrix template that a jurisdiction can follow to develop its own locally tailored Release Conditions Matrix.
• 9B. Release Conditions Matrix Presentation: This PowerPoint presentation is designed for use by local stakeholders to increase understanding of the purpose, development, and use of the Release Conditions Matrix. The PowerPoint slides include lesson plan notes for the delivery of this presentation.
• 9C. PSA Results Presentation: This PowerPoint presentation presents the combined results from the 750,000 cases used to develop the PSA and 500,000 cases used to validate the PSA. Researchers completed the PSA for these cases, which came from across the country, and tracked pretrial outcomes. The presentation includes the appearance rates and arrest-free rates for each scaled score, and the violent arrest-free rates for the presence and absence of the new violent criminal arrest flag. The PowerPoint slides include lesson plan notes for the delivery of this presentation.
• 9D. Release Conditions Matrix Templates: These (editable) templates can be used to formalize a locally developed Release Conditions Matrix. Users can fill in each grid cell of the Release Conditions Matrix with their agreed-upon release level, and can complete their release activities and conditions table and their release definitions. When local validation data is available, users can replace the template’s national outcome data for the failure to appear (FTA) scaled scores and new criminal arrest (NCA) scaled scores with their own.

Potential Fidelity Challenges

There are two central threats to fidelity relative to the Release Conditions Matrix:

• Improper planning may lead to a misunderstanding of the appropriate use of the Release Conditions Matrix. The Release Conditions Matrix is a structured tool—developed by a well-rounded group of key stakeholders—that matches a person’s scores on two PSA scales (failure to appear and new criminal arrest) to presumptive levels of pretrial release. Although its principles are relatively simple, a lack of understanding around its appropriate use can lead to improper development [3] of a locally tailored Release Conditions Matrix or misuse of a properly constructed one. More specifically, the Release Conditions Matrix is not a tool for determining whether a person should be released or detained pretrial but, instead, a tool to guide the appropriate assignment of release conditions if release is granted (hence, the name). The proper construction of a Release Conditions Matrix can serve as a reminder that people have unique needs and, by matching available supportive conditions to those needs, decision makers can increase the likelihood of a person’s pretrial success. It can also reinforce key legal and evidence- based practices (i.e., use of the least restrictive conditions; matching release conditions to needs to increase pretrial success where necessary and avoiding the assignment of conditions where the likelihood of success is already high). Thus, a properly constructed Release Conditions Matrix should not include “detention,” as detention is a not a condition of release. It also should not artificially inflate PSA scores or presumptive release levels (commonly referred to as “bump-ups”) or assign financial conditions of release. Financial release conditions have failed to demonstrate improved pretrial outcomes and, instead, tend to categorically disadvantage people without financial means. The Release Conditions Matrix should also exclude the use of symbols that over-emphasize pretrial failure (i.e., by color-filling some cells red), since the vast majority of people, even those who score highest on the PSA, are more likely to succeed than to fail. [4]
• Improper care in constructing the local Release Conditions Matrix may result in the omission of appropriate and supportive release options. An additional purpose of the Release Conditions Matrix is to ensure universal recognition of the options available to support pretrial success. The omission of available options—or the failure to consider options that might be added to those already recognized as available locally—could result in the assignment of less appropriate conditions that fail to provide the supports that will further enhance the likelihood of success.

The fidelity checklists are intended to determine whether either of these threats are present and, if they are, to direct implementation teams to the appropriate areas in need of improvement. Use Appendix B and J to document decisions regarding the process for completing the fidelity checklists and to record strategies that will be implemented to address any deficiencies identified through the assessment process.

Purpose of the Assessment Phase

The purpose of the Assessment Phase of PSA implementation is to tailor PSA-related materials for local use. These materials include the PSA Violent Offense List, the PSA Scoring Manual, and the Pretrial Assessment Report. Each is critical to the effective and appropriate use of the PSA.

Resources to Support Jurisdictions with the Assessment Phase

A complete description of the development, purposes, and appropriate use of the Violent Offense List, PSA Scoring Manual, and Pretrial Assessment Report can be found in the following Implementation Guides. The Implementation Guides also include tools to assist in the initial development of these resources.

• 10. Guide to the PSA Violent Offense List: The PSA assesses three pretrial outcomes (Failure to Appear, New Criminal Arrest, New Violent Criminal Arrest). In developing and testing the PSA, researchers established a specific definition for offenses that qualify as “violent.” This definition is neither the FBI’s Uniform Crime Report definition nor a state’s statutory definition. This guide provides the PSA definition of a violent offense and walks users through the steps of developing a local PSA Violent Offense List.
• 10A. Violent Offense List Presentation: This PowerPoint presentation is designed for use by local stakeholders to increase understanding of the PSA definition of violent.
• 11. Guide to the PSA Factors and Outcomes: This guide provides detailed information about each of the nine factors used to score the PSA and the relationship of each factor to the three PSA outcomes.
• 11A. PSA Scoring Manual: PSA assessors must strictly follow the PSA scoring rules to maintain fidelity to the assessment and ensure accurate and consistent scoring. This scoring manual provides detailed information about the people who are suitable for assessment using the PSA; definitions, rules, and questions and answers to support accurate scoring of each of the nine PSA factors; and information about the three PSA outcomes. The scoring manual can be adapted to also include local scoring guidance, as necessary.
• 12. Guide to the Pretrial Assessment Report: This guide provides information about Arnold Venture’s (AV’s) requirements related to the content of pretrial assessment reports, as well as recommendations pertaining to additional information that a jurisdiction might choose to include in their reports.
• 12A. Template Pretrial Assessment Report: This document is a template that a jurisdiction can use to create their own pretrial assessment report if their choice of PSA automation system does not auto-generate a report.

Potential Fidelity Challenges

There are a number of threats to the appropriate and effective implementation of the PSA pertinent to the Assessment Phase. Below we highlight the most critical:

• Inaccurate preparation of, or adherence to, the PSA Violent Offense List can negatively impact assessment results. Decision makers may have a preconceived notion of what constitutes a violent offense. This may be based upon federal, state, or local definitions used for crime reporting; defined in statute; or reflective of personal perspective. However, because there is not a single, universal definition of “violent”—and because states’ and people’s interpretations of “violent” vary—the researchers who developed and validated the PSA created a common definition that was universally applied throughout their research. To ensure that the PSA is used as intended, jurisdictions adopting the PSA must apply the researchers’ definition of violence to their own state statutes and develop a list of state- specific violent offenses for purposes of scoring the PSA. Failure to develop this list as described—and to continually update this list (as often as your state updates its laws) to align with changes in state law— can significantly compromise the assessment results.
• Inaccurate understanding of the PSA factors and the outcomes about which the PSA provides data may lead to misinterpretation and inappropriate use of assessment results. Failure to understand that the PSA is one tool that can be used in a larger context of pretrial improvements; the outcomes the PSA is designed to assess; and/or the PSA factors, data, and scoring rules that are central to the assessment itself can result in misinterpretation of the assessment results, false expectations about what the PSA can and cannot assess, and inappropriate claims about, or application of, the information derived from the PSA.
• Improper scoring of the PSA can result in potentially harmful pretrial release decisions. Although the PSA appears relatively simple to score, important scoring rules must be applied to ensure the integrity and validity of the assessment results. Improper scoring of the PSA—or changes to the weights, score conversions, or policies that allow for “overrides” (which are expressly prohibited)—may contribute to decisions that are either harmful to the person assessed and/or to the broader community.
• Incomplete or inappropriate information/representations on the pretrial assessment report may fail to effectively—or may inappropriately—inform decision makers. The Guide to the Pretrial Assessment Report specifies that a report include certain information, such as the PSA scores and factors. The guide also encourages sharing the report with relevant parties (e.g., judiciary, prosecution, defense). Failure to comply with these requirements may jeopardize the transparency and accuracy that is essential to the assessment process.

The fidelity checklists are intended to determine whether any of these threats are present and, if they are, to direct implementation teams to the appropriate areas in need of improvement. Use Appendixes C, D, E, and J to document decisions regarding the process for completing the fidelity checklists and to record strategies that will be implemented to address any deficiencies identified through the assessment process.

Purpose of the Training Phase

The purpose of the Training Phase of PSA implementation is to ensure that stakeholders understand the development, validation, and appropriate use of the PSA in pretrial decision making, and that PSA assessors are effectively trained in accurately scoring the PSA.

Resources to Support Jurisdictions with the Training Phase

AV and APPR have developed several resources to support stakeholder education and PSA assessor training.

• 13. Guide to Stakeholder Education: This guide emphasizes the importance of stakeholder education, identifies the stakeholders who should participate in PSA education efforts, and offers recommendations around the use of APPR’s stakeholder education materials and their customization for local use.
• 13A. Stakeholder Education Lesson Plan: This complete lesson plan (trainer’s script) serves as a support for delivering the 3-hour Stakeholder Education Presentation. It includes trainers’ notes that correspond to each PowerPoint slide in the Stakeholder Education Presentation. Its purpose is to establish a context for pretrial improvements in general by reviewing guiding pretrial legal principles; discussing pretrial assessment; sharing information about the development of the PSA; reviewing the PSA factors, scoring, and results; and discussing the purpose and function of the Release Conditions Matrix.
• 13B. Stakeholder Education Presentation: This resource is a PowerPoint slide deck. It provides prompts on those slides that should be customized for jurisdiction-specific use.
• 13C. Stakeholder Education Agenda: This document is a customizable agenda for a locally delivered stakeholder education session. It provides prompts for customizing the date of the session, start and end times, the length of each of the session’s segments, and presenters’ names.
• 13D. PSA Factors & Pretrial Outcomes: This one-page document is a stakeholder education session handout. It identifies each of the PSA’s nine factors and the pretrial outcome scales with which they are associated.
• 13E. PSA Points & Scales: This two-page document is a stakeholder education session handout. It provides the points assigned to PSA factor responses and the conversion of those points to scaled scores.
• 13F. List of References: This document is a stakeholder education session handout. It provides citations to studies and other publications relevant to the implementation of the PSA and practices that can support local pretrial improvement efforts.
• 14. Guide to Assessor Training: This resource provides information that supports planning and conducting PSA assessor training. It offers guidance around identifying qualified trainers and training participants, the timing of the training relative to PSA implementation, training logistics, and an overview of the content to be covered. It also identifies other ancillary training that may be necessary or desirable.
• 14A. Assessor Training Lesson Plan: This complete lesson plan (trainer’s script) serves as a support for delivering the 5–6-hour Assessor Training Presentation. It includes trainers’ notes that correspond to each PowerPoint slide in the Assessor Training Presentation. Its purpose is to establish a context for pretrial improvements in general by reviewing guiding pretrial legal principles; discussing pretrial assessment; sharing information about the development of the PSA; reviewing the PSA factors, scoring, and results; building skills to accurately score the PSA; discussing the purpose and function of the Release Conditions Matrix and supportive release services; and identifying the appropriate content and use of a pretrial assessment report.
• 14B. Assessor Training Presentation: This resource is a PowerPoint slide deck. It provides prompts on those slides that should be customized for jurisdiction-specific use.
• 14C. Assessor Training Agenda: This document is a customizable agenda for a locally delivered assessor training. It provides prompts for customizing the date of the session, start and end times, the length of each of the session’s segments, and presenters’ names.
• 14D. PSA Factors & Pretrial Outcomes: This one-page document is an assessor training handout. It identifies each of the PSA’s nine factors and the pretrial outcome scales with which they are associated.
• 14E. PSA Points & Scales: This two-page document is an assessor training handout. It provides the points assigned to PSA factor responses and the conversion of those points to scaled scores.
• 14F. List of References: This document is an assessor training handout. It provides citations to studies and other publications relevant to the implementation of the PSA and practices that can support local pretrial improvement efforts.
• 14G PSA Practice Cases: This document is an assessor training handout. It provides information about three fictitious cases presented during the training for purposes of practicing PSA scoring.
• 14H PSA Practice Case Worksheet: This document is an assessor training handout. It is used to record paper-and-pencil scoring of the three fictitious practice cases.
• 14I PSA Assessor Worksheet: This worksheet is provided to participants of the assessor training for post-training use. In some instances, assessors will record on paper information that is pertinent to the scoring of PSA factors and then later enter this information into their automated PSA system. This worksheet is provided for the paper-and-pencil recording of this information. (In other instances, assessors will record information directly into their automated system and will not make use of this worksheet unless their automated system is offline for a period of time.)

Potential Fidelity Challenges

There are several threats to the appropriate and effective implementation of the PSA pertinent to the Training Phase, including the following:

• Insufficient education of stakeholders may lead to significant misunderstandings about, or misuse of, PSA data. Failure to understand that the PSA is one tool that can be used in a larger context of pretrial improvements; the outcomes the PSA is designed to assess; the misinterpretation or overinterpretation of the NVCA flag; and/or the PSA factors, data, and scoring rules that are central to the assessment itself can result in misinterpretation of the assessment results, false expectations about what the PSA can and cannot assess, and inappropriate claims about, or application of, the information derived from the PSA.
• Insufficient training of PSA assessors can result in inaccurate scoring of the PSA and lead to potentially harmful pretrial release decisions. Although the PSA appears relatively simple to score, important scoring rules must be applied to ensure the integrity and validity of the assessment results. Improper scoring of the PSA may contribute to decisions that are either harmful to the person assessed and/or to the broader community.

The fidelity checklists are intended to determine whether any of these threats are present and, if they are, to direct implementation teams to the appropriate areas in need of improvement. Use Appendixes F, G, and J to document decisions regarding the process for completing the fidelity checklists and to record strategies that will be implemented to address any deficiencies identified through the assessment process.

Purpose of the Fidelity Phase

The purpose of the Fidelity Phase is to establish and adhere to a set of quality assurance practices that will increase the likelihood that the PSA is scored accurately and validated locally, resulting in reliability and increased confidence among those who rely upon the assessment’s results. Beyond the PSA, steps undertaken during this phase will identify additional performance measures that are deemed important locally and establish methods for collecting data to evaluate and transparently report on these measures.

Resources to Support Jurisdictions with the Fidelity Phase

Several resources have been developed to assist jurisdictions to establish quality assurance policies and practices and to engage in important pretrial outcome and oversight activities. In particular, a set of four quality assurance tools have been developed to support accurate and reliable PSA scoring. Use of these tools is strongly encouraged, although jurisdictions may want to customize them slightly and develop their own additional tools.

• 15. Guide to Quality Assurance: Quality assurance in the context of the PSA refers to the process of verifying that the assessment is scored accurately and consistently. This guide explains the importance of PSA quality assurance and describes the components of an effective PSA quality assurance process.
• 15A. QA Review Worksheet: Quality assurance reviewers can use this tool to record the results of their independent calculation of a person’s PSA scores which will later be compared with the scores originally determined by the PSA assessor.
• 15B. QA Monthly Spreadsheet: This spreadsheet provides reviewers with a tool for compiling in a single location the results of review worksheets for all PSA assessors and for calculating accuracy rates by each PSA factor.
• 15C. QA Spreadsheet by Officer: Similar to the QA Monthly Spreadsheet, this spreadsheet provides reviewers with a tool for compiling the results of review worksheets by individual assessor and for calculating accuracy rates by each PSA factor.
• 15D. QA Independent Audit Spreadsheet: This document is used by a neutral third party to document the findings of an independent review of randomly selected cases. It also provides instructions for conducting independent reviews and selecting cases for independent review.
• 16. Guide to Outcomes and Oversight: This guide provides information about defining and measuring pretrial outcomes, the importance of tracking the performance of the PSA and other pretrial improvements, and the ongoing oversight of PSA fidelity.
• 16A. Pretrial Outcomes Display: This document illustrates a method to document and display information about people assessed using the PSA. It uses the jurisdiction’s Release Conditions Matrix to show, in each cell, the number and percentage of people who received the corresponding PSA scores and the associated success rates for appearance at court hearings and for remaining arrest-free. Row and column totals also present Failure to Appear (FTA) and New Criminal Arrest (NCA) scaled scores.

Other Resources

• PSA Assessor Worksheet: This worksheet is provided to participants of the assessor training for post-training use. In some instances, assessors will record on paper information pertinent to the scoring of PSA factors and then later enter this information into their automated PSA system. This worksheet is provided for the paper-and-pencil recording of this information. (In other instances, assessors will record information directly into their automated system and will not make use of this worksheet unless their automated system is offline for a period of time.)
• APPR Performance Measures: This document suggests 24 performance measures pertinent to pretrial performance and PSA implementation. Each measure is accompanied by a layperson’s definition and an operational definition describing how the measure is derived.

Potential Fidelity Challenges

There are a number of threats to the appropriate and effective implementation of the PSA pertinent to the Fidelity Phase. Below we highlight the most critical:

• The lack of clear quality assurance policies and practices can lead to inaccurate PSA scoring (including the interpretation of data that affects scoring), which can adversely affect pretrial decision making and undermine confidence in the PSA. Quality assurance provides for the careful examination, at the individual assessor level (as well as on the aggregate), of whether the PSA is scored and prepared accurately and completely. While the use of an automated PSA will eliminate scoring errors, the automation will not ensure that court or criminal history data are interpreted correctly. Preventing PSA scoring errors necessitates a clearly defined and adhered to set of policies and practices that provide for careful, comprehensive, ongoing review and oversight of PSA scoring. And, when quality assurance practices such as case audits, observations, and coaching suggest deficiencies in assessors’ knowledge and skills, immediate resolution is essential.
• When coaches lack sufficient capacity, resources, or support, it is difficult for them to effectively carry out their role. Coaching provides the opportunity to address deficiencies that are identified through the quality assurance process, whether due to assessors’ need for more preparation before conducting the PSA independently, carelessness, or misunderstanding. For this reason, the effectiveness of both quality assurance reviewers and coaches is essential to PSA scoring fidelity. In order for coaches to successfully administer their duties, their skills must align with their coaching duties: they must be properly trained on PSA scoring and on effective coaching strategies; they must have the time to prepare for and follow through on coaching assignments; and they need access to resources, such as refresher trainings and peer consultation, to keep their skills fresh.
• Lack of external processes to verify local quality assurance efforts can lead to the perpetuation of scoring errors. The effectiveness of quality assurance practices relies heavily on the skills and abilities of quality assurance reviewers/coaches. The use of an independent, third-party, objective expert who is proficient in PSA scoring will provide an additional level of confidence in the skills and abilities of local assessors and in the accuracy of their work.
• Insufficient responses to data errors or to changes in circumstance can lead to misinformed decisions and/or inaccurate historical records. In some cases, assessors will either miss or misinterpret information related to one of the PSA factors; in other cases, it may come to light that circumstances have changed that affect PSA scoring, such as a change in a person’s current charges. Once a PSA has been scored—regardless of whether a pretrial assessment report has been distributed—the record must be corrected following explicit protocols and, where applicable, pretrial assessment reports should be redistributed such that all parties have available to them the most current and accurate information possible.
• Failure to locally validate the PSA can result in a missed opportunity to improve the predictive accuracy of the assessment. Validation analyses answer several important questions about the PSA’s performance, including its overall predictive accuracy for all people assessed within a jurisdiction, its predictive accuracy for specific subgroups, and its ability to properly classify people by likelihood of pretrial success. To maximize the likelihood that the PSA predicts accurately at the local level, local validation should be conducted prior to implementation (where possible) and/or every 1–3 years following implementation to account for changes in local policies and populations.
• Failure to track—and transparently share—key PSA and pretrial performance and outcome measures can result in missed opportunities for improvement and erode confidence in the pretrial system. The identification of pretrial performance measures—including but not exclusive of those related to the PSA—is a critical step in the process of designing and implementing pretrial improvements, and building and sustaining confidence in the system of pretrial justice. Also critical to building and sustaining support for the pretrial system and its processes is transparency in terms of how data is presented (in a clearly understood manner vs. difficult-to-decipher calculations; using graphic design elements vs. long, dense sections of text).
• Failure to put in place an effective process to use performance and outcome measures can hamper the ability to improve policies, practices, and outcomes. Once user-friendly performance and outcome measures are transparently shared, there must be a process to discuss their implications, openly consider opportunities for improvement, and set in motion a series of activities to ensure continuous quality improvement.

The fidelity checklists are intended to determine whether any of these threats are present and, if they are, to direct implementation teams to the appropriate areas in need of improvement. Use Appendixes H, I, and J to document decisions regarding the process for completing the fidelity checklists and to record strategies that will be implemented to address any deficiencies identified through the assessment process.

PSA Training Library

Watch this training to get the most out of this guide to high-fidelity implementation of the PSA.