To find a list of well-supported or supported programs for a given outcome, go to Find Programs that Work, and then click on the “Effectiveness rating” column to show programs that are rated 
well-supported or 
supported at the top of the search results.
The Pathways to Work Quality Review Team (QRT) handles any challenges received about a review’s findings, the inclusion of a study within the Pathways to Work Evidence Clearinghouse, or other individual judgements the Pathways to Work team makes. The QRT addresses issues with reviews so long as they are (1) submitted in writing to PathwaysClearinghouse@abtglobal.com, (2) related to a specific study or well-defined set of studies, and (3) coherently explained (and the inquirer is available to answer any clarifying questions).
When a request is submitted to the QRT, a team member first verifies the request meets the criteria listed above. After this confirmation, the team member examines the study and any related materials, discusses the review with the original study reviewers, and presents a summary of the review and any potential flaws to the QRT. The QRT then determines whether the initial review should be revised, notifies OPRE and the inquirer of its findings and, if necessary, edits any products to reflect the updated review.
An effectiveness rating is the assessment by the Pathways to Work Evidence Clearinghouse of the extent to which a given program improves a specific outcome based on the existing evidence from impact studies. The effectiveness rating suggests whether a program is likely to produce favorable results if faithfully replicated with a similar population.
For each outcome, Pathways to Work presents the effect from studies with a high or moderate rating in standard deviations (also referred to as an effect size). This effect is a standardized measure that allows us to make direct and meaningful comparisons across different outcomes, settings, and programs. Pathways to Work converts effects into 2024 dollars or percentage points to help users more easily understand the magnitude of these effects. Both the effectiveness rating and the presented effects can be compared across studies in the Pathways to Work Evidence Clearinghouse.
An effectiveness rating assesses whether a program is likely to produce favorable results if faithfully replicated with a similar population. Outcome domains with well-supported ratings are those that the evidence suggests are most likely to improve if a program were replicated with a similar population. Outcome domains with supported ratings have some evidence that the program improves them, but the evidence is less conclusive. Outcome domains that receive a rating of not supported have strong or consistent evidence that the program is unlikely to produce favorable effects.
However, because implementation challenges and successes often vary and no two implementations of a program are identical, there is no guarantee that implementing a program with well-supported or supported ratings will necessarily yield similar results.
The Pathways to Work Evidence Clearinghouse’s ability to report on the effects of a program depend on the existing evidence. If the program did not have any study findings rated moderate or high for a given outcome, that outcome receives an effectiveness rating of cannot assess support. There are several reasons why study findings may not receive a moderate or high rating. Some studies may not report on all Pathways to Work outcomes (earnings, employment, public benefit receipt, and education and training). A program may have studies that examine effects for some outcomes but not others. In other cases, the quality of evidence may vary across outcomes. In addition, we may not have all the data needed to calculate an effect size from the original study. In these cases, we cannot report the effects of a program for those outcomes.
A program can have a negative overall effect on earnings and a supported rating in earnings because of how we determine program effectiveness in a given outcome domain and how we calculate the size of a program’s effects on that outcome domain. In order to receive a supported effectiveness rating, a program must have at least one statistically significant, favorable finding and no statistically significant unfavorable findings in the given outcome domain. The program’s effects, on the other hand, are an average of all findings for a given outcome domain, including those that are not statistically significant.
Take, for example, a study that finds three effects on earnings, one of which is statistically significant and favorable, and two of which are not statistically significant and unfavorable. Because the study identified a statistically significant, favorable effect on earnings and no statistically significant unfavorable findings, the program receives an effectiveness rating of supported on earnings. In calculating its overall effect on earnings, however, we average all the findings in this domain, including the two unfavorable (but statistically insignificant) findings and the one statistically significant, favorable finding. The average of these three findings might result in an overall negative effect on earnings.
Studies with low quality ratings demonstrate little evidence that findings are attributable, in part or in full, to the program examined. A low quality rating suggests that there is a high risk of bias. Pathways to Work focuses on providing detailed information for studies rated high or moderate quality because these studies have a lower risk of bias and the programs are more likely to have contributed to the findings. In other words, the Pathways to Work Evidence Clearinghouse focuses on studies that provide the most relevant, useful information for program providers and policymakers.
