Frequently asked questions

Pathways to Work provides estimates of the following effects of a program: changes in average annual earnings, percent who are employed, average annual public benefits received, and percent with any education or training credential. For each effect from studies with a study quality rating of high or moderate, Pathways to Work estimates the impact of the program in standard deviation units (Hedges’ g effect sizes). Then Pathways to Work calculates a second effect size using the distributions of outcomes for adults with low earnings potential in the 2024 Current Population Survey Annual Social and Economic Supplement. The average study impacts in standard deviation units—using effect sizes from study-reported standard deviations when available—are then combined into outcome domain average effects by averaging effects within studies. Then average effects across studies are calculated for each outcome domain, giving more weight to studies with larger sample sizes. These averages are then also converted into 2024 dollars and percentages.

An effectiveness rating is the assessment by the Pathways to Work Evidence Clearinghouse of the extent to which a given program improves a specific outcome based on the existing evidence from impact studies. The effectiveness rating suggests whether a program is likely to produce favorable results if faithfully replicated with a similar population.

For each outcome, Pathways to Work presents the effect from studies with a high or moderate rating in standard deviations (also referred to as an effect size). This effect is a standardized measure that allows us to make direct and meaningful comparisons across different outcomes, settings, and programs. Pathways to Work converts effects into 2024 dollars or percentage points to help users more easily understand the magnitude of these effects. Both the effectiveness rating and the presented effects can be compared across studies in the Pathways to Work Evidence Clearinghouse.

An effectiveness rating assesses whether a program is likely to produce favorable results if faithfully replicated with a similar population. Outcome domains with well-supported ratings are those that the evidence suggests are most likely to improve if a program were replicated with a similar population. Outcome domains with supported ratings have some evidence that the program improves them, but the evidence is less conclusive. Outcome domains that receive a rating of not supported have strong or consistent evidence that the program is unlikely to produce favorable effects.

However, because implementation challenges and successes often vary and no two implementations of a program are identical, there is no guarantee that implementing a program with well-supported or supported ratings will necessarily yield similar results.

The Pathways to Work Evidence Clearinghouse’s ability to report on the effects of a program depend on the existing evidence. If the program did not have any study findings rated moderate or high for a given outcome, that outcome receives an effectiveness rating of cannot assess support. There are several reasons why study findings may not receive a moderate or high rating. Some studies may not report on all Pathways to Work outcomes (earnings, employment, public benefit receipt, and education and training). A program may have studies that examine effects for some outcomes but not others. In other cases, the quality of evidence may vary across outcomes. In addition, we may not have all the data needed to calculate an effect size from the original study. In these cases, we cannot report the effects of a program for those outcomes.

A program can have a negative overall effect on earnings and a supported rating in earnings because of how we determine program effectiveness in a given outcome domain and how we calculate the size of a program’s effects on that outcome domain. In order to receive a supported effectiveness rating, a program must have at least one statistically significant, favorable finding and no statistically significant unfavorable findings in the given outcome domain. The program’s effects, on the other hand, are an average of all findings for a given outcome domain, including those that are not statistically significant.

Take, for example, a study that finds three effects on earnings, one of which is statistically significant and favorable, and two of which are not statistically significant and unfavorable. Because the study identified a statistically significant, favorable effect on earnings and no statistically significant unfavorable findings, the program receives an effectiveness rating of supported on earnings. In calculating its overall effect on earnings, however, we average all the findings in this domain, including the two unfavorable (but statistically insignificant) findings and the one statistically significant, favorable finding. The average of these three findings might result in an overall negative effect on earnings.

Pathways to Work reports on the effects for study findings that are rated high or moderate quality in our four outcomes (earnings, employment, public benefit receipt, and education and training). When the quality of a study is high, that means we have strong evidence that the study findings are solely attributable to the program examined. This rating is reserved for study findings from high quality randomized controlled trials with low attrition of sample members. Pathways to Work also reports effects for moderate quality studies, where there is some evidence that the study findings are attributable, at least in part, to the program examined.

There are several reasons that some findings in a study may not be reported:

• Studies may have outcomes that are rated low quality, where there is little evidence that the study findings are attributable to the program examined. Other important factors could have influenced the study findings, and the study did not account for them. Pathways to Work does not report outcomes that are rated low quality.
• If the original study provides findings for multiple measures in a given outcome domain, Pathways to Work prioritizes findings to review and to report following the prioritization process summarized in Exhibit 3.2 of the Protocol for the Pathways to Work Evidence Clearinghouse. For example, if study authors used both surveys and administrative records to assess earnings, Pathways to Work reviewers select two sets of earnings findings for review: one measured using survey data and one measured using administrative data.
• In some cases, Pathways to Work may not have sufficient information to report on the magnitude of study findings. When we can obtain the necessary information from study authors, we calculate an effect size using study-specific data.