Frequently Asked Questions

For the Pathways Clearinghouse, the effects shown are the estimated changes in the percent of low-income adults who are employed, average annual earnings, average annual public benefits received, and percent of low-income adults with any education and training credential. The Pathways Clearinghouse takes each effect and estimates comparable effects in 2018 dollars and percentages by calculating the average impact of the intervention in standard deviation units (Hedges’ g effect sizes) across outcomes and studies and comparing that average to the distributions of outcomes for adults with low earnings potential in the 2019 Current Population Survey Annual Social and Economic Supplement. The impacts in standard deviation units are also combined into domain average effects by taking the average across studies and outcomes, giving more weight to studies with larger sample sizes. These averages are then also converted into 2018 dollars and percentages.

An effectiveness rating is the assessment of the Pathways Clearinghouse, based on the existing evidence from impact studies, of the extent to which a given intervention improves a specific type of outcome. The effectiveness rating is a holistic assessment of whether an intervention is likely to produce favorable results if faithfully replicated with a similar population. An effect size is a standardized measure that allows us to make direct and meaningful comparisons across different outcomes, settings, and interventions. Both the effectiveness rating and the effect size can be compared across studies in the Pathways Clearinghouse. The effect size is the impact found in the available research, and the effectiveness rating is the likelihood that an intervention would produce a similar, favorable effect if implemented again with a similar population and context.

An effectiveness rating assesses whether an intervention is likely to produce favorable results if faithfully replicated with a similar population. Outcome domains with well-supported ratings are those that the evidence suggests are most likely to improve if an intervention were replicated with a similar population. Outcome domains with supported ratings have some evidence that the intervention improves them, but the evidence is less conclusive. Outcome domains that receive a rating of not supported have strong or consistent evidence that the intervention is unlikely to produce favorable effects.

However, because implementation challenges and successes often vary and no two implementations of an intervention are identical, the well-supported and supported ratings do not guarantee success.

The Pathways Clearinghouse’s ability to report on the effects of an intervention are tied to the existing evidence. Not all interventions have been studied along all the domains on which we report [earnings, employment, public benefit receipt, and education and training]. An intervention may have studies that examined effects in some outcomes but not in others. In other cases, the quality of evidence may vary across outcome domains. If we did not find any studies that rated moderate or high quality that studied the intervention’s effect on outcomes in a given outcome domain, that outcome receive an effectiveness rating of no evidence to assess support. In addition, we may not have all of the data to extract an effect size from the original study (as discussed in the previous FAQ). In those cases, we cannot report on the effects on an intervention for those domains.

This has to do with the difference between how we calculate whether or not an intervention has evidence of being effective on a given outcome domain and how we calculate the size of an intervention’s effects on that outcome domain. In order to receive a supported rating, an intervention must have at least one statistically significant, favorable finding and no statistically significant unfavorable findings in the given outcome domain. Effect sizes, on the other hand, are an average of all findings for a given outcome domain, including those that are not statistically significant.

Take, for example, a study that finds three effects on earnings, one of which is statistically significant, and favorable, and two of which are statistically insignificant, but unfavorable. Because the study identified a statistically significant, favorable effect on earnings and no statistically significant unfavorable findings, the intervention receives an effectiveness rating of supported on earnings. In calculating its overall effect on earnings, however, we average all the findings in this domain, the two unfavorable, but statistically insignificant findings, along with the statistically significant, favorable effect. The average of these three findings might result in an overall negative effect on earnings.

The Pathways Clearinghouse estimates effect sizes for each finding with a high or moderate rating when the study or study authors provide sufficient information to calculate an effect size. The Pathways Clearinghouse team contacts authors to obtain the information necessary to calculate the effect size, but in some cases, sufficient information is not available.

The Pathways Clearinghouse reports on the effects for study findings that are rated high or moderate quality in our four domains [earnings, employment, public benefit receipt, and education and training]. When the quality of a study is high, that means we can be fairly confident in the study findings because the study finding is solely attributable to the intervention examined. This rating is reserved for study findings from high quality RCTs with low attrition of sample members. The Pathways Clearinghouse also reports effects for moderate quality studies, where we can be somewhat confident in the study findings.

There are several reasons that some findings in a study may not be rated. Specifically:

• Studies may have outcomes that are rated low quality, where we cannot have much confidence in the study findings. Other important factors could have influenced the study findings, and the study did not account for them. The Pathways Clearinghouse does not report outcomes that are rated low quality.
• If the original study provides findings for multiple outcome measures in a given domain, the Pathways Clearinghouse prioritizes findings to review and report based on the outcome measure, following the prioritization process summarized in Exhibit III.2 of the Protocol for the Pathways to Work Clearinghouse: Methods and Standards. For example, if study authors used both surveys and administrative records to assess earnings, Pathways Clearinghouse reviewers select two sets of earnings findings for review: one measured using survey data and one measured using administrative data.

In addition, the Pathways Clearinghouse may not have sufficient information to report on the magnitude of study findings. We calculate an effect size using study-specific data, if we can obtain from the study authors the information needed to do so.