Frequently asked questions

The Pathways to Work Evidence Clearinghouse maintains a list of services that we use to characterize programs and conduct analyses. This list is located on the website within Find Programs that Work, under the filter “Select services the program should provide.” The list nests some of the services underneath other services, to show services that are part of a broader service category. Programs can offer any combination of the services on the list, and any of the services listed may be primary services. As Pathways to Work continues to expand, other services may be added to the list.

Pathways to Work synthesis reports currently combine groups of closely related services from the list to ensure a meaningful number of studies will inform our meta-analyses. Those groups are used as part of analyses to determine what types of services are associated with better outcomes. For example, What Works to Improve Employment and Earning for People with Low Incomes combines education and training services and compares the effectiveness of these services to other types of services, such as work and work-based learning and incentives and sanctions.

The Pathways to Work Evidence Clearinghouse developed a list of services that programs provide. Some services are nested within other services. For example, occupational or sectoral training is nested within education and training, because it represents a type of training. The higher-level category of education and training also includes other types of education and training such as basic skills/bridge programs, integrated education and training, and pre-apprenticeship training. Any service, regardless of whether it is nested within another service, can be the primary service.

Employment and training programs commonly provide light-touch supports—such as money or vouchers for childcare or transportation—to address barriers to participation. Pathways to Work categorizes these kinds of services as “supportive services,” because they are explicitly designed to support employment and training. Pathways to Work also reviews programs that may improve labor market outcomes through strategies other than employment and training services, including by addressing barriers to work through provision of intensive housing, childcare, or legal assistance services. Pathways to Work categorizes these types of services as “housing supports and subsidies,” “childcare and/or early education,” or “legal assistance” services to indicate that the supports are more intensive and occur outside the context of an employment and training program.

The Pathways to Work Evidence Clearinghouse is an investment of the Administration for Children and Families within the U.S. Department of Health and Human Services. It was established under a Congressional mandate in the Consolidated Appropriations Act of 2017 (PL 115-31). Launched in 2020, the Pathways to Work Evidence Clearinghouse identifies programs that aim to improve employment outcomes and support self-sufficiency for populations with low incomes, especially recipients of public programs such as Temporary Assistance for Needy Families (TANF). It is a comprehensive resource designed to help direct employment service providers, TANF administrators, policymakers, and researchers easily access and understand the effectiveness of programs that aim to improve employment outcomes for job seekers with low incomes. Information about a program’s effectiveness is presented alongside information for better understanding the context in which it has been shown to work and how it was implemented.

The Chief Evaluation Office within the U.S. Department of Labor established the Clearinghouse for Labor Evaluation and Research (CLEAR) in 2012 to increase the accessibility of research on a wide range of labor and employment topics. CLEAR is a central source for labor-related research and evaluation evidence on a variety of programs spanning employment, training, health and safety, worker benefits, employer compliance, and other topics. CLEAR aims to help practitioners, policymakers, researchers, and the public find and use research to inform their decisions about labor policies and programs, future research, and improve outcomes for a variety of populations. Find more information about CLEAR by visiting: Clearinghouse for Labor Evaluation and Research: https://clear.dol.gov/.

The Pathways to Work Evidence Clearinghouse and CLEAR are coordinated federal efforts that have complimentary areas of focus and use similar standards to assess the quality of causal evidence in studies. Both Clearinghouses primarily focus on reviewing and rating causal impact studies. The Clearinghouses have collaborated to share studies and to combine dissemination efforts. There are also important differences between the Clearinghouses, including distinct goals, approaches to aggregating data across studies, and the types of study designs reviewed.

Goals and scope

• CLEAR is designed to be responsive to the entirety of the work and programs overseen by the U.S. Department of Labor, including, for example, compliance with occupational safety guidelines and child labor.
• Pathways to Work is more narrowly focused on programs that aim to improve self-sufficiency for individuals with low incomes.

Data aggregation

• In addition to reviewing individual studies, Pathways to Work aggregates information across studies to determine a program’s effectiveness. It rates a program’s effectiveness within each of four outcome domains: earnings, employment, public benefit receipt, and education and training.
• CLEAR primarily focuses on assessing individual studies and rates programs for effectiveness in only one of its topic areas, Reemployment Services and Eligibility Assessments (RESEA), to help states meet program requirements.

Types of studies reviewed

• In addition to reviews of causal studies, CLEAR reviews descriptive and implementation studies in some topic area evidence reviews.
• Pathways to Work does not systematically review noncausal studies but does provide implementation details for studies with a moderate or high quality study rating.
• Pathways to Work also has established cost standards and rates the quality of cost estimates from cost analyses, cost-benefit analyses, and cost-effectiveness analyses.

We send occasional mailings to users who subscribe to our email list, alerting them to new website content and open calls for studies. Our “What’s New” section of the website is also updated regularly when new content is posted to the website.

To find a list of well-supported or supported programs for a given outcome, go to Find Programs that Work, and then click on the “Effectiveness rating” column to show programs that are rated  well-supported or  supported at the top of the search results. 

The Pathways to Work Evidence Clearinghouse welcomes submissions of research studies that have evaluated a particular employment and training program. To have a study considered for review, submit the program name and its associated citations or research studies to PathwaysClearinghouse@abtglobal.com. (See the question below: How do I submit my research for consideration?) You can also submit information on studies in progress that are not yet completed. The Pathways to Work team will log all recommendations. Pathways to Work reviews programs periodically, as resources allow. As a result, programs submitted to Pathways to Work will not be reviewed immediately upon request. In addition, submitting a program for consideration does not guarantee that we will review studies of the program.

The Pathways to Work team uses prespecified keywords to systematically search databases of journal publications, evaluation reports, and unpublished literature (such as working papers). We also incorporate studies cited in literature reviews and those submitted in response to a public call for relevant studies. We screen studies for their eligibility to be included in the review against a set of predefined criteria. Then, we prioritize the eligible studies based on publication date (examining the most recent first and turning to older research as resources allow). For more information, please see the Protocol for the Pathways to Work Evidence Clearinghouse.

Occasionally, the Pathways to Work Evidence Clearinghouse issues a call for studies and invites the public to submit research. When open, we will post a link to more information on the Pathways to Work home page and will circulate the call to users who subscribe to our email list. Interested stakeholders can submit research at any time to PathwaysClearinghouse@abtglobal.com. The Pathways to Work team will log the submission for consideration. Submitting research does not guarantee that the study will be reviewed. We decide which research to review using the systematic process outlined in the Protocol for the Pathways to Work Evidence Clearinghouse.

The Pathways to Work Quality Review Team (QRT) handles any challenges received about a review’s findings, the inclusion of a study within the Pathways to Work Evidence Clearinghouse, or other individual judgements the Pathways to Work team makes. The QRT addresses issues with reviews so long as they are (1) submitted in writing to PathwaysClearinghouse@abtglobal.com, (2) related to a specific study or well-defined set of studies, and (3) coherently explained (and the inquirer is available to answer any clarifying questions).

When a request is submitted to the QRT, a team member first verifies the request meets the criteria listed above. After this confirmation, the team member examines the study and any related materials, discusses the review with the original study reviewers, and presents a summary of the review and any potential flaws to the QRT. The QRT then determines whether the initial review should be revised, notifies OPRE and the inquirer of its findings and, if necessary, edits any products to reflect the updated review.

Studies underway have a rigorous impact evaluation in progress, but no available findings at the time that Pathways to Work identifies them. Nominations for studies underway can come from federal staff, evaluators, and the public. If the evaluations meet the eligibility criteria established in the Protocol for the Pathways to Work Evidence Clearinghouse, they are added to the Studies Underway page. These evaluations need to be randomized controlled trials or comparison- group quasi-experimental designs of an employment or training program in the U.S. or Canada that focuses on populations with low incomes.

Please note that Pathways to Work is not a registry for impact evaluations and submitting evaluations does not count as registering research. If you have a study underway to share with Pathways to Work, please email the name of the program, a brief description, and links to any additional publicly available materials (such as websites or reports) to PathwaysClearinghouse@abtglobal.com.

Pathways to Work provides estimates of the following effects of a program: changes in average annual earnings, percent who are employed, average annual public benefits received, and percent with any education or training credential. For each effect from studies with a study quality rating of high or moderate, Pathways to Work estimates the impact of the program in standard deviation units (Hedges’ g effect sizes). Then Pathways to Work calculates a second effect size using the distributions of outcomes for adults with low earnings potential in the 2024 Current Population Survey Annual Social and Economic Supplement. The average study impacts in standard deviation units—using effect sizes from study-reported standard deviations when available—are then combined into outcome domain average effects by averaging effects within studies. Then average effects across studies are calculated for each outcome domain, giving more weight to studies with larger sample sizes. These averages are then also converted into 2024 dollars and percentages.

An effectiveness rating is the assessment by the Pathways to Work Evidence Clearinghouse of the extent to which a given program improves a specific outcome based on the existing evidence from impact studies. The effectiveness rating suggests whether a program is likely to produce favorable results if faithfully replicated with a similar population.

For each outcome, Pathways to Work presents the effect from studies with a high or moderate rating in standard deviations (also referred to as an effect size). This effect is a standardized measure that allows us to make direct and meaningful comparisons across different outcomes, settings, and programs. Pathways to Work converts effects into 2024 dollars or percentage points to help users more easily understand the magnitude of these effects. Both the effectiveness rating and the presented effects can be compared across studies in the Pathways to Work Evidence Clearinghouse.

An effectiveness rating assesses whether a program is likely to produce favorable results if faithfully replicated with a similar population. Outcome domains with well-supported ratings are those that the evidence suggests are most likely to improve if a program were replicated with a similar population. Outcome domains with supported ratings have some evidence that the program improves them, but the evidence is less conclusive. Outcome domains that receive a rating of not supported have strong or consistent evidence that the program is unlikely to produce favorable effects.

However, because implementation challenges and successes often vary and no two implementations of a program are identical, there is no guarantee that implementing a program with well-supported or supported ratings will necessarily yield similar results.

The Pathways to Work Evidence Clearinghouse’s ability to report on the effects of a program depend on the existing evidence. If the program did not have any study findings rated moderate or high for a given outcome, that outcome receives an effectiveness rating of cannot assess support. There are several reasons why study findings may not receive a moderate or high rating. Some studies may not report on all Pathways to Work outcomes (earnings, employment, public benefit receipt, and education and training). A program may have studies that examine effects for some outcomes but not others. In other cases, the quality of evidence may vary across outcomes. In addition, we may not have all the data needed to calculate an effect size from the original study. In these cases, we cannot report the effects of a program for those outcomes.

Very long-term outcomes are those measured 5 years or more after participants in the study’s program group are first offered services. To view very long-term outcomes for a program, click on the program’s page and navigate to the Effectiveness rating and effect by outcome domain table.

A program can have a negative overall effect on earnings and a supported rating in earnings because of how we determine program effectiveness in a given outcome domain and how we calculate the size of a program’s effects on that outcome domain. In order to receive a supported effectiveness rating, a program must have at least one statistically significant, favorable finding and no statistically significant unfavorable findings in the given outcome domain. The program’s effects, on the other hand, are an average of all findings for a given outcome domain, including those that are not statistically significant.

Take, for example, a study that finds three effects on earnings, one of which is statistically significant and favorable, and two of which are not statistically significant and unfavorable. Because the study identified a statistically significant, favorable effect on earnings and no statistically significant unfavorable findings, the program receives an effectiveness rating of supported on earnings. In calculating its overall effect on earnings, however, we average all the findings in this domain, including the two unfavorable (but statistically insignificant) findings and the one statistically significant, favorable finding. The average of these three findings might result in an overall negative effect on earnings.

The Pathways to Work Evidence Clearinghouse reports implementation details for programs with at least one outcome domain that has a well-supported or supported rating across the 10 outcome domains (increasing earnings, increasing employment, or decreasing benefit receipt in the short, long, or very long term, or increasing education and training). We focus on programs with supported outcomes because program providers and policymakers might be interested in learning more about how to implement these programs.

Pathways to Work identifies and reviews relevant studies, including impact, descriptive, and implementation studies, to collect details on how a program was implemented. We summarize information such as the amount of services received, the number of staff delivering services, and costs and benefits.

The Pathways to Work Evidence Clearinghouse provides cost information for programs that have at least one supported or well-supported effectiveness rating. Cost information is located on the details page for a program, the page which supplies more context about programs with favorable outcomes. All cost information reported on the website is based on data reported by study authors. We first convert these figures to their present value at study baseline (that is, at the time program services began, or at the time of random assignment for programs studied using randomized controlled trial designs). Specifically, we discount costs that occurred over multiple years to their value at baseline because the value of a dollar is greater in the present than in the future. We assume a real rate of return of 3 percent annually. We then report those costs in 2024 dollars. Where there are multiple studies of a program rated high or moderate quality, we compute the simple average of costs reported across those studies, and the cost figure we report reflects the average set of program services provided.

Because the cost information is based on data reported by study authors, cost information is not directly comparable across programs. The types of costs reported across programs vary, as does the amount of time that programs lasted. The Pathways to Work Evidence Clearinghouse has developed new standards for cost analyses (i.e., cost estimates), cost-effectiveness analyses, and cost-benefit analyses that are described in the Protocol for the Pathways to Work Evidence Clearinghouse. Ratings from cost study reviews are forthcoming, which will provide Pathways to Work’s assessment of the quality of cost estimates.

Studies with low quality ratings demonstrate little evidence that findings are attributable, in part or in full, to the program examined. A low quality rating suggests that there is a high risk of bias. Pathways to Work focuses on providing detailed information for studies rated high or moderate quality because these studies have a lower risk of bias and the programs are more likely to have contributed to the findings. In other words, the Pathways to Work Evidence Clearinghouse focuses on studies that provide the most relevant, useful information for program providers and policymakers.

Pathways to Work reports on the effects for study findings that are rated high or moderate quality in our four outcomes (earnings, employment, public benefit receipt, and education and training). When the quality of a study is high, that means we have strong evidence that the study findings are solely attributable to the program examined. This rating is reserved for study findings from high quality randomized controlled trials with low attrition of sample members. Pathways to Work also reports effects for moderate quality studies, where there is some evidence that the study findings are attributable, at least in part, to the program examined.

There are several reasons that some findings in a study may not be reported:

• Studies may have outcomes that are rated low quality, where there is little evidence that the study findings are attributable to the program examined. Other important factors could have influenced the study findings, and the study did not account for them. Pathways to Work does not report outcomes that are rated low quality.
• If the original study provides findings for multiple measures in a given outcome domain, Pathways to Work prioritizes findings to review and to report following the prioritization process summarized in Exhibit 3.2 of the Protocol for the Pathways to Work Evidence Clearinghouse. For example, if study authors used both surveys and administrative records to assess earnings, Pathways to Work reviewers select two sets of earnings findings for review: one measured using survey data and one measured using administrative data.
• In some cases, Pathways to Work may not have sufficient information to report on the magnitude of study findings. When we can obtain the necessary information from study authors, we calculate an effect size using study-specific data.