Introduction
Service providers, policymakers, and researchers need to know how likely specific programs are to improve employment and related outcomes if implemented in a particular setting with clients. In practice, most employment programs offer a combination of services that are designed to improve labor market outcomes (e.g., employment, earnings, education and training, and public benefit receipt). The sheer volume of research on these programs, combined with the diversity of program characteristics, study participants, and findings, can make it challenging to identify what works.
The Pathways to Work Evidence Clearinghouse is dedicated to gathering information from research and making evidence more useful to decision makers. As part of these efforts, Pathways to Work has undertaken a series of research syntheses to explore what we can learn by looking across all the data Pathways to Work has collected. This report, one in that series, uses Bayesian meta-analysis—an analytic approach that synthesizes relationships across multiple studies—to provide new evidence on the likelihood that specific programs will improve labor market outcomes and which combinations of services are most likely to be effective for different groups of people.
Research questions
- What are the most effective programs in Pathways to Work?
- Which services and combinations of services have the highest probabilities of improving outcomes?
- Which services are most effective based on characteristics of study participants?
Purpose
Policymakers, practitioners, and researchers can use this report to identify the programs, and the combination of services provided by those programs, that are most likely to improve labor market outcomes for different groups of people.
Methods
The Pathways to Work team systematically identified, categorized, and assessed studies of programs designed to improve the labor market outcomes of individuals with low incomes. The team recorded information about the study methods, the characteristics and impacts of the programs they examined, and the populations served. We identified the primary service offered by each program and the populations served by each program, and standardized impacts across studies.
For this report, we synthesized the evidence using Bayesian meta-regression to distill all available information into an assessment of the probability that a program, service, or combination of services would improve outcomes. A meta-regression estimates the relationships between impact estimates and individual study and program characteristics while holding all other characteristics constant. Bayesian meta-regression is an advanced form of meta-regression that enables us to estimate relationships between impacts and specific combinations of program and study characteristics that might appear only rarely in the data. This approach allows us to model the relationship between these characteristics and estimated impacts in a way that captures some of the complexity inherent in these relationships. We used these findings to draw policy-relevant assessments of effectiveness by calculating the probability that programs and program characteristics could be genuinely effective, given all the available evidence.
Key Findings and Highlights
- Twelve of the 127 programs examined for this analysis have more than a 90 percent probability of improving labor market outcomes for participants.
- Interventions focused on work and work-based learning have the highest probability of improving outcomes relative to programs focused on delivering other services.
- Interventions focused on work and work-based learning have a 74 percent probability of improving labor market outcomes overall (including earnings, employment, education and training, and public benefit receipt), and a 94 percent probability of improving employment outcomes specifically.
- Interventions focused on work and work-based learning were most effective when combined with employment services; these programs have an 81 percent probability of improving outcomes when offered in combination with work and work-based learning.
- No single primary service is particularly likely to improve outcomes by a substantial amount (by at least $1,000 in annual earnings).
- Though programs focused on work and work-based learning have the highest probability of improving outcomes by any amount, they are among the least likely to improve outcomes by a substantial amount relative to programs focused on delivering other services (a 21 percent probability of improving outcomes by an amount equivalent to $1,000 or more).
- Interventions focused on subsidized and transitional jobs are the most likely to improve outcomes by an amount equivalent to $1,000 or more (a 32 percent probability).
- The likelihood that a program offering a particular primary service improves outcomes differs depending on the characteristics of the population being studied. However, programs focused on work and work-based learning, employment retention services, and case management and other supports typically have the highest probabilities of improving outcomes in programs serving primarily people from racial or ethnic minority groups, women, and people with disabilities.
Citation
Shiferaw, Leah and Dan Thal. (2022). Digging Deeper into What Works: What Services Improve Labor Market Outcomes, and for Whom? OPRE Report # 2022-161, Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
