Methodology

The Urban Partnerships Initiative Online Toolkit was developed using a combination of primary and secondary research techniques. A project workgroup consisting of five TANF administrators from around the country provided feedback on the draft toolkit to create a final product that was responsive to the needs of the field. The toolkit was developed in three stages: an environmental scan to identify promising programs, selection of programs to highlight, and collection and analysis of data about the selected programs.

Environmental Scan: To gain a comprehensive profile of the field spanning the continuum of TANF services, it was necessary to adopt a broad approach to identifying best practices. An environmental scan included a thorough literature review and nominations from the field, represented by individuals from TANF Regional Offices, Urban Partnerships Initiative team leaders, and site visitors representing the online community of practice. The scan was not limited to programs that specifically targeted TANF clients, but also examined those that had valuable characteristics that could improve or enhance current welfare programs or provide a foundation for further program innovation.

The literature review was conducted to identify practices that experts and institutions cited as effective, or potentially effective, in serving clients. The process incorporated a review of published books and journal articles, as well as a systematic exploration of Web sites dedicated to the most recent information pertaining to welfare reform. Additionally, Web sites and conferences that highlighted best practices were scrutinized for contributions.

A nomination process pinpointed programs that appeared promising but had not been cited in the existing literature. Contributions were solicited from numerous internal and external partners: the Child, Family, and Education Services group of ICF International; project officers and regional officers of the Office of Family Assistance; the Welfare Peer Technical Assistance Web site; the Urban Partnerships Initiative community of practice; and Urban Partnerships Initiative team leaders. This approach incorporated the expertise of our affiliates in providing services to low-income families and communities, and identifying, assessing, and sharing best practices in welfare reform.

A total of 557 programs and practices were identified during the environmental scan. The programs were organized along the TANF continuum of services, with some programs categorized into more than one TANF service area.

Number of Programs Identified in TANF Service Area

Program Selection: It was not feasible to systematically review more than 500 programs within the scope of this project. Instead, a tiered approach was developed to prioritize the review and selection of promising practices. Urban Partnerships Initiative staff identified a top tier of 20-25 programs in each TANF service area based on referral source, availability of additional information, available evidence of program success or promise, and whether the program was still operational.

A multi-step program selection process was used to identify the programs that would be featured in each TANF service section of the toolkit. The process was developed to meet the unique purposes and constraints of this project. To ensure the process was consistent with practice in the field, we began with a review of procedures that other organizations with best practices registries have used to identify promising practices (see Appendix 1). Using procedures from other projects as a guide, our purpose was to identify promising practices that would advance TANF services in a post-Deficit Reduction Act policy environment. The process for identifying promising practices had three steps:

1. Assessment of the available evidence
2. Accordance with field-tested review criteria
3. Scoring and Urban Partnerships Initiative/Office of Family Assistance staff review

Assessment of Evidence: To conduct an assessment of available evidence, the project team collected examples of criteria that other institutions had used to classify studies and outcomes. These included examples from the What Works Clearinghouse, which pulls together a highly rigorous set of evaluations, and from the Resource Center at the Corporation for National and Community Service, which features program examples that have solved a problem but did not necessarily involve a randomized controlled trial.

The project team compared 10 different approaches that various groups were using to highlight best practices, and used these as a basis for setting standards for the toolkit. While some best practices registries, such as the What Works Clearinghouse, required that a program had been rigorously evaluated through empirical research, the team quickly realized that many of the unique TANF programs identified in our environmental scan had not been formally evaluated. Additionally, because such rigorous evaluations can take a considerable amount of time to design and implement, the team found that the TANF programs that had undergone formal evaluations and demonstrated positive outcomes often were operating in an outdated policy context such that they could not address the new requirements or restrictions specified in the Deficit Reduction Act of 2005. To identify promising practices that could advance the TANF field, the team developed alternative criteria for measuring program success.

To include practices that were demonstrating a level of successful TANF service delivery while operating under updated Federal regulations, the team established program selection procedures that aligned with the more inclusive, less restrictive guidelines of the criteria examples that had been analyzed.1 Urban Partnerships Initiative staff developed an alternative rating system to describe the levels of evidence provided by the featured programs:

• Promising Practice - has outcome evaluations showing positive effects on families' self-sufficiency

• Common Practice - widespread practice in the field (at least five cities) with practice evidence (e.g., reduced caseload)

• Innovative Practice - a novel approach that showed promise for success based on theory or practice experience

Accordance with Review Criteria: After determining the baseline groupings for the level of evidence that would be required by the programs for inclusion in the toolkit, the team had to design a method that would enable proper classification of each program. Moreover, this method had to enable the team to assign a rank to each program in order to identify which practices to include.

The team followed a commonly accepted approach to the identification of promising practices. By establishing a set of review criteria, the team scored each program and generated a standardized rating to equitably compare the merits of each program. Since this was the first project to identify primary practices in the TANF field, the team developed two new assessment tools, one for TANF processes and one for TANF services.2 To simplify the review process, these criteria were designed to be broad and encompass key features of programs serving TANF clients. Criteria were developed by project staff and field tested with the project workgroup and the Office of Family Assistance to ensure their validity. Pilot testing the review criteria with experts helped ensure the standards were relevant and responsive to the needs of the field (see Appendix 2 and Appendix 3).

Scoring and Staff Review: Program scoring and final program selection were decided by consensus among project staff. Consensus decision-making also has been used by the Emerging Practices registry for the Office on Child Abuse and Neglect. The review team included a section lead for each service area who reviewed all programs in one point along a continuum of TANF services, TANF experts who had been working on TANF and related issues for several years, and a senior expert with extensive experience in human services research and technical assistance.

ICF staff rated the level of evidence in accordance with the review criteria for top-tier programs. Levels of evidence were ranked using point values such that Promising Practices received three points, Common Practices received two points, and Innovative Practices received one point. Each program also received one point for every key feature in the review criteria that was included in the program design. A total score summed the number of points awarded for each program relative to the total number of applicable points for that section. For a number of programs, certain questions were not applicable to their mission, while for others it was difficult to determine whether the features existed, given the limited time available for program selection. In these cases, not applicable responses were not counted in the denominator so the overall program score was not affected negatively.

To make the final selections, the review team met and was presented with program synopses that described program design, the extent of evidence on program success, innovation within the program design, and a program's overall score. The review team discussed each top-tier program, using the total score as a guide; however, to identify innovative practices that were not captured by the assessment tools, the review team did not rely solely on the total score. This deliberative process allowed for more careful consideration of the merits of each program, as well as for consideration of programs that were not included initially. The review team came to a consensus about the programs to feature from each TANF service area along the continuum of TANF services.

During this process, the review team noted that a number of top programs were located in the same cities. As a result, the team developed a roadmap of best practices, centered on 14 key cities and one county in which most of the successful practices were clustered. Top-ranked programs were selected for case study review, while lower-ranked programs were selected for gathering highlights by telephone, unless they were situated near program clusters. Unranked programs were not featured in the toolkit, but were integrated into the environmental scan.

After the initial selections, a number of TANF stakeholders reviewed and approved program selections prior to site notification and data collection. This approval process included representatives from the Office of Family Assistance, the U.S. Department of Labor, and the 10 TANF Regional Offices, as well as the Urban Partnerships Initiative city team leads.

Data Collection and Analysis: Because so many programs were clustered, the project team was able to plan site visits with 47 of the 83 selected top programs. Team members notified programs about their selection for inclusion in the toolkit and scheduling phone interviews or site visits, as feasible and appropriate. All but 14 programs responded to these calls and were willing to schedule phone interviews or site visits with team members.

A data collection instrument was developed to guide team members (see Appendix 4). Not all questions were appropriate for all programs; however, they were included to provide a consistent approach to collecting information. Team members also were given points to consider for recognizing additional details about the environment in which the program was implemented; for example, the accessibility of the office location or the demeanor of the staff (see Appendix 5).

For the sites contacted by telephone, key administrators were asked to provide essential information on program design and implementation. For sites that were visited in person, program staff and administrators, as well as external stakeholders, were interviewed when possible. Site visits lasted up to a full day, depending on the type of program and availability of staff, and involved partners at all levels, including line staff, supervisors, administrators, community stakeholders, and clients. Assessment tools, forms, memoranda of understanding, curricula, and other tools used to implement the programs were obtained both from sites that received calls and those that received visits. The tools were modified to be compatible with the electronic format of the toolkit.

Following data collection, two-page program synopses were prepared describing the programs objectives, operations, and keys to successful implementation. Contact information was obtained to facilitate peer-to-peer exchanges about implementation, modification, and replication. The descriptions were shared with the sites to ensure they presented the programs accurately.

After compiling the program descriptions and tools, obtaining approvals from the sites, and ensuring consistency across the format of the program information, the materials were uploaded to a searchable, user-friendly, Web framework. This online collection of innovative practices in TANF service delivery is now accessible to the Urban Partnerships Initiative cities and the larger community of TANF stakeholders.

1 Program selection procedures developed by the Resource Center at the Corporation for National and Community Service and the Research to Practice Initiative at the Child Welfare League of America both allowed for the inclusion of innovative programs that had not yet been formally evaluated.
2 TANF processes include intake and case management. TANF services include work attachment, work retention, and transitional services.