Appendix 1

To develop criteria by which to select programs/practices for the Urban Partnerships Initiative OnlineToolkit, other selection criteria were reviewed first. Numerous organizations, including governments, universities, and nonprofits, manage databases of promising or evidence-based practices. These registries cover a wide range of topics such as health care, treatment approaches to substance abuse and mental illness, education, child abuse, and violence prevention, and use different approaches to identifying and rating practices.

Listed below are the registries that served as examples, along with the topic area, approach to practice identification, and levels of program ratings. Based upon the needs of this project, the criteria used by the toolkit team to assess these approaches included:

  1. Level of Rigor - Are programs required to have been evaluated? Does the evaluation require an experimental design?
  2. Speed of Review - Are programs reviewed in a timely fashion? How long does the review process take?
  3. Innovative Programs - What is the likelihood of identifying new or innovative practices?
Registry/Author Subject Area Approaches to Identify Practices Program Levels High Rigor Speed Innovative Programs

Blueprints for Violence Prevention (Office of Juvenile Justice and Delinquency Prevention)

violence prevention
  • initial review by Center for the Study and Prevention of Violence and a final review and recommendation from an advisory board (7 experts)
  • model programs
  • promising programs

Yes

No (11 model, 18 promising identified since 1996)

No

The California Evidence-based Clearinghouse for Child Welfare (The Chadwick Center)

child welfare
  • standard review process: theoretical soundness, clinical support, acceptance within the field, and potential for harm, documentation, and empirical support
  • well supported, efficacious practice
  • supported and probably efficacious practice
  • supported and acceptable practice
  • promising and acceptable practice
  • innovative or novel practice
  • concerning practice

Yes

?

Yes but focus is on particular priority areas

Emerging Practices (Office on Child Abuse and Neglect)

child abuse and neglect (child welfare)
  • nomination and application process
  • 12-person advisory group (field experts) consensus decision
  • effective programs
  • demonstrated
  • reported
  • innovative programs
  • programs with noteworthy aspects

Yes (experimental design)

No

(2-year consensus process)

Yes

Evidence-based Program Database (Ohio State University)

youth behavior
  • evidence-based programs recommended by research-oriented government agencies, nonprofit agencies, and independent publications
  • model program
  • promising program

~

(depends on where the practice was pulled from)

No

Unlikely

National Registry of Evidence-based Programs & Practices (Substance Abuse and Mental Health Services Administration)

mental illness & substance abuse
  • application submitted during review cycle
  • reviewers calculate ratings using the predefined review criteria
  • demonstrates one or more positive outcomes 
  • results have been published in a peer-reviewed publication or comprehensive evaluation
  • implementation documentation is available to the public

Yes (positive evaluations)

No (3-4 month submission cycle each fiscal year)

No

Promising Practices Network (RAND)

children and youth
  • nominations of designed and evaluated programs
  • review by RAND staff
  • proven program
  • promising program
  • not listed on site

Yes (all experimental)

No (evaluations must be public)

No

Promising Practices Report of Fatherhood Programs (National Fatherhood Initiative)

fatherhood
  • unknown selection process
  • proven program
  • promising program
  • emerging program

Yes (experimental design)

?

Unlikely

Research to Practice (Child Welfare League of America)

child welfare
  • nomination/application process
  • team uses evaluative tools to examine a program's existing research
  • exemplary practice
  • commendable practice
  • emerging practice
  • innovative practice

Yes (all have some research component)

?

Yes

The Resource Center (Corporation for National and Community Service)

volunteer and community service
  • a set of objective criteria
  • staff review submissions for appropriateness and relevance
  • a good idea
  • implemented and has solved an issue or problem for one program
  • implemented and has solved an issue or problem for more than one program
  • an effective practice that has been identified through evidence-based study

No

Yes

Yes

What Works Clearinghouse (U.S. Department of Education)

education
  • teams of reviewers collect, screen, and identify studies of the effectiveness of interventions
  • meets evidence standards
  • meets evidence standards with reservations
  • does not meet evidence screens

Yes (randomized controlled trials)

No

No

Urban Partnerships Initiative Toolkit (Office of Family Assistance)

TANF
  • nomination process (all programs will be recommended)
  • technical work group reviewed criteria
  • team review based on criteria and consensus decision
  • promising practice
  • common practice
  • innovative practice

~

(not experimental)

Yes

Yes

Based upon the needs and scope of work of the Urban Partnerships Initiative Online Toolkit, we proposed the following measures in devising the assessment criteria:

  1. Level of Rigor - Given the nature of the TANF programs that will be included in the toolkit, it is unlikely that many will have undergone rigorous evaluation. As judgments regarding program effectiveness cannot be consistently drawn in the absence of outcome data, modified criteria were developed on which to assess the level of evidence the programs possessed.
  2. Speed of Review - The timeline for the toolkit was compressed, limiting the scope of the project, as it would not be feasible to identify all promising TANF practices across the country. It also was not possible to thoroughly evaluate the programs for success, necessitating a reliance on program data and anecdotal evidence.
  3. Innovative Programs - Due in part to recent regulatory changes, it is important to the field that new practices are shared. Therefore, assessment criteria should not eliminate their inclusion due to a lack of evidence of success.

Listed below is the evaluative criteria that served as part of the overall program assessment:

  • Promising Practice - has outcome evaluations with positive results
  • Common Practice - widespread approach in the field (at least five cities) with practice evidence (e.g., reduced caseload)
  • Innovative Practice - a new/novel program that suggests success based upon theory or practice experience