WEBVTT 00:04.250 --> 00:08.580 TANF programs want to improve services for families to help lift them out of poverty, 00:08.580 --> 00:11.530 but it’s hard to tell what will actually work for them. 00:11.530 --> 00:17.080 Too often, those running programs simply don’t have the time or the resources for evaluations 00:17.080 --> 00:19.010 or to sift through the existing research. 00:19.010 --> 00:23.550 But what if there was a way to use the data, programs already collect, to provide timely 00:23.550 --> 00:26.480 evidence about what works to improve services for families? 00:26.480 --> 00:30.180 There is, and it’s called Rapid Cycle Evaluation. 00:30.180 --> 00:32.180 Meet Pam. 00:32.180 --> 00:36.790 She runs a TANF program in a mid-sized urban county, and knows from experience that clients 00:36.790 --> 00:40.980 that attend orientation are more likely to enroll in the program and receive the help 00:40.980 --> 00:41.980 they need. 00:41.980 --> 00:45.260 But, only half of eligible families are showing up for orientation. 00:45.260 --> 00:49.989 If she can increase attendance at orientation, she’s more likely to improve the lives of 00:49.989 --> 00:52.530 the families she has spent her career trying to help. 00:52.530 --> 00:56.289 She’s heard that simple “nudges”—like mailing a reminder postcard-- can encourage 00:56.289 --> 01:00.399 participation and is curious if there’s something similar she could do to get more 01:00.399 --> 01:03.950 people to attend the orientation and enroll in the program. 01:03.950 --> 01:09.030 First, Pam and her team spent a bit of time exploring why only half the people were going 01:09.030 --> 01:10.030 to orientation. 01:10.030 --> 01:15.310 They found the enrollment paperwork had become long and confusing, making it hard for clients 01:15.310 --> 01:19.320 to remember the important next step of coming to orientation. 01:19.320 --> 01:23.560 Could something as simple as providing appointment reminder cards make a difference? 01:23.560 --> 01:26.680 Even small changes like this come with high stakes. 01:26.680 --> 01:31.070 She could buy materials, train staff, and roll out her idea to the hundreds of people 01:31.070 --> 01:34.770 she serves each month, but what if the cards don’t work? 01:34.770 --> 01:40.130 Pam can’t afford to waste those resources and that time without actually improving orientation 01:40.130 --> 01:41.130 attendance. 01:41.130 --> 01:43.920 That’s where Rapid Cycle Evaluation can help. 01:43.920 --> 01:48.100 She can quickly test her idea using data she already collects, before rolling it out to 01:48.100 --> 01:49.540 the whole program. 01:49.540 --> 01:55.400 First, Pam designed a reminder card and piloted it by asking a few case managers to use it 01:55.400 --> 01:58.700 with a few clients, and to give her feedback on how it worked. 01:58.700 --> 02:03.369 Based on feedback, they updated the design and included the address on the card. 02:03.369 --> 02:06.420 Then, they developed a detailed plan for further testing. 02:06.420 --> 02:12.490 A small, simple, low-cost intervention like a reminder card won’t address all the reasons 02:12.490 --> 02:17.250 why clients struggle to make their orientation sessions, so she knew she needed to define 02:17.250 --> 02:19.930 success before going any further. 02:19.930 --> 02:25.060 She determined that improving orientation attendance by 10 percentage points was a meaningful 02:25.060 --> 02:26.060 improvement. 02:26.060 --> 02:30.920 An evaluation partner looked at previous research and told her this was a reasonable outcome. 02:30.920 --> 02:35.310 They knew what they wanted to do, and how they would measure success, but how would 02:35.310 --> 02:38.560 they know if it was the cards making the difference? 02:38.560 --> 02:44.180 Randomly assigning some participants to receive the reminder card, and some to get the business-as-usual 02:44.180 --> 02:49.010 paperwork would allow Pam to compare the orientation attendance rates between the groups and determine 02:49.010 --> 02:51.610 whether the new strategy was working. 02:51.610 --> 02:55.379 Pam and her evaluation partner looked at the number of clients enrolling in the program 02:55.379 --> 02:59.410 each week and determined it would take about 12 weeks to tell whether the strategy was 02:59.410 --> 03:00.730 paying off. 03:00.730 --> 03:05.400 Throughout the testing period, Pam talked with staff and clients to understand and document 03:05.400 --> 03:07.060 how the cards were working. 03:07.060 --> 03:11.709 After 12 weeks, they compared the average attendance rate of clients who received the 03:11.709 --> 03:15.890 new card with the average attendance rate of those that didn’t and found a difference 03:15.890 --> 03:20.720 of 8 percentage points—which didn’t quite meet her benchmark. 03:20.720 --> 03:25.819 Based on feedback from staff and clients, Pam updated the design of the cards, added 03:25.819 --> 03:31.310 a phone number to reschedule orientation appointments and re-tested the improved version. 03:31.310 --> 03:37.800 After the second test, Pam and her evaluation partner found a difference of 13 percentage points. 03:37.840 --> 03:42.280 The results exceeded her definition of success of 10 percentage points and Pam decided to 03:42.280 --> 03:44.319 scale up the reminder card program-wide. 03:44.319 --> 03:50.620 Based on her current numbers, that 13 percentage point improvement in orientation attendance 03:50.620 --> 03:56.310 means Pam can expect about 150 more families each year to enroll in the program and start 03:56.310 --> 03:58.800 on their path to economic security. 03:58.800 --> 04:01.740 Not bad for a simple reminder. 04:01.740 --> 04:06.580 For a reminder about how Rapid Cycle Evaluation can help you, plan a strategy that addresses 04:06.580 --> 04:10.780 the problem and is meaningful enough to plausibly have an effect. 04:10.780 --> 04:12.950 Decide how you will measure success. 04:12.950 --> 04:18.440 Implement the strategy and monitor implementation and review/revise and use evidence to inform 04:18.440 --> 04:22.880 decision-making, visit our website at the link shown here.