The world’s largest education impact bond delivers on results midway through the program

A gregarious third grade student, Bhairavi, loves school and particularly loves working collaboratively with her classmates on group projects and hands-on, experiential learning. She believes “studying will allow me to find a good job, through which I can work hard and support my family.” Bhairavi’s school is run by Gyan Shala, which provides unique learning environments and support for students and teachers alike, primarily in underprivileged areas in the state of Gujarat in India. One such teacher is Naghma, who was herself a Gyan Shala student, and after recently completing grade 12, has returned to serve as a teacher for one of the schools in her community.

Bhairavi and Naghma’s Gyan Shala schools are supported as part of a program known as the Quality Education India Development Impact Bond (QEI DIB), which aims to improve education outcomes for over 200,000 primary school students over a four-year period from 2018 to 2022. These students are in four regions of India, and represent students in government and private schools, as well as program-specific learning centers.

About the QEI DIB

Like other impact bonds, the QEI DIB shifts the focus from the achievement of inputs to outcome achievement. UBS Optimus Foundation provided $3 million of upfront risk investment, which is reinvested in the program annually if outcomes are achieved for the previous year (known as capital recycling). In total, the program has an outcome funding pot of $9.2 million, provided by the Michael and Susan Dell Foundation, Comic Relief, Larry Ellison Foundation, and Mittal Foundation. The consortium of funders, convened by British Asian Trust, covers service delivery costs and an interest rate up to 8 percent per annum if outcomes are met, and incentive payments to service providers if targets are exceeded. An additional $1.4 million in evaluation and performance management support has been funded primarily by the Foreign, Commonwealth & Development Office (formerly DFID) and the UBS Optimus Foundation.

The QEI DIB is much larger than its predecessor, the Educate Girls DIB, which came to a close in 2018.  Not only does the QEI DIB serve many more students, but it is also unique in that it features four different service providers, each delivering a different intervention. After one year of the program operating with three service providers, a new partnership with Educational Initiatives and Pratham InfoTech Foundation (El-PIF) was formed, bringing an ed tech solution into the mix. There was also an additional region added for one of the original service providers, Kaivalya Education Foundation (KEF). Based on the ethos of adaptive learning that the impact bond model embraces, following year 1 results, one model was discontinued—an indirect model from Society for All Round Development (SARD)—when achievement was below the expected performance that underpinned the selection of the program. The year 2 service providers and interventions that they delivered are described in detail in Table 1 below.

Table 1. Quality Education India Development Impact Bond interventions and targets

Results framework

Given the dire statistics on learning across India, the QEI DIB seeks to improve the literacy and numeracy outcomes of children in marginalized communities. Whether the four interventions in the program achieve this learning is measured through an independent evaluation conducted by ConveGenius Insights (formerly Gray Matters India or GMI), using a quasi-experimental design. Learning targets are measured through the GMI tests—written and oral assessments developed locally in accordance with the curriculum. These learning targets equal the difference in growth in learning over time between those receiving the interventions and a comparison group. The targets are based on expected intervention effect sizes taken from a meta-analysis of global and Indian literature on comparable education interventions. Across all service providers and across subjects, the average expected learning gain is 25 scale score points in a given year. Payments to the investor, the UBS Optimus Foundation, are based on a price per outcome times the learning outcome times the number of students enrolled and are reinvested into the program at the end of each year if outcomes have been achieved and there are positive returns. A cap is placed on cumulative outcomes in a given year (not more than 125 percent of the outcome) and a maximum internal rate of return of 8 percent at the end of the four-year DIB. This translates into a base case potential return of $745,000 for the investor.

Year 2 learning achievement results

Two years into the impact bond, average results are promising for the three interventions that were evaluated (an evaluation was not conducted for the fourth intervention due to the COVID-19 pandemic). As can be seen in Table 2, the four service providers reached 104,833 students across 711 schools over the two-year period. On average, the cumulative achievement across all grades over two years demonstrates that QEI DIB schools substantially overperformed relative to targets.

The table shows that there are different targets for each service provider; this is related to the expectations around what each type of intervention could potentially achieve based on the literature. There are also differences in achievement between service providers: For example, while KEF achieved a scale score that was two to three times the target, EI-PIF and SARD with their direct model achieved 7.5 and six times the target, respectively. Disaggregated results by literacy and numeracy demonstrate that all assessed service providers overperformed except for KEF in Gujarat, which underperformed for grade 4 in language and math (while overachieving in both subjects for grade 3) relative to the comparison group.

Table 2. Overall performance on learning outcomes

Drivers of these results

The successful second-year results of the program offer many opportunities for further examination and learning. A few in particular stand out as possible key drivers of these results.

As Brookings has previously written, the high-stakes environment of a development impact bond, and the scrutiny that provides, is a key piece of the theory of change for driving more and better results for any impact bond. In the case of the QEI DIB, this provided a mission-mode focus for the service providers—for example, EI-PIF adopted an initiative titled “Mission 35” which targeted using 35 minutes of the program per student per session and also doubled up classes when devices were available to maximize access to the intervention. The high-stakes environment of a DIB also leads to some challenging decisions—for example, dropping the SARD-Indirect model and adding the EI-PIF, a program with a very successful track record prior to the DIB. However, this also may have pushed the SARD-direct model in year 2 to achieve its high performance.

Additionally, Brookings has identified active use of data as central to outcomes-based financing. In the case of the QEI DIB, the service providers were able to tap into existing data-driven cultures and systems to actively manage and validate programmatic decisionmaking. One example of this is the Journey application, which KEF uses to provide school leaders with real-time data on indicators across seven categories of learning inputs (including parent contact, teacher capacity, and remedial classes) that are tied to the intervention’s outcomes achievement. Another example is from EI-PIF, which has a data-minded organizational culture, including through access to daily and weekly email trackers and individual student and school evaluations. Lastly, SARD actively assigned its support services to schools based on performance. One potential challenge for the QEI DIB, as well as other impact bonds, is the balance between focusing on the outcome metrics stated in the impact bond terms (such as math and reading) to the detriment of other subject areas. This is an area to examine in all impact bonds more closely in the future.

While the above drivers plausibly explain the success of the programs in the context of the impact bond, there may be other explanations for the substantial overperformance in learning (up to 7.5 times targets in one group). One explanation is that on average, students were performing at a lower level than expected for their grade at baseline, which left more room for improvement in learning. Another is that during these past two years, the comparison group has been performing below expectations, which increases the gap between the treatment and comparison groups and leads to a larger effect-size.

Learnings and looking forward

The results from the first two years of the QEI DIB are promising, with many children experiencing substantial learning gains in QEI DIB-supported classrooms and programs. The key issue facing the QEI DIB and many other education initiatives around the world is tackling learning losses that will likely result due to the COVID-19 school closures. The impact this will have on learning relative to the predetermined targets, and challenges to those targets, is yet to be seen.

Additionally, as the QEI DIB crosses the halfway point into the second half of implementation, there will be pressure to ensure that learning gains continue to rise. With an evaluation methodology structured around annually measured learning gains relative to a control group, overperformance in one year could potentially lead to challenges the following year(s) maintaining that same level of performance. This is because rates of gain on most human development indices often begin to taper at higher levels of performance, as there is less room for improvement at these higher intervals. For example, in education programs the literature suggests that substantial learning gains for students at higher proficiency levels are more difficult to achieve than for those at lower proficiency levels, who are catching up with their peers. This challenge of overperformance in one year leading to difficulties for the following year’s results is one which permeates all impact bonds with an annual evaluation design. In the case of the QEI DIB, this challenge could (unfortunately) potentially be mitigated by learning losses due to the COVID-19 pandemic.

The drivers of the year 2 QEI DIB results, the challenges experienced during implementation, and broader lessons for the sector from the QEI experience were discussed in an event hosted by Brookings on October 15. The robust conversations at this event add more depth and nuance to the QEI experience from the perspectives of the investor, outcome funder, service providers, and evaluator. We encourage you to watch the recording and read our summary of that conversation.

Brookings is committed to quality, independence, and impact in all of its work. Activities supported by its donors reflect this commitment and the analysis and recommendations are solely determined by the scholar. Support for this publication was generously provided by the British Asian Trust.