Promising Evidence on Personalized Learning

The Bill & Melinda Gates Foundation has engaged RAND to carry out an ongoing study of foundation-funded schools that are employing promising approaches to personalized learning. This research is part of the foundation's public commitment to spread effective practices across districts and charter networks, develop innovative roles for teachers, and support implementation of college-ready standards. This is the second report in a series focused on the achievement data, school design characteristics, and teacher and student perceptions of schools that are implementing personalized learning. The achievement findings in this report focus on 62 public charter and district schools that are pursuing a variety of personalized learning practices. In a smaller set of 32 schools, the report examines details of personalized learning implementation and the relationship of implementation to outcomes. The concept of personalized learning has been around for some time, but the adoption of personalized learning approaches has increased significantly in recent years due in part to rapid advances in technology platforms and digital content. Although there is not yet one shared definition


Executive Summary
The Bill & Melinda Gates Foundation has engaged RAND to carry out an ongoing study of foundation-funded schools that are employing promising approaches to personalized learning.
This research is part of the foundation's public commitment to spread effective practices across districts and charter networks, develop innovative roles for teachers, and support implementation of college-ready standards. This is the second report in a series focused on the achievement data, school design characteristics, and teacher and student perceptions of schools that are implementing personalized learning. The achievement findings in this report focus on 62 public charter and district schools that are pursuing a variety of personalized learning practices. In a smaller set of 32 schools, the report examines details of personalized learning implementation and the relationship of implementation to outcomes.
The concept of personalized learning has been around for some time, but the adoption of personalized learning approaches has increased significantly in recent years due in part to rapid advances in technology platforms and digital content. Although there is not yet one shared definition of personalized learning, leading practitioners in the field generally look for the following: (1) systems and approaches that accelerate and deepen student learning by tailoring instruction to each student's individual needs, skills, and interests; (2) a variety of rich learning experiences that collectively prepare students for success in the college and career of their choice; and (3) teachers' integral role in student learning: designing and managing the learning environment, leading instruction, and providing students with expert guidance and support to help them take increasing ownership of their learning.
Although these core principles are common among the schools in the study, there is considerable diversity in the details of the schools' instructional models because innovation was encouraged in the competitive grant programs they participated in. Each school received funding after rigorous evaluation of its leadership team and instructional

Student Achievement Results
Students made gains in mathematics and reading that were significantly greater than a comparison group made up of similar students selected from comparable schools. These results are consistent with the findings reported last year but with a sample nearly three times as large. The gains in both mathematics and reading at schools that have implemented personalized learning approaches were relatively large compared with gains in studies of schools with other types of interventions. A majority of the schools had statistically positive results. Importantly, although students started out mostly performing below the national averages in mathematics and reading, they generally ended with scores near or above the national averages after two years in personalized learning schools.These results suggest that the effects of personalized learning on student achievement are promising.

School Design Characteristics
The implementation findings rely on data collected in schools implementing personalized learning with funding from the NGLC program. There are 32 such schools that also administered the Northwest Evaluation Association's Teachers reported using a variety of data and other resources to inform their instructional decisions, and all administrators reported that their schools used data from different sources to understand student progress. About half of administrators reported that their schools were developing personalized goals for students, and two-thirds were providing data to and discussing data with students.   This was particularly true for schools that opened in 2012.
The administrators cited high staff turnover as a common problem. Site visit participants suggested that mid-year teacher departures were disruptive, particularly in new schools, which tended to have smaller staffs.

Relating Implementation to Outcomes
While no single element of personalized learning was able to discriminate between the schools with the strongest achievement outcomes and the others in the sample, we did identify three elements that, when present together, were able to do so. Among all the schools in the analysis, the most successful schools were the only ones that showed strong evidence of implementing all three: (1) student grouping strategies driven by data and responsive to student needs; (2) providing data to students and including them in discussions about how the data relate to the students' personal learning goals; and (3) learning spaces that are supportive of, or do not hinder, implementation of other personalized learning strategies.

National Comparison of Survey Results
This study compared the teacher and student survey results from the 32 schools in the implementation analysis to results from administering nearly identical questions to a national sample of teachers and students. Grunwald Associates administered the national survey during summer 2015.
The national results are intended to provide context for the findings from the personalized learning schools to help understand the ways in which the experiences of students and teachers in these schools differed from the experiences of students and teachers nationally. To facilitate this comparison, we first weighted the national survey results to more closely reflect the personalized learning sample in terms of geographic locale (e.g., urban), grade level, subject taught (by teachers), and gender (of students). However, we lacked the necessary data to include family income in the student survey weighting process and the national sample appears to be somewhat more affluent than the personalized learning sample.
The use and characteristics of learner profiles and emphasis on student choice in the personalized learning schools were similar to those in the national sample. However, teachers in personalized learning schools were more likely than those in the national sample to use technology for personalization and to report that their school's data system is useful. Teachers in the personalized learning sample also reported greater use of instructional practices that support competency-based learning. Students in both samples agreed or strongly agreed that there was an emphasis on making them aware of instructional goals and tracking progress toward mastery. Finally, students in the personalized learning schools were more likely to report that they were able to make choices about their learning and that their mathematics and ELA instruction incorporated aspects of complex, student-centered instruction.
The adoption of personalized learning approaches has increased significantly in recent years due in part to rapid advances in technology platforms and digital content.

Continued Progress
Promising Evidence on Personalized Learning

Conclusions
Although implementation of personalized learning varied considerably across the 32 schools studied, our findings suggest that the schools were employing a number of practices that support personalization. Teachers at most schools were using data to understand student progress and make instructional decisions, all schools offered time for individual academic support, and the use of technology for personalization was widespread. However, some strategies, such as competency-based progression, were less common and more challenging to implement. The schools that exhibited the greatest achievement growth were all implementing three elements of personalized learningstudent grouping, learning spaces that support personalized learning, and opportunities for students to discuss their learning data with teachers.
The achievement findings are largely positive and promising.
They indicate that compared to peers, students in schools using personalized learning practices were making greater progress over the course of two school years and that those students who started out behind were catching up to perform at or above national averages. We conducted a set of rigorous sensitivity analyses and concluded that they generally support these results. However, it is important to use caution in interpreting these results as causal effects of personalized learning. Although the analyses used the best estimation strategies possible given the nature of the data and limits to implementing a strong experimental design, it was not possible to separate personalized learning effects from other potential school effects.
RAND will produce a more comprehensive report with additional details in 2016.

About the Comparison Group
Despite the increased interest in personalized learning, the field lacks evidence about its effectiveness. This study is designed to address this need using the most rigorous method that could be applied in the circumstances, namely a matched comparison group. Northwest Evaluation Association, through its standard service known as "virtual comparison group" (VCG), drew on its large national database of testing data to identify students who had starting performance similar to the personalized learning students and who were attending schools serving similar populations. This enabled us to make "apples to apples" comparisons of learning growth between the students at the personalized learning schools and a similar population of students attending other schools.
The observable characteristics of the comparison students were well matched to those of personalized learning students in the 62 schools in the achievement analysis. However, the comparison students could possess other unidentified or unobserved differences from the personalized learning students that could confound efforts to measure the impact of the personalized learning environment. Such differences could bias our estimates of personalized learning treatment effects in either direction. The VCG approach also assumes that the students in the comparison groups are attending more traditional schools that are not using personalized learning practices, but there is no way to verify this assumption.

Continued Progress
Promising The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis.
Guided by the belief that every life has equal value, the Bill & Melinda Gates Foundation works to help all people lead healthy, productive lives. In developing countries, it focuses on improving people's health and giving them the chance to lift themselves out of hunger and extreme poverty. In the United States, it seeks to ensure that all people-especially those with the fewest resources-have access to the opportunities they need to succeed in school and life. Based in Seattle, Washington, the foundation is led by CEO Susan Desmond-Hellmann and Co-chair William H. Gates Sr., under the direction of Bill and Melinda Gates and Warren Buffett.
To download the Continued Progress report, visit www.rand.org/t/RR1365.

For More Information
Visit RAND at www.rand.org

Explore the RAND Corporation
View document details

Browse Reports & Bookstore
Make a charitable contribution

Distribution Rights
Reports funded by the Bill & Melinda Gates Foundation are published on "Open Access" terms, under the Creative Commons Attribution 4.0 Generic License (CC BY 4.0) or an equivalent license. All users of the publication are permitted to copy and redistribute the material in any medium or format and transform and build upon the material, including for any purpose (including commercial) without further permission or fees being required.

Research Report
This report is part of the RAND Corporation research report series. RAND reports present research findings and objective analysis that address the challenges facing the public and private sectors. All RAND reports undergo rigorous peer review to ensure high standards for research quality and objectivity.
The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis.
This electronic document was made available from www.rand.org as a public service of the RAND Corporation.