Contracted Evaluation of the School District of Philadelphia’s Renaissance Initiative

About the evaluation

Over the last decade, the School District of Philadelphia (SDP) embarked on a critical effort to improve the outcomes of students in its low-performing schools. This effort, aimed to dramatically improve student achievement by shifting school management to either external charter providers, called Renaissance Charters, or by placing schools in a network that receives additional district supports, is named the Renaissance Initiative. The District’s School Reform Commission (SRC) requested a mixed-methods evaluation of the Renaissance Initiative to better understand how reforms were implemented and the effectiveness of those reforms on student outcomes. The District selected Mathematica and its partner, Research For Action, to conduct a three-year evaluation from 2017-2020. Evaluation activities included a detailed literature review, a comprehensive mixed-methods study, and a cost study.

Key Research Questions

The key questions the three-year study examined were:

  • How did the seven schools that transitioned to a Renaissance Charter or to a district-run turnaround school in 2016-2017 implement the transition? What successes and challenges did the schools encounter?
  • Among schools transitioning to a district-run turnaround school or a Renaissance Initiative School between 2013-2014 and 2017-2018, is there evidence that transitioning improved SPR scores?
  • Is there evidence of differences in impacts between Renaissance Charter schools and district-run turnaround schools?
  • Is there evidence that schools transitioning in different years were more effective at improving SPR measures?
  • How did schools change how they allocated funds once they became Renaissance Charter or a district-run turnaround school?

Click here to view the full evaluation results slide deck shared with the School Board. Products from each of the study years are linked below.

Year 1

Overview

Evaluation activities in the first year included a literature review and an implementation study. The literature review grounded the evaluation in a better understanding of the research around turnaround strategies enacted by districts and charters, and whether the current strategies that SDP is implementing remain the strongest options. The implementation study documented the experience of seven schools that entered the Renaissance Initiative in the 2016-2017 school year, describing schools’ implementation of turnaround supports based on participants’ reported experiences.

Key findings

Turning around schools requires extensive ongoing work and is a demanding process. Staff and parents in some schools acknowledged improvements in culture and climate, often attributed to a sense of cohesion and collaboration among the staff. Other schools, though, continue to face substantial challenges despite two years of receiving supports. Several key findings emerged:

  • School leaders who promote a shared vision of collaboration motivate staff. Staff noted that school leaders influence the success of turnaround efforts. Actions such as collaborating in planning meetings and acknowledging student and staff accomplishments facilitate a shared vision and motivate staff to work toward successfully turning around the school.
  • Some turnaround supports could be more effective with increased collaboration or differentiation. Staff in most schools reported receiving adequate supports but noted that these supports could be used more effectively.
  • Addressing behavior and trauma remains paramount to improving student academic achievement. Across schools, behavior and nonacademic issues continue as major challenges to successful turnaround. Accordingly, almost all staff noted a need to prioritize health and safety over the other Renaissance Initiative supports.
  • Programming and resources that demonstrate a school’s mission to support students and families have largely resolved initial community resistance. Staff and parents reported a negative perception of and initial resistance to the Turnaround Network or Renaissance Charter designation. Staff in most schools reported that academic and family engagement programming and providing resources to community members helped improve community support and parental involvement over the past two academic years.

Year 1 Products

 

Year 2

Overview

During the second year of the study, we estimated the impact of the Renaissance Initiative after two years on school- and student-level outcomes in the seven schools that entered the Renaissance Initiative in 2016-2017 (two charters and five district-run turnaround schools). We used an analytic design that adjusts for fixed differences between treatment and comparison schools, and for any district-wide changes that affect both the treatment and comparison schools at the same time. We also excluded all schools receiving district supports before 2018-2019 school year from the comparison group.

Key findings

Among the seven schools that entered the Renaissance Initiative in 2016-2017, we found evidence that overall SPR scores improved over two years. This improvement is driven by gains in climate and progress scores. Evidence suggests that climate improved more in Renaissance Charter schools, but we found no evidence of improvement in achievement scores over two years. Among students in the five district-run turnaround schools, we found no impacts on behavior or achievement outcomes over two years, but estimates are imprecise.

Year 2 Product

 

Year 3

Overview

In the final year of the evaluation, we updated the analyses of the seven schools that entered the Renaissance Initiative in 2016-17. We also added more schools to the analysis in order to examine how different cohorts of schools that entered turnaround at different points in time were impacted by the transition. The analysis allowed us to learn about the amount of time needed for improvement but did not break out the analysis by model (beyond charters vs district run schools). The other culminating evaluation activity was to explore schools’ cost allocations to learn about how schools that went into turnaround used their budgets to support improvements.

Key findings

Evidence indicates that transitioning to a district-run turnaround school or Renaissance Charter school improved the SPR measure of climate. We also found evidence that Renaissance Charters were more effective at improving climate than district-run turnaround schools, and there was some evidence that district-run schools were more effective than comparison schools at improving progress scores. Evidence indicates that schools in the 2013-2014 and 2016-2017 cohorts were more effective at improving climate compared with the 2017-2018 cohort, and that schools in the 2013-2014 cohort have been more effective at improving achievement than the two other cohorts. Among students at the five schools that became a district-run turnaround school beginning in the 2016-2017 school year, we found no effects on behavior or academic outcomes compared with students in comparison schools. However, we found evidence that lower-achieving students in those schools benefited most from joining the Turnaround Network in 2016-2017. Relative to students in the comparison schools (similar schools that did not enter turnaround), lower-achieving students in the 5 district-run turnaround schools that transitioned in 2016-2017 improved ELA and math scores more than the higher-achieving students in those schools.

The cost study found that the timing of turnaround may have influenced spending decisions. Discretionary spending was limited by model specifications, but there was room for tailoring and opportunities for additional sources of funding. Early investments in school climate and administrative spending may be helping to drive positive impacts. The impact analysis suggests improving academic achievement takes time. However, even in the early years, those with strongest improvements invested more in general and special education expenses. Deeper learning about Renaissance Charters’ impact and spending could occur with more fine-grained data from those schools.

Year 3 Product