ORE Planned Research Agenda, 2023-24

Our planned 2023-24 research and evaluation projects are listed here by topic. If a project spans multiple topics, we have organized it by its primary topic.

Expand each project description to learn more about the questions we’re asking.

Assessments

Descriptive study of participation on District-wide assessments administered in 2022-23 (includes College and Career assessments)

This is a knowledge-building study. The questions we’re asking include:

  • Which tests were administered by the Office of Assessment in 2022-23 school year?
  • What were the participation rates for these tests?

A Summary of District-Wide Assessments Administered to SDP Students During the 2022-23 School Year

Descriptive study of 2023-24 PSSA results

This is a knowledge-building study. The questions we’re asking include:

  • What percentage of students scored proficient or advanced on the PSSAs this year, and how does it compare to prior years?
  • What percentage of students scored proficient or advanced on the PSSAs this year, and how does it compare to prior years, by grade level?

Descriptive study of 2023-24 Keystone results

This is a knowledge-building study. The questions we’re asking include:

  • What percentage of students scored proficient or advanced on the Keystone exams this year, and how does it compare to prior years?
  • What percentage of students scored proficient or advanced on the Keystone exams this year, and how does it compare to prior years, by grade level?

Descriptive study of Star end-of-year participation, performance, and growth

This is a knowledge-building study. The questions we’re asking include:

  • What were the participation rates for the Star tests throughout the 2023-24 school year?
    a) Did participation rates differ by grade bands?
    b) Did participation rates differ by race/ethnicity?
  • What were the patterns in distribution of students across performance levels throughout the 2023-24 school year?
    a) How did performance levels on Star tests differ by grade bands?
    b) How did performance levels on Star tests differ by race/ethnicity?
  • Did students at each grade level demonstrate growth from fall to spring in the 2023-24 school year?
  • How did performance this year compare to last year?

Study of the relationship between Star computer-adaptive tests and Star curriculum-based measures

This is a knowledge-building study. The questions we’re asking include:

  • What is the relationship between Star CAT and Star CBM performance?
  • How accurately do the Star CBMs predict performance on the Star CAT?

Study of the relationship between student performance on Star and PSSA assessments

This is a knowledge-building study. The questions we’re asking include:

  • What is the correlation between Star and 2022-23 PSSA performance
  • What is the classification accuracy of Star when predicting 2022-23 PSSA performance?

Relationship Between Star Assessments and PSSA Performance, 2022-23

Study of the relationship between student performance on Star and Keystone assessments

This is a knowledge-building study. The questions we’re asking include:

  • What is the correlation between Star and 2022-23 Keystone performance
  • What is the classification accuracy of Star when predicting 2022-23 Keystone performance?

Description of year-over-year PSSA performance among the cohort of 7th grade students who completed the PSSAs in 2022-23

This is a knowledge-building study. The questions we’re asking include:

  • What was the change in the cohort’s PSSA performance level distribution from 2018-19 to 2022-23?
  • What percentage of students maintained, improved, or had decreased performance year-over-year ?

Description of 2022-23 PreK Gold Assessment results

This is a knowledge-building study. The questions we’re asking include:

  • Do PreK 4 children in SDP programs who meet or exceed expectations on the Teaching Strategies GOLD Literacy and Math domains also perform well on the Star and KEI assessments in kindergarten?

Curriculum and Instruction

Evaluation of the implementation of standards-aligned core instructional materials for math

This evaluation is related to strategic action 3.2 in the Accelerate Philly strategic plan. The questions we’re asking include:

  • Year 1: Do teachers, school leaders, and central office support staff believe they have enough materials, training, and support to implement Illustrative Math with integrity/fidelity (as intended)?

Illustrative Math Implementation Study 2023-24: Focus Group Findings

English Learners

Study of English Learner (EL) home language and enrollment trends in the School District of Philadelphia: 2014-15 to 2022-23

This is a knowledge-building study. The questions we’re asking include:

  • Which home languages were spoken by SDP students between 2014-15 and 2022-23?
  • What size was the EL population in District and Alternative schools between 2014-15 and 2022-23

English Learner (EL) Home Language and Enrollment Trends in the School District of Philadelphia: 2014-15 to 2022-23

Study of the enrollment mobility of School District of Philadelphia English Learners

This is a knowledge-building study. The questions we’re asking include:

  • What are the within-year and between-year mobility patterns of students in high school (public and charter) from SY2021-2022 to SY 2023-2024?
    (a) Are English Learner students more mobile than non-English Learner students? Do EL students have distinct within-year or between-year mobility trends?
    (b) What are the differences, if any, in Long-Term English Learner (LTEL) students’ mobility patterns, compared to English Learner students?
  • Do EL students have distinct mobility trends and reasons for changing schools as indicated by their exit codes?

This is a Philadelphia Education Research Consortium (PERC) project. PERC is currently a partnership between Research for Action and ORE.

Evaluation of Quality Teaching for English Learners (QTEL) professional development, and coaching

This evaluation is required by a grant. The questions we’re asking include:

Part 1 – QTEL Summer Institute:

  • To what extent were participants satisfied with the QTEL Summer Institute, and did they perceive their participation as being useful and beneficial to their practice?
  • How did participant knowledge of quality instruction for ELs change after participation in the QTEL Institute?
  • To what extent do participants agree with statements about their ability to teach ELs and their attitudes and beliefs related to teaching ELs after the QTEL Institute?

Part 2 – Implementation of the QTEL model and related changes in practices:

  • How frequently did QTEL participants report using QTEL practices during the school year?
  • What did participants report as the primary challenges to implementing the QTEL model?
  • To what extent did participants report implementation of the QTEL model changing their approach to teaching ELs?
  • In what ways did respondents find that implementing the QTEL model benefited students?

Part 3 – Participant attitudes and beliefs after QTEL implementation:

  • To what extent do participants agree with statements about their ability to teach ELs and their attitudes and beliefs related to teaching ELs after the program year?

Implementation and outcomes of the Together is Better program: 2022-2023

This evaluation is required by a grant. The questions we’re asking include:

  • To what extent were teachers satisfied with the support provided by SupportEd?
  • To what extent did teachers collaborate and engage in practices that result in quality instruction in support of ELs?
  • What are participants’ beliefs about teaching EL students and their school’s approach to teaching EL students?
  • What did participants perceive to be the benefits of and challenges to co-teaching, and what supports are needed for co-teaching in general?

A Summary of District-Wide Assessments Administered to SDP Students During the 2022-23 School Year

Health and Nutrition

Case studies of 19 SDP schools receiving SNAP-Ed-funded programming that identify barriers and facilitators to sustainable Policy, Systems, and Environment changes in schools

This study is grant-funded.

  • How do SNAP-Ed implementers and schools understand sustainability and SNAP-Ed’s PSE initiatives?
  • What factors facilitate implementing sustainable PSE initiatives?
  • How do SNAP-Ed implementers and schools understand their roles in the partnership?

Descriptive information from the 2021-22 Youth Risk Behavior Survey (YRBS) results

This study is required by a grant. The questions we’re asking include:

  • What trends exist in Philadelphia youth risk behavior topics overall and across student subgroups?
  • What trends exist in Philadelphia youth risk behavior topics over time?

Youth Risk Behavior Survey: Selected Results from the School District of Philadelphia (2021)

High School Graduation and Beyond

Study of the relationship between 9th grade On-Track and graduation rates for recent cohorts

This is a knowledge-building study.  The questions we’re asking include:

  • What is the relationship between 9th grade On-Track status and on-time graduation for recent cohorts?
  • Have historic relationships between these factors persisted during District efforts to increase 9th grade On-Track rates?

Ninth Grade On-Track Rates and Their Relationship to On-Time Graduation for First-Time SDP Ninth Graders in 2016-17 through 2018-19

Analysis of student experiences that are predictive of passing the Biology, English, and Algebra Keystone exams

This is a knowledge-building study. The question we’re asking is:

  • What data predicts that an SDP student will score Proficient or Advanced on each of the three Keystones?

The When and How of Keystone Exams in the School District of Philadelphia

Descriptive study of pathways of 2022-23 graduates

This is a knowledge-building study. The questions we’re asking include:

  • For students graduating in 2022-23, which Act 158 pathways did they complete?
  • How did pathway completion patterns differ by student demographic groups?

Descriptive information from the 2022-23 Senior Exit Survey

This is a knowledge-building study.  The questions we’re asking include:

  • What post-secondary education plans do students have within one year of graduation?
  • What support systems did students have to help them navigate career and post-secondary education options?

2022-23 Senior Exit Survey: District-Level Report

Descriptive information from the 2024 graduate follow-up survey

This is a knowledge-building study. The question we’re asking is:

  • What are the post-graduation experiences of recent SDP graduates (classes of 2019-20, 2020-21, and 2021-22)?

Results from the 2024 Graduate Follow-up Survey

Descriptive information from the 2022-23 Career & Technical Education (CTE) follow-up survey

This is a knowledge-building study. The question we’re asking is:

  • What do recent CTE graduates report about their post-graduation experiences, and the extent to which those experiences are related to their CTE pathway?

Results from the Spring 2023 CTE Graduate Follow-up Survey

Descriptive information from the 2021-22 and 2022-23 Senior Exit Survey summer melt data

This is a knowledge-building study. The questions we’re asking include:

  • What is the percentage of college-intending seniors who do not enroll in college their first fall after graduation?
  • Did summer melt rates vary by student demographics?
  • Did summer melt rates differ from the previous year?
  • What are some high school experiences associated with a reduction in the odds of summer melt?

Descriptive information from the 2023-24 Senior Exit Survey

This is a knowledge-building study. The questions we’re asking include:

  • What post-secondary education plans do students have within one year of graduation?
  • What support systems did students have to help them navigate career and post-secondary education options?

Evaluation of The Urban Technology Project 2023-24 Americorps Digital Service Fellows program

This evaluation is required by a grant. The questions we’re asking include:

  • To what extent were the mentoring and tutoring sessions implemented as intended?
    • How many students were served mentoring/tutoring through Tech Club during the program year?
    • How frequently did Tech Club mentoring/tutoring sessions occur throughout the program year?
  • To what extent are participating Digital Literacy Teachers and Digital Service Fellows satisfied with the program?
    • To what extent are Digital Literacy teachers satisfied with their Digital Service Fellows and the Urban Technology Project program management, support, and overall programming throughout the program year?
    • To what extent are Digital Service Fellows satisfied with the program management, support, and overall programming throughout the program year?
  • To what extent did Digital Service Fellows’ technical, professional, and communication skills improve? How many IT certifications did they obtain?
  • To what extent do Digital Literacy Teachers observe improvement in student engagement in Digital Literacy class and social emotional skills from baseline to follow-up for (1) all students in Digital Literacy class and (2) students in Tech Club?
    • To what extent do Digital Literacy Teachers’ observe improvement in student engagement in Digital Literacy Class for (1) all students in Digital Literacy Class and (2) students participating in Tech Club mentoring/tutoring from mid- to end of year?
    • To what extent do Digital Literacy Teachers’ observe changes in students’ social-emotional skills for (1) all students in Digital Literacy Class and (2) students participating in Tech Club mentoring/tutoring from mid- to end of year?
  • To what extent do students participating in Tech Club tutoring/mentoring exhibit increased engagement in Digital Literacy Class and social-emotional skills from pre- to post-program?
  • How many students who participate in Tech Club mentoring or tutoring attend at least 90% of instructional days?

Remediation and Enrichment

Evaluation of 2023 Summer Programs

This is a knowledge-building study of a major district initiative.  The questions we’re asking include:

  • What were summer program enrollment/attendance rates?
  • What were staff, student, and family experiences of summer programs?

2023 Summer Programs in SDP: Offerings, Attendance, Survey Results, and Recommendations

Evaluation of 2024 Summer Programs

This is a knowledge-building study of a major district initiative.  The questions we’re asking include:

  • What were summer program enrollment/attendance rates?
  • What were staff, student, and family experiences of summer programs?

School Climate

Descriptive study of students perspectives on school climate and instructional environment using 2021-22 PSES data and cognitive group interviews

This is a knowledge-building study.  The questions we’re asking include:

  • What are student responses on the key school climate issues covered in the Philly School Experience Survey (PSES), including school safety, relationships with staff and peers, and physical school environments?
  • How do student experiences in school safety and climate differ by school?
  • To what extent do qualitative findings from student focus groups corroborate or offer further insights on findings from quantitative data from the PSES?

Understanding and Measuring Perceptions of School Climate and Safety: Insights from the Philly School Experience Survey

Evaluation of Schoolyard Redesign Project

This evaluation is required by a grant. The questions we’re asking include:

  • What student activities are happening on the playground during school hours and how have these activities varied before and after redesign
  • What school staff activities are happening on the playground during school hours and how have these activities varied before and after redesign?
  • Are there any changes to school climate or achievement after playground installation?

Redesigning Schoolyards at Two Schools: Summary of Interviews and Observations

Descriptive study of student perceptions of Youth Court

This is a knowledge-building study. The questions we’re asking include:

  • What are student perceptions of potential benefits of Youth Court in their school
  • Are there ways in which students think Youth Court could be improved at their school?
  • Is there a relationship between schools implementing a YC and the rate of suspensions?

Student Outcomes and Perspectives on Youth Court in the School District of Philadelphia

System Trends

Summary of Philly School Experience Survey 2022-23 results

This is a summary of survey results. The questions we’re asking include:

  • What were the main system-level trends in results for PSES core topics in 2022-23?
  • What do PSES results tell us about school conditions related to the Accelerate Philly priority areas?

2022-23 Philly School Experience Survey Results

Descriptive study of Philly School Experience Survey 2022-23 support staff survey results

This is a knowledge-building study.  The questions we’re asking include:

  • What are District support staff perceptions of school climate, school leadership, equity and inclusion, and their professional capacity?
  • To what extent do these perceptions vary by school?

Study of Education for Children and Youth Experiencing Homelessness (ECYEH) grant funded activities during 2022-23 school year

This study is required by a grant. The questions we’re asking include:

  • During the 2022-23 school year, did ECYEH approaches result in the identification of school-age students as experiencing homelessness at some point during the year?
  • How many students did ECYEH support in terms of the following services: enrollment assistance, uniforms, school supplies, and transportation assistance?
  • Was ECYEH able to implement the TEEN program for students in addition to professional development opportunities for educators, school personnel, and others?
  • How do students experiencing homelessness compare to other District students on measures of attendance and achievement?

Education of Children and Youth Experiencing Homelessness (Analysis of 2022-23 Data)

Study of graduation rate trends over time

This is a knowledge-building study. The questions we’re asking include:

  • What are the graduation rates for the 2019-20 9th grade cohort for different public school sectors in Philadelphia?
  • What are the longitudinal trends in graduation rates?

2022-23 Four-Year High School Graduation Rates in Philadelphia

Descriptive information from the October 2023 enrollment snapshot

This is a knowledge-building study. The questions we’re asking include:

  • What are the enrollment numbers for different public school sectors in Philadelphia in 2022-23 school year?
  • What are the year-over-year and long-term trends in enrollment?

Philadelphia Public School Enrollment, 2023-24

Descriptive information from 2023-24 School Selection

This is a knowledge-building study of a major district initiative. The question we’re asking is:

  • What were the patterns in student qualifications, application submission, admission offers received, and student acceptances of admission offers?

Fiscal Year 2025 Budget Priorities Survey results

This is a stakeholder engagement effort designed to inform budget priorities for Fiscal Year 2025. The questions we’re asking include:

  • What issues and resources do students, staff, families and other stakeholders view as the highest priorities for the SDP?
  • How do stakeholders perceive the budget development process?
  • What are stakeholders’ priorities for federal title grant investments?

Fiscal Year 2025 Budget Priorities Survey Results

2024-25 and Beyond

Study of the relationship between student performance on 2023-24 Star and Keystone assessments

This is a knowledge-building study. The questions we’re asking include:

  • What is the correlation between Star and Keystone performance?
  • How accurately did Star predict performance on the Keystone?

Study of the relationship between student performance on 2023-24 Star and PSSA assessments

This is a knowledge-building study. The questions we’re asking include:

  • What is the correlation between Star and PSSA performance?
  • How accurately did Star predict performance on the PSSA?

Study of the Relationship between the Philly School Experience Survey and Measures of School Improvement

This is a knowledge-building study. The questions we’re asking include:

  • What is the relationship between the six core PSES topics and measures of student learning, attendance, dropout and staff retention?
  • To what extent are school scores on the six core topics associated with improvement in measures of school improvement over time?