Methodology
How should we measure school quality (with publicly available data)?
Research
Locovote ranks school-quality measures according to insights from “Race and the Mismeasure of School Quality”. That study shows that traditional ratings echo student demographics more than actual school effectiveness. Using randomized school-assignment data, the authors found that schools enrolling higher shares of White students are not inherently better at educating children, they only appear so because of selection bias.
Here are the measures we consider in reverse-order of importance:
-
Test score levels: Traditional proficiency ratings, specifically the share of students scoring "proficient", are heavily influenced by student demographics, not by school quality. They exhibit strong racial correlations and poor predictive accuracy, essentially measuring neighborhood characteristics rather than educational effectiveness.
-
Test score progress: Student Growth Percentiles (SGPs) measure year-over-year improvement, capturing how much schools contribute to learning independently of students' backgrounds. These progress ratings are more accurate than achievement levels and are far less correlated with demographics.
-
Race-balanced progress: Student growth measures in which racial bias is statistically removed through regression adjustment. This approach removes demographic bias while simultaneously improving the predictive accuracy of true school quality.
Do the findings in
The findings in their paper were specific to New York City and Denver, two large urban districts. It is possible that applying their conclusions to comparisons across districts may not be appropriate. I suggest considering both race-balanced progress and test score progress measures. Locovote includes test score levels so users can view these traditional measures, but they are not good indicators of school quality.
If you're interested in reading more about the research, a copy of their paper is available here.
Data
The figure below examines the relationship between a district's racial composition and test-score progress. Each point represents a school district; the x-axis shows the proportion of White students, and the y-axis shows the average Student Growth Percentile. The size of each point reflects the number of students in that school district.
The next chart illustrates how the relationship between test score progress and demographics has evolved over time. Each point is the slope estimate for a given year and subject, error bars denote 95% confidence intervals.
The data are sourced from this dataset. The quotation below provides additional context for interpreting trends in the slope estimates presented in the graph above.
"Student growth percentile (AVG_SGP) was calculated as a median for 2017. In 2018 and onward, it is a mean. In 2021, a baseline SGP method was used to compare growth from 2019 to 2021, following the COVID-19 pandemic. For all other years, a cohort referenced model is used. For more information on SGP calculations, please see the Student Growth page on DESE's website."