The volume of research on charter schools has grown enormously over the past decade. Even so, the sum of our knowledge is a lot less than some of us in the research field have suggested when we talk with journalists and policymakers. We researchers have a lot more questions about charter schools to answer, especially if we expect our work to inform policy and practice. As I describe here, the uncertainties in existing knowledge fall into three categories: (1) questions about methodologies used to measure charter school impacts, (2) important student outcomes that have received little attention, and (3) the challenge of identifying school practices and policy levers that promote better outcomes.
Methodology: Strengths and limitations of lottery-based experimental studies
Charter school research begins with one big advantage over research on most other education interventions: We can often take advantage of natural experiments to measure impacts. To ensure fairness among applying students, oversubscribed charter schools typically use random lotteries to determine admission offers (famously depicted in the documentary Waiting for Superman). As researchers, we love these lotteries because they give us ideal comparison groups: students who were just like those offered admission in every respect except the luck of the draw. The admissions lottery creates a randomized controlled trial, which, when analyzed appropriately, can produce the strongest possible conclusions about the charter school’s impact on its students.
Various researchers (myself included) have exploited charter school admissions lotteries to measure their effects on test scores and, occasionally, on longer-term outcomes. I expect that admissions lottery data will continue to produce valuable knowledge for years to come as we find ways to connect lottery information with a wider variety of educational outcomes. Even so, there are limits to what we can learn from charter lotteries, because most charter schools are not sufficiently oversubscribed to need admissions lotteries to allocate seats, and they therefore cannot be included in lottery-based studies. One national study by my colleagues at Mathematica found that less than 15 percent of charter middle schools could be included in a lottery-based analysis. Our subsequent national study of charter school management organizations found that adequate data for a rigorous experimental analysis was available for only 12 of 161 schools.
Because lottery studies can include only a handful of charter schools, the question is, are those handful representative of charter schools generally? Don’t bet on it. Schools that need lotteries because they have a surplus of applicants might well be better than those without waiting lists. If so, it would be a big mistake to generalize broadly from lottery-based studies.
Methodology: How much can we learn from CREDO’s large-scale studies?
As a complement to lottery-based studies, we need studies using different methods that can be applied to a lot more charter schools, giving us information about performance across the sector. Stanford University’s CREDO has conducted large-scale, statewide (and nationwide) studies seeking to meet this need. CREDO’s studies do not rely on admissions lotteries, instead identifying matched-comparison groups of students enrolled in conventional public schools who have similar characteristics to charter students. Although CREDO’s approach is not as causally rigorous as the lottery-based approach, it can be applied to far larger numbers of charter schools and therefore could be usefully complementary to lottery-based studies.
But policymakers and journalists would do well to interpret CREDO’s findings cautiously. CREDO’s particular approach to using a matched comparison group is unorthodox. The standard practice in matched-comparison studies involves matching students before they begin the treatment and then following the treatment group and the matched-comparison group over time. In contrast, for many charter students, CREDO identifies comparison students and begins comparing achievement trends years after the charter students enrolled in a charter school. For example, CREDO typically assesses the performance of a charter elementary school by comparing its students’ achievement growth in 4th and 5th grade with the achievement growth of a comparison group that was matched at the end of 3rd grade, which was the first time they took state tests. But many 3rd graders in charter schools would have been enrolled in the same charter schools since kindergarten. Did they have the same achievement trajectories in kindergarten to 3rd grade as in 4th and 5th grades? No one knows. Indeed, the students identified for the matched-comparison group at the end of 3rd grade might be a completely different group of students than those who would have been matched at the beginning of kindergarten.
The enormous number of students included in CREDO’s studies gives it the ability to detect small effects, but that carries hazards of its own. Having a very large sample is no help if the research design incorporates a bias that systematically overestimates or underestimates the true impact. A wrong result that is very precisely measured is just as wrong as one that is imprecisely measured. And the precision can be misleading, because an analysis that includes very large numbers is more likely to be “statistically significant”—even if it is wrong.
In short, CREDO’s unconventional approach might lead to overestimating or underestimating the true average effect of charter schools—we don’t know. CREDO should systematically compare its estimated impacts for individual schools with the impacts found for the same schools in lottery-based studies. My colleagues and I have done exactly that in our own studies of charter schools, and we have found that a matching approach can successfully replicate lottery-based results (if matching occurs before charter school enrollment). CREDO could conduct a similar test of its own methods, perhaps in collaboration with researchers who have worked with lotteries. If CREDO’s results successfully replicate lottery-based results in the schools that use lotteries, we would have more reason for confidence in them.
Outcomes beyond middle school test scores
Although CREDO could test the validity of its methods (and refine them if necessary), it is constrained—as are other researchers—by the limited coverage of existing state testing regimes. Most states test students only in grades 3 to 8 and one high school grade. As a consequence, it is difficult to measure the test-score effects of charter high schools and even harder (with existing data) to measure the effects of the first four grades of charter elementary schools. As a field, almost all of our knowledge about the test-score effects of charter schools comes from grades 4 to 8.
More broadly, the field has little information about the effects of charter schools on student outcomes other than test scores, particularly long-term outcomes. Studies of effects on students beyond high school graduation are increasing, but there still aren’t many. A lottery-based study of Boston charter schools suggests promise for their impacts on college success, but that might be because Boston has unusually good charter schools. The two studies of college success and adult earnings that have the widest scope come to different conclusions, with large positive effects reported in Florida and negative or no effects in Texas. This could be because Florida charter schools are better than Texas charter schools, but it is also possible that the difference in findings is because of methodological differences: one of the studies (or both!) might be wrong. We need more lottery-based studies in more locations (and more validation tests of nonexperimental methods) to make sense of these results. This might be particularly important if, as others have argued, the long-term impacts of charter schools are not predicted very well by their short-term impacts on test scores.
And of course, there is more to schooling than preparing students for college and career. Public schools were originally established to prepare students for democratic citizenship. That purpose is arguably more important now than ever given the current state of American political discourse. And it is of particular interest for charter schools, which are operated autonomously and outside the control of elected officials. There is almost no evidence on the effects of public schools on their students’ civic preparation. Earlier this year, Mathematica released findings from the first rigorous, lottery-based study of the impact of charter schools on the subsequent voter registration and election participation of their students. We found that Democracy Prep, a charter-school management organization based in New York City, produces large positive impacts on eventual registration and voting rates. This kind of evidence is needed for a much wider range of charter schools.
Outcomes: How do charter schools affect students in conventional public schools?
Neither lottery-based studies nor CREDO-style studies provide any information about possible impacts of charter schools on students who remain in conventional public schools. The indirect effects of charter schools on non-choosing students could be positive, if competition promotes improved performance across all schools, or negative, if resources and motivated families are drained from conventional public schools. Indirect effects might be of critical importance because the majority of students are enrolled in conventional public schools in nearly every community across the country. Not surprisingly, indirect effects are central to the policy debate. In Massachusetts in 2016, the state’s voters were asked to decide whether a cap on charter school growth should be lifted. Boston’s charter schools were widely recognized as having large positive effects on their own students, but voters nonetheless rejected charter expansion, fearful of possible negative effects on district schools.
To be sure, researchers have made valiant attempts to measure effects on students in district schools, which is not easy to do. To date, most of the existing studies have found that that the effect of charter schools on students in nearby district schools is either neutral or slightly positive. But the issue is hardly settled, and it seems likely that these indirect effects would depend in part on specific elements of state policy and funding.
What makes a successful charter school or a successful charter policy?
This brings us to a final challenge that is directly related to the value of research for policy and practice. As a field, we know little about what makes a successful charter school and even less about what makes a successful charter policy. And that’s the kind of information that would be really useful to policymakers and educators. Indeed, learning from successful new models was one of the original purposes of charter school laws.
Researchers are trying to understand what distinguishes successful charter schools from unsuccessful ones. They’ve been able to see, for example, that some practices of “no excuses” charters—which set high expectations for academics and behavior—seem to produce positive test-score effects. Studies of charter school management organizations and of New Orleans charter schools have also produced suggestive information on characteristics of effective charters. Charter schools are, at least in principle, good vehicles for learning from innovation, because they were set up to promote innovation and permit widely varying approaches.
Even so, the examination of characteristics of schools producing large test-score impacts may lead educators astray if the schools that succeed in raising test scores are not the same ones that succeed in improving long-term outcomes. So we need more studies of long-term effects not only to get a better sense of charters’ average impact but also to identify truly successful charters from which to learn.
Identifying characteristics of effective charter policies is even more difficult than identifying characteristics of successful charter schools, because doing so requires a way to compare the impacts of different policies, presumably across different charter school authorizers or different states. Doing that well requires examination of the multiple student outcomes discussed above in many different states. We researchers have a lot to do on that front.
Fortunately, the field is making progress on many of these questions. But for now, policymakers and journalists should recognize that the research on the effectiveness of charter schools is far from definitive. Anyone who tells them otherwise should be viewed with skepticism.
