Education Policy Research

Reports

• “Survey of Outcomes Measurement in Research on Character Education Programs.” Ann Person, Emily Moiduddin, Megan Hague-Angus, and Lizabeth Malone, December 2009. Through the Partnerships in Character Education Program, the federal government distributes about $25 million annually to state and local education agencies for character education programs. This report documents the constructs measured in studies of 36 randomly selected programs. It also includes a framework for systematically describing and assessing measures of character education outcomes and provides a resource for evaluators to help identify and select measures of outcomes for character education programs. Among student level outcomes, those most often measured were: academic content (measured for 14 programs), prosocial dispositions and interpersonal strengths (each measured for 11 programs), discipline issues and interpersonal competencies (each measured for 13 programs), and substance use and intrapersonal competencies (each measured for 11 programs).
• “Using State Tests in Education Experiments: A Discussion of the Issues.” Henry May, Irma Perez-Johnson, Joshua Haimson, Samina Sattar, and Phil Gleason, November 2009. Securing data on students’ academic achievement is typically one of the most important and costly aspects of conducting education experiments. As state assessment programs have become practically universal and more uniform in terms of grades and subjects tested, the relative appeal of using state tests as a source of study outcome measures has grown. However, the variation in state assessments—in both content and proficiency standards—complicates decisions about whether a particular state test is suitable for research purposes and poses difficulties when planning to combine results across multiple states or grades. This paper aims to help researchers evaluate and make decisions about whether and how to use state test data in education experiments. It outlines the issues that researchers should consider, including how to evaluate the validity and reliability of state tests relative to study purposes; factors influencing the feasibility of collecting state test data; how to analyze state test scores; and whether to combine results based on different tests. It also highlights best practices to help inform ongoing and future experimental studies. Many of the issues discussed are also relevant for nonexperimental studies.
• New Leaders for New Schools, a nonprofit organization committed to training school principals, heads the Effective Practice Incentive Community (EPIC), an initiative that offers financial awards to effective educators. Two new reports present methods for identifying effective EPIC schools and teachers using value-added models:
• "Measuring School Effectiveness in Memphis: Year 2." Liz Potamites, Duncan Chaplin, Eric Isenberg, and Kevin Booker, October 2009. This report presents the methods used to identify effective schools in the Memphis City Schools during the second year of the project.
• "Measuring School and Teacher Effectiveness in the EPIC Charter School Consortium: Year 2." Liz Potamites, Kevin Booker, Duncan Chaplin, and Eric Isenberg. October 2009. This report presents the methods used to identify effective schools and teachers for a consortium of over 140 charter schools in 17 states and the District of Columbia during the second year of the project.
• "Do Typical RCTs of Education Interventions Have Sufficient Statistical Power for Linking Impacts on Teacher Practice and Student Achievement Outcomes?" Peter Z. Schochet, October 2009. For randomized control trials (RCTs) of education interventions, it is often of interest to estimate associations between student and mediating teacher practice outcomes, to examine the extent to which a study’s conceptual model is supported by data, and to identify mediators most associated with student learning. This paper develops statistical power formulas for such exploratory analyses under clustered school-based RCTs using ordinary least squares (OLS) and instrumental variable (IV) estimators, and uses these formulas to conduct a simulated power analysis. The power analysis finds that for currently available mediators, the OLS approach will yield precise estimates of associations between teacher practice measures and student test score gains only if the sample contains about 150 to 200 study schools. The IV approach, which can adjust for potential omitted variable and simultaneity biases, has very little statistical power for mediator analyses. For typical RCT evaluations, these results may have design implications for the scope of the data collection effort for obtaining costly teacher practice mediators.
• “Technical Methods Report: The Estimation of Average Treatment Effects for Clustered RCTs of Education Interventions.” Peter Schochet, August 2009. This paper examines the estimation of two-stage clustered randomized controlled trial designs (RCTs) in education research using the Neyman causal inference framework that underlies experiments. The key distinction between the considered causal models is whether potential treatment and control group outcomes are considered to be fixed for the study population (the finite-population model) or randomly selected from a vaguely defined universe (the super-population model). Appropriate estimators are derived and discussed for each model. Using data from five large-scale clustered RCTs in the education area, the empirical analysis estimates impacts and their standard errors using the considered estimators. For all studies, the estimators yield identical findings concerning statistical significance. However, standard errors sometimes differ, suggesting that policy conclusions from RCTs could be sensitive to the choice of estimator. A key recommendation is that analysts test the sensitivity of their impact findings using different estimation methods and cluster-level weighting schemes.
• “Impacts of Comprehensive Teacher Induction: Results from the Second Year of a Randomized Controlled Study.” Eric Isenberg, Steven Glazerman, Martha Bleeker, Amy Johnson, Julieta Lugo-Gil, Mary Grider, Sarah Dolfin, and Edward Britton, August 2009. Teacher induction programs are designed to support new teachers and reduce teacher turnover. Comprehensive teacher induction, however, goes beyond traditional induction by relying on carefully selected and trained full-time mentors and also includes an intensive curriculum involving instructional support and professional development, opportunities to observe experienced teachers, and assessment tools for teachers that permit ongoing evaluation of practice and constructive feedback. Findings from the second year of Mathematica’s national evaluation show that compared to business-as-usual induction programs, comprehensive induction did not lead to measurable improvements in student test scores, percentage of teachers remaining in their district or in the profession, or in qualifications of the teachers who were retained. This was true whether induction programs were delivered as one-year or two-year programs.
  Executive Summary.
• “ABCTE Teachers in Florida and Their Effect on Student Performance.” Christina Clark Tuttle, Tara Anderson, and Steven Glazerman, August 2009. Do alternatively certified teachers produce student achievement gains on par with other teachers? This study used propensity score matching to identify a comparison group of teachers in Florida to address this question for the American Board for Certification of Teacher Excellence (ABCTE). Using data on 30 teachers over two years, the authors found no differences in gains in reading between students of ABCTE and non-ABCTE teachers. However, students of ABCTE teachers scored lower than their counterparts on the state math test.

Journal Articles

• “How Accountability Pressure on Failing Schools Affects Student Achievement.” Hanley Chiang, Journal of Public Economics, October 2009. Although the threat of sanctions on low-performing schools can raise student test scores in the short run, the extent to which these test score improvements are due to schools' manipulation of the accountability system has remained uncertain. This article evaluates the relative importance of educational reforms and gaming behavior in generating test score gains by threatened schools. The author uses a regression discontinuity design to estimate medium-run effects on student test scores from having attended a threatened elementary school. Threat-induced math improvements from elementary school largely persist at least through the first one to two years of middle school, while evidence for persistence of reading improvements is less consistent. Sanction threats tend to raise school spending on instructional technology, curricular development, and teacher training.
• "Statistical Power for Random Assignment Evaluations of Education Programs." Peter Z. Schochet, Journal of Educational and Behavioral Statistics, March 2008. This article examines theoretical and empirical issues related to the statistical power of impact estimates for experimental evaluations of education programs. The author considers designs where random assignment is conducted at the school, classroom, or student level, and employs a unified analytic framework using statistical methods from the literature. Focusing on standardized test scores of elementary school students, this article discusses appropriate precision standards and, for each design, the required number of schools to achieve those standards using empirical values of intraclass correlations, regression R2 values, and other parameters. Clustering effects vary by design but are typically large. As a result, large school samples are required for education trials, and many evaluations will have sufficient power to detect precise impacts only for relatively large subgroups of sites.
• “After-School Program Effects on Behavior: Results from the 21st Century Community Learning Centers Program National Evaluation.” Susanne James-Burdumy, Mark Dynarski, and John Deke, Economic Inquiry, January 2008. This paper presents evidence on after-school programs’ effects on behavior from the national evaluation of the U.S. Department of Education’s 21st Century Community Learning Centers after-school program. Findings come from both of the study’s components: (1) an elementary-school component based on random assignment of 2,308 students in 12 school districts, and (2) a middle-school component based on a matched comparison design including 4,264 students in 32 districts. Key findings include higher levels of negative behavior for elementary students and some evidence of higher levels of negative behaviors for middle school students.
• "Academic Achievement and School Functioning Among Non-Incarcerated Youth Involved with the Juvenile Justice System." Jonathan D. Brown, Anne W. Riley, Christine M. Walrath, Philip J. Leaf, and Carmen Valdez, Journal of Education for Students Placed at Risk, January-March 2008. This article reports on the education problems of youth involved with the juvenile justice system but not incarcerated. More than half demonstrated deficits in academic functioning, with standard achievement scores as low as five standard deviations below the normative mean. Non-Caucasian youth and those who received special education services or lived in an urban area had lower achievement. These findings suggest that youth involved with the justice system but not incarcerated demonstrate problems in academic achievement similar to incarcerated youth and may benefit from targeted education interventions. 

Issue Briefs

• "Infusing Academics into Career and Technical Education." Joshua Haimson, James R. Stone, III, and Donna Pearson, Trends in Education Research, Issue Brief #3, December 2008. Integrating academic learning into career and technical education (CTE) classes can be challenging for educators and curriculum developers but can be aided by securing detailed feedback from CTE teachers. Drawing on a recent study, this issue brief identifies challenges developers faced in infusing more math into CTE curricula and notes that incorporating academic learning into CTE requires substantial time, effort, and other resources.

Other

• "Good Things Come in Small Packages: Lessons Learned from the Ohio High School Transformation Initiative." Sarah Jones, Monica Martinez, and Cindy Cai. In Saving America's High Schools, edited by Becky A. Smerdon and Kathryn M. Borman, November 2009. The American educational system is in a continuous state of reform, yet outcomes are less than satisfactory. This chapter reports on a small reform initiative in Ohio showing that, while small schools have benefits, overall, the structure needs to be rethought to maximize resources, meet all education requirements set by the state, while simultaneously providing students with diverse options.
• What Works Clearinghouse (WWC)
• Administered by Mathematica for the U.S. Department of Education's Institute of Education Sciences, the WWC produces a variety of reports that assess and summarize education research. WWC reports can help educators make important decisions about what curriculums to implement, what products to purchase, and what methods to use in their classrooms and schools.
  
  See the WWC's latest releases at whatworks.ed.gov and explore available Practice Guides, Intervention Reports, and Quick Reviews or take a guided tour of the site.