I was recently reviewing materials for my course, “Tools for Impact Evaluation,” which I teach at Princeton University’s Woodrow Wilson School of Public and International Affairs. In the first lecture, before I plunge into methodological topics, I discuss the ways evaluation, evidence, and policy connect. In particular, I mention several evidence-building activities the federal government has recently undertaken. I stress that Congress established a Commission on Evidence-Based Policymaking, which Paul Decker and Ron Haskins described in a post on this blog.
We the People
When I read the commission’s enabling legislation, a sentence grabbed my attention: “Without evidence, the federal government is an ineffective fiduciary on behalf of the taxpayer.” This is a fundamental principle enshrined in the first paragraph of the Constitution of the United States:
We the People of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defence, promote the general Welfare, and secure the Blessing of Liberty to ourselves and our Posterity, do ordain to establish this Constitution of the United States of America.
The mandate is specified in Article 1, Section 9, which states that Congress should “prepare a regular Statement and Account of the Receipts and Expenditures of all public Money.”
To me, this means that for the federal government to meet its fiduciary duties, it must demonstrate that the 1,500 different programs it administers, at a projected cost of $4 trillion annually in 2017, are effective. However, according to the U.S. Government Accountability Office, only 37 percent of program managers reported that an evaluation had been completed within the past five years of any program, operation, or project in which they were involved. Another 40 percent reported that they did not know if an evaluation had been completed. Despite the federal government’s best intentions, building the evidence for decision making is still an elusive goal—and one that our new president-elect, Donald Trump, should prioritize in his administration.
The Orszag Memo
Several years ago when I first taught my course, I read what I consider is the most important directive for evidence-based policy of at least the past decade. Peter R. Orszag, at the time the director of the Office of Management and Budget (OMB), noted that impact evaluations “help the Administration and Congress determine how to spend taxpayer dollars effectively and efficiently, by investing taxpayers’ resources.” The memo cannot be clearer that evaluation is an important tool the government should use to meet the Constitutional mandate.
Orszag noted what the Government Accountability Office study later confirmed—federal managers whose programs were evaluated reported that they were hindered by a lack of resources to implement the evaluation findings in their budget decisions. Therefore, Orszag did the obvious—offer an incentive to agencies for high-priority evaluation, which he called the “Voluntary FY [Fiscal Year] 2012 Evaluation Initiative.” The main goals of this initiative were to show how the agencies’ FY 2012 funding priorities were evidence-based or otherwise subject to rigorous evaluation and to assess agencies’ capacity to support evaluation and suggest pathways for strengthening that capacity.
Orszag’s idea gained traction and, since FY 2012, the Executive Office of the President has committed to a broad-based set of activities to better integrate evidence and rigorous evaluation in budget, management, operational, and policy decisions.
Building the Evidence Base
In July 2016, about four months after President Obama had signed the Evidence-Based Policymaking Commission Act of 2016, OMB prepared five white papers intended to provide the commission with background information on topics relevant to its work.
In the second paper—Using Administrative and Survey Data to Build Evidence—OMB describes its portfolio of evidence, consisting of foundational and policy-specific categories. The first category includes aggregate indicators, population descriptions, trends and correlations, and estimated effects of a specific intervention on an outcome but without reference to a particular policy or program. The second category includes implementation evaluations; impact evaluations (using experimental or quasi-experimental methods); and cost, benefit-cost, cost-effectiveness, and regulatory impact analyses.
The paper cites Mathematica’s evaluations numerous times as examples of rigorous contributions to evidence building. Among our reports cited are the National Job Corps Study and Longer-Term Follow-Up Study: Impact and Benefit-Cost Findings Using Survey and Summary Earnings Records Data in the job training field, and Home Visiting Programs: Reviewing Evidence of Effectiveness in the family support field.
The First Class
In teaching public policy students at the Wilson School, I can use my 25 years of experience of working at Mathematica to present students with examples from my own projects, as well as other contributions of the organization. Students love to hear about these real-world examples from an experienced practitioner.
I like to open the group discussion with an anecdote. Several years ago, Anita Summers, former chair of Mathematica’s board of directors, noted that her son, Lawrence Summers, former secretary of the Treasury and director of the National Economic Council, called one day to say that he heard his mother rave about everything Mathematica produced—and he discovered she was not exaggerating. I am proud to be affiliated with an organization that is recognized by a broad spectrum of users of evidence—federal program managers, members of Congress, heads of executive departments, among other—for its important contributions to developing evidence to inform policy discussions and decisions. This is what motivates me in my work at Mathematica and in teaching students who represent the next generation of policymakers, practitioners, and researchers serving the public good. And it’s why, even at eight in the evening, my students could go for hours talking about how evaluation contributes to building the evidence necessary to assess whether “We the People” have an effective government.
