Show Me the Evidence: Obama's Fight for Rigor and Results in Social Policy
319
Show Me the Evidence: Obama's Fight for Rigor and Results in Social Policy
319eBook
Available on Compatible NOOK devices, the free NOOK App and in My Digital Library.
Related collections and offers
Overview
The first comprehensive history of the Obama administration's evidence-based initiatives. From its earliest days, the Obama administration planned and enacted several initiatives to fund social programs based on rigorous evidence of success. Ron Haskins and Greg Margolis tell the story of six—spanning preschool and K-12 education, teen pregnancy, employment and training, health, and community-based programs.
Readers will appreciate the fast-moving descriptions of the politics and policy debates that shaped these federal programs and the analysis of whether they will truly reshape federal social policy and greatly improve its impacts on the nation's social problems.
Based on interviews with 134 individuals (including advocates, officials at the Office of Management and Budget and the Domestic Policy Council, Congressional staff, and officials in the federal agencies administering the initiatives) as well as Congressional and administration documents and news accounts, the authors examine each of the six initiatives in separate chapters. The story of each initiative includes a review of the social problem the initiative addresses; the genesis and enactment of the legislation that authorized the initiative; and the development of the procedures used by the administration to set the evidence standard and evaluation requirements—including the requirements for grant applications and awarding of grants.
Product Details
| ISBN-13: | 9780815725701 |
|---|---|
| Publisher: | Rowman & Littlefield Publishers, Inc. |
| Publication date: | 12/01/2014 |
| Sold by: | Barnes & Noble |
| Format: | eBook |
| Pages: | 319 |
| File size: | 3 MB |
About the Author
Read an Excerpt
Show Me the Evidence
Obama's Fight for Rigor and Results in Social Policy
By Ron Haskins, Greg Margolis
Brookings Institution Press
Copyright © 2015 The Brookings InstitutionAll rights reserved.
ISBN: 978-0-8157-2571-8
CHAPTER 1
RON HASKINS
Introduction: The Obama Strategy for Attacking Social Problems
In the sixth century B.C., a young Jewish man named Daniel was captured by the Babylonian king, Nebuchadnezzar, to serve as an adviser. While being held in captivity, Daniel and several of his companions refused to eat the food or drink the wine offered by the king, referring to the act of eating their captor's food as "defilement." But the official in charge of the king's slaves, who was responsible for their health and well-being, told Daniel that he was afraid that the king would kill him if Daniel began to show the effects of poor nutrition, in contrast to the young men of the court, who regularly ate from the royal table. Daniel then suggested the following experiment: "Please test [us] for ten days, and let us be given some vegetables to eat and water to drink" while the other men continue to eat from the king's table. The official followed Daniel's suggestion. At the end of ten days the appearance of Daniel and his fellow vegetable eaters seemed better and "they were fatter than all the youths who had been eating the king's choice food."
Daniel was intent on making a point about nutrition, but the way that he proved his point—nearly 3,000 years ago—opens whole vistas in human understanding. Two ideas implicit in Daniel's approach are crucial. First, the appeal to evidence, along with theory, is one of the two pillars of science. Why stand around arguing about abstractions when you can collect evidence to answer a question about what causes what? Second, Daniel instructs the chief chamberlain to gather a particular type of evidence: evidence from a planned experiment. If you want to understand whether A causes B, assign people to two groups, administer A to one group, and keep everything else that the groups experience as similar as possible. Then, after some period of time, measure B in both groups. If the two groups differ with respect to B, A must have caused the difference. While his experiment could be improved in many ways, especially by better measurement of outcomes and by assigning people to the two groups at random to ensure their initial comparability, Daniel was nonetheless onto something big.
Discovering the Obama Evidence-Based Initiatives
The Obama administration and the federal bureaucracy are full of Daniels. Sometime during the summer of 2010, around 18 months into the new administration, I became aware of a home visiting initiative that the administration was implementing, formally called the Maternal, Infant, and Early Childhood Home Visiting initiative. Home visiting is a health and educational intervention to help young, usually single, mothers make good decisions in their personal life and adopt parenting practices that help their babies flourish. In undertaking the home visiting initiative, the administration had decided to fund only home visiting programs that had strong evidence of success. In September 2010, Kathy Stack, a senior career official at the Office of Management and Budget (OMB) who also was a friend of mine, called to say that she had heard that I was looking into the Obama initiative on home visiting. When I told her that I was writing a brief paper about the home visiting initiative, she asked if I wanted to have lunch to discuss not only the initiative but also other evidence-based initiatives that the administration was planning. I eagerly accepted. Over lunch, Kathy described the full scope of what the administration was doing and remarked that people should know more about the initiatives because they could be precedent setting. Well, I had an idea about how to let people know more about the Obama evidence-based initiatives—namely, by writing a book about them. I left our lunch determined to raise enough money to hire a research assistant and learn everything that I could about the initiatives. My initial idea was to describe how the initiatives had been conceived; how the administration got them through Congress; how "evidence-based" was defined; how the administration managed to communicate its evidence-based concept to potential program operators around the nation; how the administration ensured that money went to program operators who were using evidence-based programs; and how the administration would ensure that the programs were having the intended impacts. Eventually, the William T. Grant Foundation, which for many years had been focusing on how social science evidence informs public policy, agreed to finance my enterprise, and I was bound for the Promised Land—especially after I hired Greg Margolis, who became the legs—and, in large part, the brains—of my operation to tell the story of the administration's evidence-based ventures.
What does it mean for the Obama administration to create "evidence-based initiatives?" It means that the administration strives to be as certain as possible that federal dollars are spent on social intervention programs that have been proven by rigorous evidence to work. What does it mean to say that the programs "work"? It usually means that someone, following a scientifically rigorous version of Daniel's approach, randomly assigned people to two groups, one of which had the intervention program and one of which did not, and found that the outcome of the group that had the intervention was superior to that of the other group in one or more important ways. For example, poor and low-income mothers who participated in a home visiting intervention were found to have better outcomes in several areas than similar mothers who had not participated, depending on the particular study. Improvements included less smoking and drinking by the mothers during pregnancy, reduced rates of child abuse and harsh parenting, healthier children, lower levels of parent depression, and higher achievement test scores of children.
As we began to learn more about the Obama evidence-based initiatives, a second meaning of "evidence-based" became clear. Besides striving to fund primarily programs that already had been shown to work, the administration would also insist on rigorous ongoing evaluation of nearly every program that it funded. Running what eventually became six evidence-based initiatives, administration officials in the four executive agencies (Department of Labor, Department of Health and Human Services, Department of Education, and the Corporation for National and Community Service) greatly increased the likelihood of good evaluations by requiring entities applying for federal grant funds to give detailed information on how they would evaluate their project and to set aside money in their budget to pay for the evaluation. The agencies also provided projects with technical assistance to improve the quality and implementation of their evaluations.
It gradually dawned on me that the Obama approach to increasing the impact of government social intervention programs by greatly expanding the use of evidence was an idea with very broad applications and great promise for improving social outcomes. I originally came to Washington in 1985 from the University of North Carolina on a one-year fellowship sponsored by the Society for Research in Child Development and the American Association for the Advancement of Science to learn more about how to use social science research to improve the nation's social policy, especially policies that could boost the development of children from poor and minority families. My plan was to stay in Washington for a year or two, learn through observation and participation in the policy process about the uses of social science in the formulation of public policy, and return to the university to write a book about improving policy through more frequent and sophisticated use of social science knowledge. But I was captured by Washington and wound up working as a staffer on the House Ways and Means Committee for 14 years and as a White House staffer for a year. But I never lost my desire to return to scholarship, and at last—getting a little long in the tooth—I joined the Brookings Institution in January 2001 to finish my career by writing about poverty and economic opportunity. I also wanted, to the extent possible, to work with Congress, the executive branch, and congressional and executive branch agencies to use social science to improve policy. I saw then, as I do now, that a productive approach to using social science to improve policy is to conduct policy analysis, classically defined as the application of reason, evidence, and a values framework in choosing between alternative policies. But the entire enterprise hinges on being able to use social science methods to determine whether social programs are producing their intended effects.
Within my first five minutes in Washington, I realized that the influence of research, program evaluation, and policy analysis on policy formulation in Congress, the administrative agencies, and the White House was tenuous at best. When I left Capitol Hill and came to Brookings in 2001, I was asked to give a talk to the Brookings trustees about how research influenced public policy. In a somewhat whimsical vein, I summarized my thinking in a pie chart similar to the one shown in figure 1-1. The pieces of the pie chart represent the factors that I had seen determine the votes of members of Congress in committee and on the floors of the House and Senate. While the exact size of the pieces, which are proportional to their relative influence on policy formulation, varied from issue to issue, it was rare that social science research or policy analysis had much of a direct impact. The lesson here is that the U.S. government is a free-wheeling, wide-open form of democracy, conducted in many cases as a blood sport. Political philosophy, to apply a grandiose term to the simpler shibboleths by which many members of Congress operate (such as "Small government is best," "The IRS is evil," "Welfare destroys the incentive to work," "A rich country should not tolerate child poverty") trumps social science evidence on all but rare occasions. In that context, refined thinking about how research and evidence should influence policy must fight for a place at the table. The purpose of the Obama evidence-based initiatives is to pull up another chair and set a place for evidence.
While still on the Hill, I came to accept and even appreciate this messy world in which evidence would have to fight to play a role. On the other hand, it was impossible not to see that evidence was not always used in ways that enlightened debate. Consider the normal social science activity of conducting a literature review to determine what is known about how to attack a social problem. Such a review often focuses on both what is known about the causes of a social problem (such as child abuse, teen pregnancy, dropping out of school) and the effectiveness of programs designed to attack the causes. After years of reading and writing such reviews, I began to realize that even after completing thorough, fair-minded, and unbiased reviews, two equally impressive scholars might reach different conclusions in large part because of their different interpretations of the evidence. But such disagreements between scholars after careful review of the evidence were milder than and different in kind from the differences in the positions—supposedly based on the weight of evidence—often taken in the political world. In politics, evidence is typically used as a weapon—mangled and used selectively in order to claim that it supports a politician's predetermined political position. That is policy-based evidence, not evidence-based policy. This and similar problems would have to be overcome or bypassed if social science evidence were to play a more constructive role in formulating social policy. Indeed, this is the goal of the Obama administration's commitment to evidence-based policy.
The Guts of the Evidence-Based Initiatives
The more I came to understand what the Daniels in the Obama administration were up to, the more I came to see a clear, appropriate, and greatly expanded role for evidence in the policy process. Here is the essence of what could be expected from a full and vigorous implementation of the Obama approach. First, new funding for social programs would go primarily, but not exclusively, to programs that showed strong evidence of success. Let me quickly note that the requirement for evidence-based policy does not mean that the legislation authorizing spending should contain language specifically intended to ensure that the money is spent primarily on successful programs. In fact, as will be seen, it may be better to keep the legislative language about evidence somewhat loose. Later, the administration in power at the time and career professionals in the federal agencies can include definitions in regulations and funding announcements to ensure that funded programs meet well-defined standards of evidence. Evidence is constantly piling up, but legislation moves at a speed that is ponderous at best. As a result, it seems better to give an important role to administrative officials, who can stay abreast of the evidence. Second, new programs that received funding would be implemented at the state and local level and would be subject to continuous evaluation to make sure that they were having their intended impact; if they were not, the programs would be reformed until they produced the expected results or they would be terminated. Third, over the course of many years, programs that already were being funded by federal agencies would be brought under the evidence-based umbrella. Implementing this vision, federal agencies would become fanatics about continuously evaluating their programs through the use of rigorous designs. They also would insist on substantially revising or terminating programs that failed and then using the money saved to invest in programs that either already had developed or were in the process of developing strong evidence of success.
At some point, the flexibility of any administration to implement federal social programs and the extent to which it can change or terminate programs that are not working are limited by statutory provisions. Many social programs are based on statutory language that specifies who can participate in the program, how long they can participate, and what type of activities can be paid for with federal dollars. Yet in many programs the language is elastic, providing an agency at least some leeway in allocating funds. A federal agency may not be able to end a program that evaluations have shown to be ineffective, but often it may be able to use regulations and grant announcements to change the type of activities that program funds can be used to support. Similarly, often the federal agency can take money away from programs that do not meet quality measures or minimum outcome requirements.
Why would senior officials in the Obama administration invest so much energy and political capital in trying to force the federal government to change the way that it funds social programs? Here is an example of the kind of federal grant making that the administration's demand for evidence-based proposals is intended to eliminate. Consider Title I of the Elementary and Secondary Education Act, passed by Congress in 1965 and signed into law by President Johnson as part of his War on Poverty. The law has three primary goals: improving financial equity across the states; giving states incentives to participate in the federal educational accountability and testing regime; and providing funds for programs that help children, especially poor children, learn. Regarding the last goal, which is the most important and the only one that we take up here, the act authorizes a host of activities by local school systems to equalize the educational opportunities given to advantaged and disadvantaged children. Over the nearly 50 years since the program was authorized, Title I has cost taxpayers around $320 billion—even more if that figure is adjusted for inflation. Yet there is only modest evidence that all that money has actually improved achievement test scores, increased high school graduation rates, increased either college entry or college graduation rates for poor or minority students, or had a meaningful positive impact on any measure of student performance. If all that money had been spent on programs known to improve some aspect of educational performance, Title I could have achieved much more than it has and more poor children could have received a better K–12 education, which may in turn have increased their college enrollment rates and increased the nation's economic mobility while reducing income inequality.
(Continues...)
Excerpted from Show Me the Evidence by Ron Haskins, Greg Margolis. Copyright © 2015 The Brookings Institution. Excerpted by permission of Brookings Institution Press.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
Table of Contents
Foreword Jim Manzi ix
Acknowledgments xiii
1 Introduction: The Obama Strategy for Attacking Social Problems Ron Haskins 1
2 The Maternal, Infant, and Early Childhood Home Visiting Initiative 31
3 The Teen Pregnancy Prevention Initiative 67
4 The Investing in Innovation Initiative 102
5 The Social Innovation Fund Initiative 132
6 The Workforce Innovation Fund Initiative 168
7 The Trade Adjustment Assistance Community College and Career Training Initiative 188
8 So Far, So Good 213
Appendix A Interviewees 241
Appendix B Questionnaire 249
Appendix C Share of Formula and Competitive Grant Funds in Six Evidence-Based Initiatives 253
Appendix D Abbreviations 255
Notes 259
Index 297