The Declassification Engine: What History Reveals About America's Top Secrets

by Matthew Connelly Narrated by Chris Henry Coffey

Overview

SHORTLISTED FOR THE CUNDILL HISTORY PRIZE ¿ Every day, thousands of new secrets are created by the United States government. What is all this secrecy really for? And whom does it benefit?

“A brilliant, deeply unsettling look at the history and inner workings of `the dark state'.... At a time when federal agencies are increasingly classifying or destroying documents with historical significance, this book could not be more important.” -Eric Schlosser, New York Times best-selling author of Command and Control

Before World War II, transparent government was a proud tradition in the United States. In all but the most serious of circumstances, classification, covert operations, and spying were considered deeply un-American. But after the war, the power to decide what could be kept secret proved too tempting to give up. Since then, we have radically departed from that open tradition, allowing intelligence agencies, black sites, and classified laboratories to grow unchecked. Officials insist that only secrecy can keep us safe, but its true costs have gone unacknowledged for too long.

Using the latest techniques in data science, historian Matthew Connelly analyzes a vast trove of state secrets to unearth not only what the government really did not want us to know but also why they didn't want us to know it. Culling this research and carefully examining a series of pivotal moments in recent history, from Pearl Harbor to drone warfare, Connelly sheds light on the drivers of state secrecy- especially incompetence and criminality-and how rampant overclassification makes it impossible to protect truly vital information.

What results is an astonishing study of power: of the greed it enables, of the negligence it protects, and of what we lose as citizens when our leaders cannot be held to account. A crucial examination of the self-defeating nature of secrecy and the dire state of our nation's archives, The Declassification Engine is a powerful reminder of the importance of preserving the past so that we may secure our future.

Editorial Reviews

MARCH 2023 - AudioFile

Narrator Chris Henry Coffey brings conviction and sense of purpose to a history of government document keeping that could not be more timely. The billions spent to store and protect the unmanageable accumulation of classified documents have not prevented repeated and damaging leaks. Just as damaging has been the loss of government openness and accountability, virtues highly prized in American government before WWII. The arguments in this work can be a bit repetitive, but many of the revelations are persuasive, and Coffey maintains a brisk pace and a clear purpose. His frank, forceful delivery underscores a narrative that is exceedingly frank and critical of a flawed system badly badly in need of reform. D.A.W. © AudioFile 2023, Portland, Maine

Publishers Weekly

01/09/2023

Columbia University historian Connelly (Fatal Misconception) forcefully critiques the “exponential growth in government secrecy” since WWII. Drawing on his work at the History Lab, which uses advanced data mining techniques to “sift and sort through” millions of declassified documents for insights into “what the government did not want us to know, and why they did not want us to know it,” Connelly argues that the “relentless” and “massive” accumulation of secret information has “served the interests of people who wanted to avoid democratic accountability.” Examining declassified documents and metadata related to nuclear weapons, cryptography, UFO sightings, battle plans, the 1954 Guatemala coup (long believed to have been coordinated by the Eisenhower administration at the behest of the United Fruit Company), and more, Connelly contends that the rise of state secrets has undermined government efficiency, buttressed the military-industrial complex, and fostered conspiracy thinking. He also contends that the more information is classified, the harder it is to track and protect, making it vulnerable to exploitation, and highlights arbitrary and ineffective policies, including the classification of material after it’s already entered the public domain. Though the data analysis and history lessons can be dense, Connelly enlivens the narrative with sharp character sketches and acerbic wit. It’s an impassioned and well-informed wake-up call. (Feb.)

From the Publisher

Shortlisted for the Cundill History Prize • A Kirkus Best Nonfiction Book of the Year

“It may be the most presciently timed book ever written. . . . It's the love story between America and its secrets.” —Jon Stewart

“Connelly has defined an existential crisis: the suppression of American history. . . . The Declassification Engine makes the case that the culture of secrecy diminishes democracy. And it has now become a culture of destruction as well.” —Tim Weiner, The New York Times Book Review

“Fascinating and urgent. . . . If you believe in the founding principles of the American form of government, then the stakes could scarcely be higher.” —Patrick Radden Keefe, Foreign Affairs

“Harrowing. . . . Connelly’s book unearths disturbing tales. . . . Readers will doubtless look to The Declassification Engine to make sense of the classified files that are now in the news. Yet to insist on the timeliness of Connelly’s research may be to miss its most powerful lesson. There is a much sadder story detailed in the pages of The Declassification Engine—a story about the existential threat that secrecy poses to civic knowledge.” —The Washington Post

“Connelly has written a gripping and sobering account of the exponential increase in government secrets. He persuasively argues that the United States needs a new strategy to handle classified material, demonstrating that both our national security and the health of our democracy are at stake.” —The Christian Science Monitor

“A brilliant, deeply unsettling look at the history and inner workings of ‘the dark state.’ The number of things that truly must be kept secret is small.  The vast amount of information classified by the government is simply a means of wielding enormous power without real oversight. Again and again, Connelly reveals, secrecy has been used to hide mistakes, avoid embarrassment, cover up incompetence, and mislead the public. At a time when federal agencies are increasingly classifying or destroying documents with historical significance, this book could not be more important. An inscription at the entrance to the National Archives says it best:  ‘Eternal vigilance is the price of liberty.’ ” —Eric Schlosser, New York Times best-selling author of Command and Control

“In The Declassification Engine, Matthew Connelly provides an incisive, unexpected account of the history and practice of official secrecy, offering a glimpse into a world that truly exists in the shadows. By showing the corrosive effects of state secrecy, he successfully makes the case for a different attitude to public information.” —Anne Applebaum, New York Times best-selling author of Twilight of Democracy

“The Declassification Engine is an outstanding expose of the secrecy-industrial complex that is suffocating our democracy. Matthew Connelly describes in vivid detail how the dark state became rooted in our national-security institutions and provides common-sense prescriptions for restoring transparency.” —Craig Whitlock, New York Times best-selling author of The Afghanistan Papers

“A profoundly important work of scholarship, one that addresses core questions about American democracy and the challenges to the nation’s venerable tradition of open government. Connelly’s findings are deeply troubling but also hopeful, showing us how data science can be used to help us better understand the past and thereby point the path to a more enlightened future.” —Fredrik Logevall, Pulitzer Prize-winning author of Embers of War

“Matthew Connelly has played three essential roles in the struggle against government secrecy: advocate, archive-maker, and historian. In The Declassification Engine, he combines all three into an unforgettable account, one that is full of fresh and startling revelations that demonstrate how much of our own history has been kept hidden from us.” —Nicholas Lemann, author of Transaction Man: The Rise of the Deal and the Decline of the American Dream

“This is an absorbing account of the evolution of government secrecy, and an insightful exploration of the relationship between transparency, accountability, and self-government. At a moment when democratic renewal seems absolutely urgent, Connelly’s fascinating study could hardly be more relevant.” —Jameel Jaffer, director of the Knight First Amendment Institute at Columbia University, and former director of the ACLU’s Center for Democracy

“An impassioned indictment of America’s culture of official secrecy. . . . Compelling.” —Washington Independent Review of Books

“The U.S. government is hopelessly awash in secret information, and this gripping history describes how we got that way and lays out the dismal consequences. . . . [Connelly] delivers a wild, page-turning ride packed with intelligence mistakes, embarrassing decisions, expensive failed weapons programs, and bizarre research that has ranged from the silly to the murderous. . . . Yet more evidence, brilliantly delivered, of the extent of the U.S. government’s dysfunction.” —Kirkus Reviews (starred)
 
“What [Connelly] discovered was unnerving: a highly fallible, exorbitantly expensive (over $18 billion annually, by Connelly's estimate), virtually uncontrollable [classification] system that ultimately renders its administrators unaccountable to the American taxpayers funding it. . . . One hopes this book will generate serious discussion of the issue.” —Booklist

Library Journal

12/01/2022

Connelly (global history, Columbia Univ.; Fatal Misconception) advocates for the necessity of a declassification engine to tame the U.S. government's vast amount of secret, classified information. Knowing that this is a tall order, the author meticulously makes his case, while also outlining the history of classified information and deftly illustrating the deep symbiosis between capitalism and national security strategy. Connelly states this emerged during World War II, with Pearl Harbor, the Manhattan Project, and the inception of the Cold War acting as catalysts. Reasons for keeping parts of the public record classified include protecting sensitive information about valuable allies and hiding governmental incompetence. The mutual enmity between some civilian leaders—including presidents—and the military brass directly led to the Vietnam quagmire. The global war on terror, with its nebulous focus on national security, gave the government broad, unprecedented powers to surveil citizens. The information age has added to the glut of data captured and parsed by agencies such as the National Security Agency. Connelly also considers the banality of secrecy, making clear that much of the government's classified information is mundane and unproductive. VERDICT Perfect for readers intrigued by the intersection of politics and history.—Barrie Olmstead

MARCH 2023 - AudioFile

Narrator Chris Henry Coffey brings conviction and sense of purpose to a history of government document keeping that could not be more timely. The billions spent to store and protect the unmanageable accumulation of classified documents have not prevented repeated and damaging leaks. Just as damaging has been the loss of government openness and accountability, virtues highly prized in American government before WWII. The arguments in this work can be a bit repetitive, but many of the revelations are persuasive, and Coffey maintains a brisk pace and a clear purpose. His frank, forceful delivery underscores a narrative that is exceedingly frank and critical of a flawed system badly badly in need of reform. D.A.W. © AudioFile 2023, Portland, Maine

Kirkus Reviews

★ 2022-10-21
The U.S. government is hopelessly awash in secret information, and this gripping history describes how we got that way and lays out the dismal consequences.

Connelly, a professor of international history at Columbia, writes that more than 28 million cubic feet of secret files rest in archives across the country, with far more in digital server farms and black sites. Nonetheless, government secrets are not secure. “Washington has been shattered by security breaches and inundated with leaks,” writes the author. Global hackers often access classified files, and dissenters (Julian Assange, Edward Snowden, Chelsea Manning, et al.) regularly extract material. Readers may be surprised when Connelly points out that the first 150 years of American history were essentially secret-free. Even diplomats often avoided encoding their communication. A new era began in 1931 with the groundbreaking for a national archive, and Franklin Roosevelt appointed the first archivist three years later. At this point, the “dark state” began its epic growth, which Connelly recounts in 10 unsettling chapters and the traditional yet still dispiriting how-to-fix-it conclusion. The author delivers a wild, page-turning ride packed with intelligence mistakes, embarrassing decisions, expensive failed weapons programs, and bizarre research that has ranged from the silly to the murderous. A large percentage of classified information, including the famous WikiLeaks revelations, isn’t secret but available in old newspapers. Everyone agrees that democracy requires transparent government. Congress has passed many laws restricting unnecessary classification and requiring declassification after a long period, but they are often dead letters. Officials occasionally required to review records for “automatic” declassification almost always keep them secret. Plus, the bloated archives are so underfunded that staff members have insufficient technical capacity to recover historical records. Destroying them en masse is cheaper, and this is being done. Interestingly, Connelly points out that historians are more likely to study World War II and the early Cold War because 1970s and later material is largely locked away.

Yet more evidence, brilliantly delivered, of the extent of the U.S. government’s dysfunction.

Product Details

BN ID:	2940175432344
Publisher:	Penguin Random House
Publication date:	02/14/2023
Edition description:	Unabridged

Read an Excerpt

PREFACE: Should This Book Be Legal?
 
There I was, sitting at a massive conference table inside a multibillion-dollar foundation, staring at the wood-paneled walls. I was facing a battery of high-powered attorneys, including the former general counsel to the National Security Agency, and another who had been chief of the Major Crimes Unit at the U.S. Attorney’s Office in the Southern District of New York. The foundation was paying each of them about a thousand dollars an hour to determine whether I could be prosecuted under the Espionage Act.
 
I am a history professor, and my only offense had been to apply for a research grant. I proposed to team up with data scientists at Columbia University to investigate the exponential growth in government secrecy. Earlier that year, in 2013, officials reported that they had classified information more than ninetyfive million times over the preceding twelve months, or three times every second. Every time one of these officials decided that some transcript, or e-mail, or PowerPoint presentation was “confidential,” “secret,” or “top secret,” it became subject to elaborate protocols to ensure safe handling. No one without a security clearance would see these records until, decades from now, other government officials decided disclosure no longer endangered national security. The cost of keeping all these secrets was growing year by year, covering everything from retinal scanners to barbed-wire fencing to personnel training programs, and already totaled well over eleven billion dollars. But so, too, were the number and size of data breaches and leaks. At the same time, archivists were overwhelmed by the challenge of managing just the first generation of classified electronic records, dating to the 1970s. Charged with identifying and preserving the subset of public records with enduring historical significance but with no increase in staff or any new technology, they were recommending the deletion of hundreds of thousands of State Department cables, memoranda, and reports, sight unseen. The costs in terms of democratic accountability were incalculable and included the loss of public confidence in political institutions, the proliferation of conspiracy theories, and the increasing difficulty historians would have in reconstructing what our leaders do under the cloak of secrecy.
 
We wanted to assemble a database of declassified documents and use algorithms to reveal patterns and anomalies in the way bureaucrats decide what information must be kept secret and what information can be released. To what extent were these decisions balanced and rule-based, as official spokesmen have long claimed? Were they consistent with federal laws and executive orders requiring the preservation of public records, and prompt disclosure when possible? Were the exceptions so numerous as to prove the existence of unwritten rules that really served the interests of a “deep state”? Or was the whole system so dysfunctional as to be random and inexplicable, as other critics insist?
 
We were trying to determine whether we could reverse engineer these processes, and develop technology that could help identify truly sensitive information. If we assembled millions of documents in databases, and harnessed the power of high-performance computing clusters, it might be possible to train algorithms to look for sensitive records requiring the closest scrutiny and accelerate the release of everything else. The promise was to make the crucial but dysfunctional declassification process more equitable and far more efficient. We had begun to call it a “declassification engine,” and if someone did not start building and testing prototypes, the exponential increase in government secrets—more and more of them consisting of data rather than paper documents—might make it impossible for public officials to meet their own legal responsibilities to maximize transparency. Even if we failed to get the government to adopt this kind of technology, testing these tools and techniques would reveal gaps and distortions in the public record, whether from official secrecy or archival destruction.
 
The lawyers in front of me started to discuss the worst-case scenarios, and the officers of the foundation grew visibly uncomfortable. What if my team was able to reveal the identity of covert operatives? What if we uncovered information that would help someone build a nuclear weapon? If the foundation gave us the money, their lawyers warned that the foundation staff might be prosecuted for aiding and abetting a criminal conspiracy. Why, the most senior program officer asked, should they help us build “a tool that is purpose-built to break the law”? The only one who did not seem nervous was the former ACLU lawyer whom Columbia had hired to represent us. He had argued cases before the Supreme Court. He had defended people who published schematics of nuclear weapons—and won. He had shown how any successful prosecution required proving that someone had possession of actual classified information. How could the government go after scholars doing research on declassified documents?
 
The ex–government lawyers pointed out that we were not just academics making educated guesses about state secrets—not when we were using high-performance computers and sophisticated algorithms. True, no journalist, no historian, can absorb hundreds of thousands of documents, analyze all of the words in them, instantly recall every one, and rank each according to one or multiple criteria. But scientists and engineers can turn millions of documents into billions of data points and use machine learning—or teaching a computer to teach itself—to detect patterns and make predictions. We agree with these predictions every time we watch a movie Netflix recommends, or buy a book that Amazon suggests. If we threw enough data at the problem of parsing redacted documents—the ones in which government officials have covered up the parts they do not want us to see— couldn’t these techniques “recommend” the words most likely to be hiding behind the black boxes, which presumably were hidden for good reason?

Videos