Read an Excerpt
The Problem with Education Technology
(Hint: It's Not the Technology)
By Ben Fink, Robin Brown University Press of Colorado
Copyright © 2016 University Press of Colorado
All rights reserved.
ISBN: 978-1-60732-447-8
CHAPTER 1
The Problem with Education Technology
(Hint: It's Not the Technology)
TEACHERS VS. TECHNOLOGY — ROUND ONE, FIGHT!
In today's society, college is ambiguous. We need it to live, but we also need it to love. ... Teaching assistants are paid an excessive amount of money. The average teaching assistant makes six times as much money as college presidents. In addvition, they often receive a plethora of extra benefits such as private jets, vacations in the south seas, a staring roles in motion pictures. Moreover, in the Dickens novel Great Expectation, Pip makes his fortune by being a teaching assistant.
It was our last day at the 2012 Conference of College Composition and Communication (CCCC) in St. Louis, the biggest annual gathering of college writing teachers in the country. We were done with our own presentation, and we could eat only so many ribs in a day ... so we were looking through the conference program to see what else was on offer ... when a full-page listing caught our eye. It was for a "Special Session." Something particularly Big and Important. This Special Session, the program told us, would be a "vigorous but civil debate" on Automated Essay Scoring (AES), the technology that lets computers score writing assignments.
We were intrigued. When an academic conference has to specify that an event will be "civil," you know something's up.
The ballroom was vast and long. There were about two hundred seats facing a long table and a giant PowerPoint screen. At one end of the table sat Paul Deane and Chaitanya Ramineini, researchers from Education Technology Services (ETS): the College Board, the minds behind standardized tests like the SAT, GRE, and AP exams, and — along with their for-profit colleagues Pearson and Vantage — the loudest promoters of AES. They were here today to present their newest innovation: the e-Rater Scoring Engine, a computer program that — according to an impressive set of studies — can score a student essay with the reliability of a human.
At the other end of the table sat Les Perelman, the longtime (now emeritus) Director of Undergraduate Writing at MIT — and the best-publicized crusader against AES. He authored the first paragraph of this section, which he fed, along with another page-and-a-half of similar drivel, into e-Rater. And it got a 6, the highest-possible score. The point seemed clear and obvious: machines can't read. They can't understand. And they can't, and certainly shouldn't, replace humans in educating our children.
(Between them, fittingly, sat Carl Whithaus, director of the University Writing Program at UC Davis, who works to integrate AES into human-centered teaching. We'll get back to him later.)
Les was the perfect foil to Deane and Ramineini. They were impeccably dressed; he wore a suit that didn't fit quite right and looked a little sweaty. Their speech was memorized and newscaster-perfect; he spoke off the cuff and made no effort to hide a general-purpose ethnic accent. Their PowerPoints were sleek, branded, and serious; his was homemade and funny, complete with gifs of rabbit holes and the Twilight Zone.
None of this was by accident. They were each playing their parts to a T: the bloodlessly efficient technocrats versus the righteous, rumpled, lone defender of flesh-and-blood teaching. "Vigorous but civil" be damned — this was a brawl. An agon. A pundit war between good and evil, man and machine. We, the human educators of CCCC, sat and winced as the forces of evil took their best shot — and then rejoiced as our champion tore them limb from limb, down to their last unfounded assumption and logical fallacy.
He was, after all, fighting for our lives. Or at least our livelihoods. Humanistic objections to robo-teaching aside, we were precisely those humans AES stands poised to eradicate. AES, which can grade thousands of student essays in mere seconds, could convince a budget-wary public that small class sizes and individualized instruction are unnecessary luxuries. In a political moment where education funding is under constant attack, it's not hard to imagine administrators and elected officials (even the well-meaning ones) using AES as a rationale to lay us off. So when we see Les up there proving that AES doesn't work, won't work, can't work, of course we cheer loudly.
The problem is, outside the cozy confines of CCCC, he's losing. We're losing. Education technology — computer-based classroom instruction, online K–12 schools, MOOCs (Massive Open Online Courses), as well as AES — is all the rage. It's what's getting all the media attention, foundation funding, and government grants. It's the cutting edge, the thing all forward-thinking principals, superintendents, policymakers, and executives (both public and private) are talking about. Les's dogged defense of old-fashioned teaching might make us cheer, but it's making everyone else yawn.
And maybe most frustrating of all, the science all seems to be on their side. Sure, Les's argument makes intuitive sense — how could you trust a program that gave that pile of s#!t he wrote the highest-possible score? — but then again, how can you argue with scores of reliable, data-driven studies?
This is the problem we'll tackle in this book. We'll explain the technology, how it works, and what it does (and doesn't do). We'll explain where it comes from, and how it's come to take the form that it has, in the classroom and in our culture. We'll also explain why the debate over the technology has taken the shape it has — a shape that stops us from understanding the real problem or doing anything about it. Finally, we'll explain what that real problem is, and how we — teachers, students, citizens — can work to solve it.
Because there is a problem. It's just quite a bit subtler, tougher, and more complicated than the standard "civil debate" would have us believe.
THE FIGHT OVER COMPUTER GRADING (STARTED BEFORE COMPUTER GRADING)
Human versus machine. Good versus evil. Teachers versus technology. Kids versus computers. This is a Big Issue, and not just for writing teachers.
A few weeks after the conference, it made the New York Times. "Facing a Robo-Grader? Just Keep Obfuscating Mellifluously," read the headline. The story, by longtime Times reporter Michael Winerip, starts with a recent study that "concluded that computers are capable of scoring essays on standardized tests as well as human beings do" — at the rate of 48,000 essays per minute.
It gives some space to ETS representatives, including Paul Deane. But most of the story belongs to Les, who demolishes the software with his usual gusto:
The e-Rater's biggest problem, he says, is that it can't identify truth. He tells students not to waste time worrying about whether their facts are accurate, since pretty much any fact will do as long as it is incorporated into a well-structured sentence. "e-Rater doesn't care if you say the War of 1812 started in 1945," he said.
Mr. Perelman found that e-Rater prefers long essays. A 716-word essay he wrote that was padded with more than a dozen nonsensical sentences received a top score of 6; a well-argued, well-written essay of 567 words was scored a 5.
An automated reader can count, he said, so it can set parameters for the number of words in a good sentence and the number of sentences in a good paragraph. [...] e-Rater likes connectors, like "however," which serve as programming proxies for complex thinking. Moreover, "moreover" is good, too.
Gargantuan words are indemnified because e-Rater interprets them as a sign of lexical complexity. "Whenever possible," Mr. Perelman advises, "use a big word. 'Egregious' is better than 'bad.'"
The ETS representatives make some important points in response, which we'll get to later. But they don't — and can't — refute Les's basic claims. Computers can't read. They can't understand truth. They can only grade by counting things: length of essays, length of sentences, and length of words. So, obviously, they shouldn't be used in place of humans. Right?
Not so fast. Take a look at this:
In March, Les Perelman attended a national college writing conference and sat in on a panel. [...] "It appeared to me that regardless of what a student wrote, the longer the essay, the higher the score," Dr. Perelman said. A man on the panel from the College Board disagreed. "He told me I was jumping to conclusions," Dr. Perelman said. "Because M.I.T. is a place where everything is backed by data, I went to my hotel room, counted the words in those essays and put them in an Excel spreadsheet on my laptop. [...]
He was stunned by how complete the correlation was between length and score. "I have never found a quantifiable predictor in 25 years of grading that was anywhere near as strong as this one," he said. "If you just graded them based on length without ever reading them, you'd be right over 90 percent of the time." The shortest essays, typically 100 words, got the lowest grade of one. The longest, about 400 words, got the top grade of six. In between, there was virtually a direct match between length and grade. He was also struck by all the factual errors in even the top essays. [...]
How to prepare for such an essay? "I would advise writing as long as possible," said Dr. Perelman, "and include lots of facts, even if they're made up." This, of course, is not what he teaches his M.I.T. students. "It's exactly what we don't want to teach our kids," he said.
This is another story from the New York Times. Like the first one, it's about a fight between Les and ETS at CCCC. Like the first one, Les is criticizing ETS for scoring essays based on length, rather than truth or quality. And like the first one, it's pretty darn convincing.
Only this story isn't from March 2012. It's from March 2005. And it wasn't about AES at all.
This story was, rather, about the then-brand-new essay portion of the SAT. (Which is now defunct, due in no small part to Les's efforts.) It seems the SAT essay had all the same problems Les now identifies with AES. But computers didn't grade these essays.
Here's where the plot thickens. Yes there's a problem with the technology, but it's not just about the technology — take away the computers, and the problem persists.
To understand why, first we need to understand more about how the computers work, how they're programmed, and what, exactly, they're programmed to do.
HOW THE TECHNOLOGY WORKS
How does a computer grade an essay? If you believe the hype, including some tech companies' marketing materials, AES works pretty much like HAL in 2001: it passes the Turing test. It can talk, think, read, and respond just like a human. It "understands the meaning of written essays."
That's a pretty big claim. And it's false. There is indeed a fruitful and fascinating line of research pointing to this kind of "total" Artificial Intelligence (AI): a theory of human cognition and language that could model the near-impossibly complex workings of the mind, as meanings are made, communicated, and understood. But so far, it's purely academic, with pretty much no practical application. We know a whole lot more about the structures of the brain, the mind, and language than we used to, but we're still a long way from a machine that really "understands meaning."
So what do AES systems do? Professor Patricia Freitag Ericsson explains:
These programs treat essays as pure information that can be mined for some abstracted set of words that, at least to their promoters, equates to meaning. The shifting, dynamic relationships that these words have to each other, to society, and to different readers is invisible to these information-seeking machines. The machines can tell users whether writers have matched the words in an essay with words in a database (or a triangulated database matrix), but they cannot assess whether this mix of words conveys any meaning.
The people who make the AES technology know this full well. When they're writing seemingly-scholarly articles, rather than doing PR, they fully own up to it. Here's Dr. Randy Elliot Bennett, the "Norman O. Frederiksen Chair in Assessment Innovation" at ETS:
It is important to note that, regardless of what vendors may claim, a machine does not read, understand, or grade a student's essay in the same way as we like to believe that a human rater would. It simply predicts the human rater's score.
How does it do this? By using a second, distinct, and far less ambitious form of AI: an expert system. Expert systems are meant to emulate the external behaviors of flesh-and-blood experts. Not to be experts, or even to try, just to act like them in the most surface-level way. Give them the same inputs you'd give a human expert, and they'll produce the same outputs. (The best-known example of an expert system is Watson, the machine that beat Ken Jennings at Jeopardy, and now does medical diagnosis.)
How do you build an expert system? It's pretty simple, actually:
1. Define a discrete phenomenon, identifiable by human experts given certain data. Like a disease, which doctors can identify based on a bundle of symptoms; or "good writing," which teachers can identify based on a bundle of traits.
2. Assemble a body of experts. Load up the program with terabytes of examples from lots of real human experts — reams and reams of charts, test data, notes, etc.
3. Describe and segment the behaviors involved in "expertise." Basically Taylorism, twenty-first century-style: break down the work human experts do into as many discrete parts as possible, and teach the computer program to imitate each of those parts.
4. Define the "markers" that emerge as significant identifiers. Those patterns of symptoms, test results, traits, etc. that lead human experts to reach one conclusion versus another.
5. Rank and relate these markers to each other. In order to emphasize the ones that human experts rely on the most, the ones that "really matter."
6. Test and repeat. And intelligent systems, like Pandora, will "learn" from experience.
That's it. Expert systems are not independently smart. They can't replace human experts, or even "think" like them. And they're not trying to. All they do is match input data with output conclusions, based on all the past examples programmed into them — they match the evidence the human experts saw with the conclusions they reached based on that evidence.
THE TECHNOLOGY'S "EXPERTISE" (IS NO MORE LIMITED THAN THAT OF ITS HUMAN COUNTERPARTS)
So: AES doesn't "read" essays, let alone "understand" them. It just uses its algorithms to predict the behaviors of certain human "experts."
And these "experts" are, frankly, rather predictable. We're not talking about English teachers pouring lovingly over the work of their twenty students. We're talking about the kind of human "expertise" that ETS and its many, many fellow testing companies have always relied on: standardized-test raters, sitting in a room, reading hundreds of formulaic essays at breakneck speed and assigning points based on the presence of a few formally-defined "primary traits." The same essay scorers Les criticized, rightly, in the Times story from 2005.
If you are under the age of sixty and grew up in the United States, you've probably taken some kind of standardized essay test. You may (but probably don't) remember the prompt. It was vaguely moral and social issue-focused — like what Emerson might have encountered at Harvard back in the day. Here's a recent example:
Identifying with a group makes people feel secure with and trust one another because of what they have in common. They might share the same interests, language, beliefs, ethnicity, or cultural background. However, by limiting their identities to a specific group, people may miss important opportunities to connect with and understand others.
Assignment: Is a strong group identity a good thing or a bad thing? Plan and write an essay in which you develop your point of view on this issue. Support your position with reasoning and examples taken from your reading, studies, experience, or observations.
We'll bracket some very important questions here. (Is this prompt fair to students of various ethnic/racial/gender/class backgrounds? And does anyone — test-makers, test-takers, test-scorers — actually care about the prompt, or the answers being offered?)
The question we need to ask is: once thousands and thousands of students have tried to answer this prompt in twenty-five-minute timed writing tests, and thousands of teachers have gathered for a weekend in a hotel to score them all (working to supplement their meager salaries with summer work), how do they do it?
Quickly. Very quickly. In ETS scoring sessions, the model all other companies follow, the readers work like maniacs, reading against the clock. No more than two to three minutes per essay. (They also take periodic breaks for "re-norming" sessions, where they compare each other's ratings and work toward the "inter-rater reliability" that ETS prides itself on — the same consistency of input-to-output that the AES software strives for.)
Within those two to three minutes, what are they looking for? The basic ETS scoring rubric has remained the same for decades. It has five categories, and raters assign a score of 0–6 for each. Here they are, slightly paraphrased:
1. Point of view, evidence, and support.
2. Organization and coherence — as marked by transition and linking words.
3. Vocabulary.
4. Sentence complexity.
5. Grammar, usage and mechanics.
What's most striking is what's missing. Nothing about getting a complex idea into words. Or presenting big, bold claims in convincing ways. Or crafting language, stories, and arguments that can move, delight, or agitate an audience. Or any other things that (we think you'll agree) actually make for good writing.
(Continues...)
Excerpted from The Problem with Education Technology by Ben Fink, Robin Brown. Copyright © 2016 University Press of Colorado. Excerpted by permission of University Press of Colorado.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.
