“The 100 Worst Ed-Tech Debacles of the Decade”

This is a list from Audrey Watters (link from Palko). 100! Wow—that’s a long list. But it is for a whole decade.

I doubt this’ll make it on to Bill Gates’s must-reads of the year, but I liked it.

Just to give you a sense, I’ll share the first and last items on Watters’s list:

100. The Horizon Report

In 2017, just a week before Christmas, the New Media Consortium abruptly announced its immediate closure “because of apparent errors and omissions by its former Controller and Chief Financial Officer.” The organization, which was founded in 1994, was best known for its annual Horizon Report, its list of predictions about the near-future of education technology. (Predictions that were consistently wrong.) But as the ed-tech sector is never willing to let a bad idea die, the report will live on. EDUCAUSE purchased many of NMC’s assets, and it says it will continue to publish the higher ed version of the Horizon Report. The Consortium for School Networking, CoSN, has taken up the mantle for the K-12 version.

1. Anti-School Shooter Software

The most awful education technology development of the decade wasn’t bankrolled by billionaire philanthropists. Rather it emerged from a much sicker impulse: capitalism’s comfort with making money off of tragedy. It emerged from this country’s inability to address gun violence in any meaningful way. And that is despite a decade that saw a steady rise in the number of school shootings. “10 years. 180 school shootings. 356 victims,” CNN reported this summer. That is despite, at the beginning of the decade, a shooting that left 20 second graders dead. instead of gun reform, we got anti-school shooter software, a culmination sadly of so many of the trends on this list: surveillance, personalization, data mining, profiling, Internet radicalization, predictive analytics.

For a while, many ed-tech evangelists would bristle when I tried to insist that school security systems and anti-school shooting software were ed-tech. But in the last year or so, it’s getting harder to deny that’s the case. Perhaps because there’s clearly a lot of money to be made in selling schools these products and services: shooting simulation software, facial recognition technology, metal detectors, cameras, social media surveillance software, panic buttons, clear backpacks, bulletproof backpacks, bulletproof doors, emergency lockdown notification apps, insurance policies, bleeding control training programs, armed guards, and of course armed teachers.

“Does It Make More Sense to Invest in School Security or SEL?” Edsurge asked in 2018. Those are the choices education technology now has for us apparently: surveillance or surveillance.

26 thoughts on ““The 100 Worst Ed-Tech Debacles of the Decade”

  1. I didn’t read all 100 – but it strikes me as a bit harsh in the sense that at least (many of) these people were trying to do something positive.

    To me – as a non education expert – it seems like improving education, especially for disadvantaged kids, is just a REALLY HARD problem so most things are bound to fail. This is true in my area of research as well. I’m as snarky as the next guy… but it seems to me that mocking failure just makes it harder for people to be honest about their work, and move on when things haven’t turned out as planned.

    It’s easier to be a critic than a carpenter. Perhaps the author could do us all a favor and follow up with a list of their top 100 amazingly succesful contributions to improving education so that we can work towards implementing these revolutionary (or even incremental) advances into our struggling schools.

    Don’t get me wrong – the list points out some clearly hairbrained, underhanded, and possible downright rotten schemes. But overall, it also looks like many efforts with reasonable intentions that simply didn’t work.

    • I agree with your general premise, but I think you’re missing the hype, which is what it really takes to make this list, not just failure. I think where she’s coming from is that there are two different types of failures: scams and well-intentioned busts, but that both require a certain salesmanship to raise money which is not characterized by an honest assessment of the possibility of failure. Some of that, of course, comes from true believers, but I think her point is that a lot of it is just hype. What she’s asking for, I think, is honesty *before* the project begins. That may mean that lots of things get financed at much lower levels or in much smaller pilots. When the vast majority of innovations fail, what’s wrong with that? (The answer, of course, is that some worthwhile innovations may not happen at all. I leave it to others to figure out whether that’s a real risk here.)

    • One of the keys to understanding the ed reform battles is that, barring a few sociopaths, almost everyone believes they are doing something great for kids. Even the hucksters have convinced themselves they are “doing well by doing good.” Some players (on both sides but disproportionately among movement reformers like Chait) tend to forget this, but very few people get involved with education with anything less than the best intentions.

      • How could anyone really know this either way? And even if one could, does it really matter? I mean what if E. Holmes has sincerely trying to make blood testing better at the start?

        • Keep in mind, I was responding to this: “it strikes me as a bit harsh in the sense that at least (many of) these people were trying to do something positive.”

          But to your point, yes it is a subjective statement (albeit one based on a few decades of experience). The bigger idea here is that if everyone has good intentions then intentions don’t have any place in the discussion. This is one of those cases where should start with the assumption of good faith.

      • This conversation centers on what innovators/hucksters do–and there were plenty of details about that in the article–but I read (okay, scanned) this list as commentary on the larger phenomenon. As I’ve learned standing next to my teacher wife at parties for 20 years, everybody has an exciting idea for how to fix education, whether it’s vouchers or iPads or flipped classrooms or charter schools. Of course there are going to be some swindlers, but their existence isn’t necessary because legislatures and school boards and school administrations are fully stocked with well-intentioned people who want to say yes to this stuff.

  2. this hardly seems fair:

    > Test Prep
    > [Year](http://2015trends.hackeducation.com/business.html) after [year](http://2016trends.hackeducation.com/business.html) after [year](http://2017trends.hackeducation.com/business.html) after [year](http://hackeducation.com/2018/12/31/top-ed-tech-trends-money) , the most well-funded startups in education technology were those offering tutoring and test prep. Neither of these are progressive trends. These can [exacerbate educational inequalities](https://academiccommons.columbia.edu/doi/10.7916/D8K64R8K) as more affluent families can afford to give their children extra academic support (or at least some extra tips and tricks on how to do well on standardized tests).
    > To counteract some of these inequalities — in test prep and in the testing itself — the [College Board partnered with Khan Academy in 2015](https://www.insidehighered.com/news/2015/06/02/college-board-and-khan-academy-team-ease-access-new-sat) to provide free online SAT prep courses (after years of insisting that test prep would actually [make no difference](https://www.washingtonpost.com/news/answer-sheet/wp/2017/05/09/can-coaching-truly-boost-sat-scores-for-years-the-college-board-said-no-now-it-says-yes/) in how well one performed on the exam). One year later, the head of the College Board [David Coleman boasted](https://www.youtube.com/watch?time_continue=3&v=kQ_u79-jc_k) “Never in my career have I seen a launch of technology on this scale that has broken down the racial divisions that so haunt this nation — never.” And in 2017, the College Board released [data](https://www.collegeboard.org/releases/2017/average-score-gains-on-redesigned-sat) that showed “20 hours on free Official SAT Practice on Khan Academy is associated with an average score gain of 115 points, nearly double the average score gain compared to students who don’t use Khan Academy.” One problem though: not all students practiced the same amount, and students with highly educated parents tended to spend more time doing so. [As Matt Barnum and Sarah Darville write in Chalkbeat](https://www.chalkbeat.org/posts/us/2019/09/10/college-board-paul-tough-david-coleman-research-sat/) , “The College Board’s research doesn’t show whether Khan Academy truly /caused/ the score increases. Perhaps the students who used Khan most were particularly motivated or were using other study aids.”
    > Perhaps test prep will never undo the inequalities baked into the system. Perhaps, just perhaps, it was never really meant to.

    so it’s a scandal that the test prep that the College Board made freely available online… uh, isn’t yet backed by an RCT? what exactly is this even complaining about? this is just a complaint about the SAT in general that has nothing to do with this initiative which seems unambiguously net good.

    • messed up formatting! let me try again.

      this hardly seems fair:

      > Test Prep

      > [Year](http://2015trends.hackeducation.com/business.html) after [year](http://2016trends.hackeducation.com/business.html) after [year](http://2017trends.hackeducation.com/business.html) after [year](http://hackeducation.com/2018/12/31/top-ed-tech-trends-money) , the most well-funded startups in education technology were those offering tutoring and test prep. Neither of these are progressive trends. These can [exacerbate educational inequalities](https://academiccommons.columbia.edu/doi/10.7916/D8K64R8K) as more affluent families can afford to give their children extra academic support (or at least some extra tips and tricks on how to do well on standardized tests).

      > To counteract some of these inequalities — in test prep and in the testing itself — the [College Board partnered with Khan Academy in 2015](https://www.insidehighered.com/news/2015/06/02/college-board-and-khan-academy-team-ease-access-new-sat) to provide free online SAT prep courses (after years of insisting that test prep would actually [make no difference](https://www.washingtonpost.com/news/answer-sheet/wp/2017/05/09/can-coaching-truly-boost-sat-scores-for-years-the-college-board-said-no-now-it-says-yes/) in how well one performed on the exam). One year later, the head of the College Board [David Coleman boasted](https://www.youtube.com/watch?time_continue=3&v=kQ_u79-jc_k) “Never in my career have I seen a launch of technology on this scale that has broken down the racial divisions that so haunt this nation — never.” And in 2017, the College Board released [data](https://www.collegeboard.org/releases/2017/average-score-gains-on-redesigned-sat) that showed “20 hours on free Official SAT Practice on Khan Academy is associated with an average score gain of 115 points, nearly double the average score gain compared to students who don’t use Khan Academy.” One problem though: not all students practiced the same amount, and students with highly educated parents tended to spend more time doing so. [As Matt Barnum and Sarah Darville write in Chalkbeat](https://www.chalkbeat.org/posts/us/2019/09/10/college-board-paul-tough-david-coleman-research-sat/) , “The College Board’s research doesn’t show whether Khan Academy truly /caused/ the score increases. Perhaps the students who used Khan most were particularly motivated or were using other study aids.”

      > Perhaps test prep will never undo the inequalities baked into the system. Perhaps, just perhaps, it was never really meant to.

      so it’s a scandal that the test prep that the College Board made freely available online… uh, isn’t yet backed by an RCT? what exactly is this even complaining about? this is just a complaint about the SAT in general that has nothing to do with this initiative which seems unambiguously net good.

      • It’s just so unfair and unequal that hard working people can use the resources they generate from that work to learn more and become more wealthy. It’s really just time we take everyone’s money away so incompetent people *really have a chance* for a change. It’s pretty clear from social science that incompetent people are making headway, but we really need a big push to *finally* remove the inequalities that they suffer from!!

        All this ed tech that helps competent people become more competent really has to be stopped. Competence is making people unequal, and that’s not fair.

        • Anon:

          See items 1 and 100 in the above post. It seems that the only thing competent about these people is that they’re competent at making money by ripping people off.

        • Just to continue on this theme: A common feature of this blog is the idea that just because people have fancy credentials, or a lot of money, or adoring media coverage, that doesn’t mean that there’s any substance behind their claims. Worldly success can come from all sorts of sources. Lots of people achieve worldly success by promising things they can’t deliver, or by working the back rooms.

          I’m not saying that all or even most of the items on the “100 Worst Ed-Tech Debacles” are scams—I haven’t looked into them, and I have no idea—but I’ll definitely push back against the naive default attitude that success comes from competence. Unless you circularly define “competence” as being “whatever it takes to get success,” so that the pizzagate guy was competent at getting media attention and grant funding, the gremlins guy was competent at getting papers with very little data or theory published in economics journals, the beauty-and-sex-ratio guy was competent at getting papers with statistical errors published in biology journals, the sleep guy was competent in giving a Ted talk, etc.

        • But isn’t that the purpose of credentials? To allow a shortcut to knowing which claims are trustworthy and which not?

          It’s ok to question specific credentials but I find a lot of posts on here are just purely anti-credential.

        • Rahul:

          Yes, I think credentials are useful. I work at a credential-granting institution! Credentials provide some information, but this information is superseded by seeing what people actually have done. In the case of the Why We Sleep guy, I’m not questioning his credentials, I’m saying that he does bad work, despite having good credentials.

        • Andrew: That’s fine.

          What I mean is when we criticize the hierarchy of Journals for example. I think there should be credible journals and less credible ones. If the “good” ones publish crap they should lose some of their credibility currency.

          But to move to a system where everyone just posts articles in some sort of “flat” system e.g. arxiv would be counterproductive.

        • > All this ed tech that helps competent people become more competent really has to be stopped

          You really think SAT test prep is becoming more competent? It’s an arms race. It’s the exact opposite of what you describe, it sucks resources AWAY from acquiring actual skills or content knowledge INTO a meaningless competition.

        • Thanks for this mpledger – Toby Morris’ cartoons are so insightful. What a pity that nothing much has changed in the six years since he wrote it.

          (PS How are you – long time no see)

      • The Chalkbeat article linked has a lot more in it than “not backed by an RCT.” It includes the acknowledgment by the College Board that “The notion that a free tool would disrupt those patterns on day one would be unbelievable, really,” he said. And the authors of the Chalkbeat article’s final claim: “It’s impossible to tell what the overall effect has been, but Coleman’s claims from 2016 seem overstated.” So it’s not just that the overall claim is unproven — it’s that the initial claim wasn’t even believed by the organization that produced it.

    • Almost all of these are characterized by not being based on research and not involving people with teaching experience. Sure none of them meant to be harmful, but the arrogance of assuming that they know better because they have funding is the common theme.

  3. Alfie Kohn (whose view in homework is discussed in one of the links) has written a book that connects this article to the previous post:

    -snip-

    In this collection of provocative articles and blog posts originally published between 2010 and 2014, Alfie Kohn challenges the conventional wisdom about topics ranging from how low-income children are taught, to whether American schools have really fallen behind those in other countries. Why, he asks, do we assume learning can be reduced to numerical data? What leads us to believe that “standards-based” grading will eliminate the inherent limitations of marks? Or that training students to show more “grit” makes sense if the real trouble is with the tasks they’ve been given to do?

    https://www.alfiekohn.org/sbm/

  4. A major theme behind all this for me, apart from the specifics behind all these cases, is that far too much effort, thought, and energy for improving teaching and learning these days go into developing and learning new tech tools, “going digital”, whatever.
    There are so many other things education still worth talking and thinking about!

  5. Reading the list (and especially the photos) made me think that Andrew’s linking his blog to Watters’ list is a classic case of putting the cat(s) among the pigeons.

Leave a Reply to Anonymous Cancel reply

Your email address will not be published. Required fields are marked *