Jurassic AI extinction

Coming to theaters near you, some summer, sometime, maybe soon

This is Jessica. I mostly try to ignore the more hype-y AI posts all over my social media. But when “a historic coalition of experts” comes together to back a statement implying the human race is doomed, it’s worth a comment. In case you missed it, a high profile group of signatories came together to back the statement: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Intellectuals sometimes like to make statements, and some even try to monetize them, but how did this group arrive at this particularly dramatic sentence? There’s been plenty of speculation, including among experts, about scenarios where AIs like LLMs do things like trick people, or perpetuate stereotypes, or violate data privacy, and how these things could get worse when they are deployed with more agency in the world. But how did we get from here to extinction all of a sudden? Why couldn’t we just emphasize safety or something? We already have plenty of centers and initiatives for AI safety. Are we now going to see rebranding around extinction? 

It makes one wonder if this coalition is privy to some sort of special knowledge about how we go from ChatGPT not apologizing for its errors to populations dying off. Because if we can’t foresee how that’s going to happen, why make extinction the headline? What about just making the world less objectionable/unfair/deceptive/anything else we actually have evidence AI can contribute to? What exactly does upping the ante contribute to the already overhyped AI landscape? 

Maybe we need to put ourselves in the shoes of the AI researcher who has spent their career being celebrated for their advancements to the technology, but who hasn’t really engaged in much criticism involving AI’s negative potential. It seems reasonable to imagine that they might feel some pressure to sound the alarm in some way, given how much louder the critics are now that the hype machine is finally recognizing that it’s worth hearing about the darker side. I think many of us sense that it’s gotten harder be a techno-optimist without acknowledging other people’s worries, even if you haven’t really paid attention until now.

So maybe extinction appears as an easy way to signal being concerned about AI. It puts you in the headlines for your altruistic urges, but under a concern that’s indefinite enough that an equally vague gesture toward solutions, like “we need to focus more on responsible AI or AI safety” feels satisfactory. At least for me, extinction is not very emotionally evocative. It’s a nice clean way to signal terror without having to get too specific. We might as well be making vague references to something like “closing the loop.” 

But its confusing … when did it become cool again to be the dramatic AI guy? I can remember a faculty candidate once unironically mentioning the singularity in their job talk when I was a grad student, and suddenly everyone sat up a bit, like Did he actually just say that? It was sort of an unwritten rule that you couldn’t get all sci-fi extremist and expect to be taken seriously. But now existential risks seem to be experiencing a resurgence. This would not necessarily be a problem if there was some logical argument or model to warrant the extreme projection. But where’s the evidence that it’s extinction that we need to guard against, rather than the more mundane (and much more probable) human bias gets amplified, self-driving car causes wreck, family man gets put in jail due to automated face recognition etc. kind of world? 

This seemingly rapid leap–from how language models that predict the next word can generate human-seeming text to experts warning about the extinction of the human race–makes me think of Plato expressing his fear of the poet in The Republic, who he thought was so dangerous that he should be banished from the city. My understanding is that Plato’s fear didn’t seem so extreme at the time, because there was no sharp distinction perceived between what we would now call works of art versus other types of objects that seemed to possess their own creative principle, like nature. So the idea that a poem that could be in the world like a natural object carried weight with people. But even Plato wasn’t talking about human extinction from the words of the poet. His concerns with art were more with its potential for moral corruption. At any rate, Plato’s fear starts to seem pretty down to earth relative to the words of these AI experts. 

I can understand how AI researchers who have been raising concerns about AI safety for years would find this slightly annoying (many of whom are women – Margaret Mitchell, Abeba Birhane, and many others – whose work tends to go unmentioned when a Hinton or LeCun speaks up). Someone responsible for contributing to the underlying technology becomes concerned after being pretty quiet for years and its a massively newsworthy event that paints that person the new spokesperson for safe AI. I’m glad there’s some recognition when the people responsible for some of the key technical innovations say they’re not convinced it’s all good – being up front about the limitations of the methods you develop is healthy.  But when a bunch of people jump to align themselves with an extreme sentence that seems to come out of nowhere, it must be frustrating to those who’ve spent years trying to get the community to the point of recognizing any consequences at all. 

81 thoughts on “Jurassic AI extinction

  1. I’m sure the various signatories had many reasons for signing this statement. One reason that has not gotten much attention is the financial incentives of the organizations that have initiated these recent petitions and statements. These groups depend for their funding on convincing donors that AI is an Existential Threat. For them, these are basically fund-raising letters similar to those that we receive every day from politicians and political parties.

  2. Just to be cynical, could it have to do with money and ego? Isn’t there sort of an arms race going on, so it’s seems like maybe a good idea that participants signal concerns but make them distant and irrelevant/unlikely enough to not really have to do anything substantive at the moment?
    Also, from an ego perspective, it really makes what you are doing sound on par with Manhatten Project or something.
    Disclaimer, I don’t know much about this field (just some about neural nets, GPT, and what I’ve read in the news), but when I read the statement, that’s what came to mind.

  3. This is nothing more than a transparent attempt to regulatory capture at an early stage it is completely ignorable except in so far as it tells us about exactly how broken our economy is.

    • DL,

      Along similar lines:

      “Indeed, focusing on this particular threat might exacerbate the more likely risks. The history of technology to date suggests that the greatest risks come not from technology itself, but from the people who control the technology using it to accumulate power and wealth. The AI industry leaders who have signed this statement are precisely the people best positioned to do just that. And in calling for regulations to address the risks of future rogue AI systems, they have proposed interventions that would further cement their power. We should be wary of Prometheans who want to both profit from bringing the people fire, and be trusted as the firefighters.”

      https://www.fast.ai/posts/2023-05-31-extinction.html

    • My thought entirely. I’m sure the people at OpenAI (really should be calling it ClosedAI), Google etc. have some concerns about AI safety, but regulating AI in the same way that the nuclear industry is regulated, for example, would be a great way to eliminate potential competition.

  4. Same hype strategy was used for gene editing a few years ago. You get the “experts” to start discussing concerns about how dangerous the tech could be, or the ethics of it, or whatever bikeshedding aspect that everyone can have an opinion about.

    Meanwhile, the tech continues to not really work the way you would think from a shallow understanding and fails to deliver on the hype. It seems to be a standard marketing/propaganda technique people should learn to recognize.

  5. One thing they’re concerned about is military uses of AI; e.g., having AI in control of weapons systems, including nuclear weapons systems.

  6. Speaking as a statistician who is right now transitioning away from medicine to work on AI safety; I think it’s quite simple. I really am just concerned that superhuman-level AI’s could kill everyone.

    And I care a LOT less about all the other concrete risks you mentioned. Even though they’re far more likely, it’s an expected value calculation, and they just don’t weight.

    I just don’t think it’ll be all that long before we can see a language model that could successfully do things like:

    -hack computer systems
    -subdivide big conceptual problems like “escape and replicate” into small sub-steps
    -farm out those small problems to up to 10,000 copies of itself to perform in parallel
    -get more resources to run those copies of itself (either earning money to purchase it, stealing money, or hacking access to GPU’s)
    -extract its weights by hacking or train a model similar to itself out onto the internet
    -distribute on the internet and hack into various devices, become essentially unkillable
    -do distributed training, and get more powerful

    … then do whatever it wants? Which is very unclear, but I don’t want to play the game of “I hope we trained our new world dictator to be sufficiently nice.”

    You don’t have to buy that this is the specific path we’ll see, there are lots of other ways it could actually play out.

    But the notion of making something that’s smarter than humans, and can replicate itself to vast scale while running much faster than humans can stop –

    and we’re still actively building out these kinds of capabilities, with impressive upward slopes –

    Isn’t existential risk the right framing for something like this?

    • Yes, exactly. Whether they’re right or not, the alarmists see the AIs as a sort of competing species, with the near-term potential to outdo us at the very thing that enabled us to smash down every other species that has gotten in our way. If they’re self-replicating, their “interest” is in further self-replication and to the extent that we stand in the way, too bad for us. But…

      Are AIs self-replicating? I have no idea. So far, their replication depends entirely on human effort, which admittedly is not in short supply.

      Do AIs occupy the same ecological niche as human beings? Not exactly. As far as I can see, they don’t eat corn or beans or lettuce or meat, so why should they want to push us off the land? AIs do eat electricity, which could be a problem. At the moment, they are very far from managing the mining, construction, transportation, knowledge, and organization needed to create electrical systems, but they could very conceivably recruit human minions to manage their electricity acquisition.

      Or maybe they’ll decide we’re cute and fawning, like dogs, and keep us around for the ego strokes and companionship.

      Nothing to worry about? Maybe you’re just not a worrier, Jessica!

    • “And I care a LOT less about all the other concrete risks you mentioned. Even though they’re far more likely, it’s an expected value calculation, and they just don’t weight.”

      I think we should all be very, very skeptical about this sort of reasoning. You could basically justify anything, e.g. we’re building rockets to intercept asteroids and so protection against planetary extinction, and by comparison the rights of people group A don’t matter, etc.

      I think this is actually part of the ploy. As Jessica mentions, focus on big vague potential threats that kinda of get you off the hook for working to countenance the harm AI doing right now.

        • Yeah, I’m a big fan of AI and in general a technological optimist. My point is just that this sort of “big scary potential threat trumps concrete harms” is a form of utilitarianism/futurism that makes for poor ethical reasoning.

      • There has been millions of dollars spent on assessing whether there are asteroids that represent an existential threat, and people also have targeted asteroids to see if they can change their trajectories. The point is asteroids, to the best of our knowledge, do not represent such a threat. Many people do actually think extinction is a possible risk from AI.

        I think this can of course be a classic bootleggers and baptists situation where the views of some sincere but misguided people are co-opted by greedy business leaders who see the moral point as being in their selfish interest to promote, but that is a different question.

        If you believe that AI poses such a risk then it is the true that issues about algorithmic bias or whatever else pale in comparison. Moreover, there are tons of people already concerned about and working on those issues (and the media had a field day with racist robots too, so it’s hardly like this is just an attempt to get in the press)

        • Jamie:

          I’m much more scared by killer robots than racist robots, although I guess racist robots can implicitly be killer robots if they tell the cops to shoot you.

        • Andrew – I believe Robocop and other famous law enforcement robots kill without prejudice, which may be reassuring :-|

        • Jamie, you say “The point is asteroids, to the best of our knowledge, do not represent such a threat” but that’s not true at all. More broadly, meteors, whether originating in the asteroid belt or elsewhere, do represent a threat. You can pick your level of disasterness, from “tens of millions of people dead and a huge setback for civilization” (something like 1 in a million chance in any given year) through “major extinction event that wipes out all large mammals among other species” (perhaps 1 in 100 million).

          The fact that earth will -eventually- be struck by a massive meteorite is not really in question. It had happened tens of times in the past and it’s not about to stop.

          Of course it does not necessarily follow that we should be spending money on this issue. I’m not making a claim about that either way.

        • Phil said:

          “The fact that earth will -eventually- be struck by a massive meteorite is not really in question. ”

          +10!

          This is or alien invasion is the only actual existential risk humans face. Every other kind of “existential” risk that has been claimed – from AI to nukes to climate change – is unlikely to create instantaneous extinction and/or would provide **ample** time for humans to mitigate any significant specific threat that emerges. With regard to AI, even if some rogue AI or AI developer tries to wipe out all of human kind, other AIs can be used to suppress it.

          It’s sad that people are undermining their own credibility and the credibility of all of science and technology by making claims like this.

  7. I suggest doing some readings about their concerns instead of trying to psychoanalyze them? They’re not hiding their reasoning, and they have good cause to be worried (and “AI might kill everyone” is a real problem the way “people might use AI in ways that don’t 100% align with my politics” isn’t).

  8. Whatever you think of the merits, I don’t think the characterisation of these concerns as coming out of nowhere is an accurate reflection of history. There are folks (e.g. Yudkowsky and co) who have been concerned about and working on AI safety from an x-risk perspective for (I think) well over a decade; it’s just that noone else paid them much attention till now. Given the nature of the risk they perceive, I don’t think those folks will find it at all annoying at all that Hinton, LeCun et al. are also raising concerns, only that they’re doing so at what they would argue is an extremely late stage.

  9. To be fair, demanding “AI safety” does more harm than spreading the fear of extinction. AI safety will only lead to some certification process for compliance reasons and in the end we will still end up with AI-guided killer robots — but certified ones!

  10. The people who created the statement actually think AI can cause human extinction. I know this since I’m reasonably close to that community. Most speculations in the comments here about other motivations are simply wrong.

    The reason why this feels “sudden” is, in my opinion, that people worried about extinction didn’t feel like voicing this concern in the past due to fear of being viewed as crazy. That is the reason why I personally say more about it than I did before, but I am not a public figure. Now that Hinton and Bengio spoke up, there was an opportunity for all the people worried about extinction to come clean and actually say what they think, and the letter is one result of that process.

    Note that the worries themselves are not sudden, only your perception of it. There are lots of texts that explain quite clearly our current best guess why AI might lead to human extinction. Some of the best ones are the Wikipedia article on alignment (https://en.wikipedia.org/wiki/AI_alignment) and a recent paper (https://arxiv.org/pdf/2209.00626.pdf).

    • As someone who bounced off some of those communities ten years ago and is very tired to hear they are still around, I know many major figures in them are really into race ‘science’ and ideologies like longtermism! https://aeon.co/essays/why-longtermism-is-the-worlds-most-dangerous-secular-credo Some of them seem to be personally corrupt and are friends with fashy billionaires which are also red flags when deciding whether to take their words seriously.

      • As Emile Torres wrote the article you just sent, let me link to a different article calling into question the trustworthiness of Torres herself:

        https://markfuentes1.substack.com/p/emile-p-torress-history-of-dishonesty

        In general, a few high-level remarks:
        – It is true that many people working on AI existential safety were originally motivated by longtermism. I would guess it is fewer than 50% of people, but hard to tell.
        – The question of whether AI poses an extinction risk is an empirical question and the answer should thus not depend on your underlying worldview. In principle, you should be able to look at the arguments and simply tell whether you find it plausible or not that AI leads to extinction, irrespective of whether they come from longtermists or not.

        • Interpreting a technical argument requires experience and technical knowledge. I don’t have either for machine learning (just an undergraduate course or two back in the Neolithic), but I do have experience of people in these spaces pushing race-and-IQ (and anyone who has sat and watched terrible Americans knows what racial hierarchy of IQ they are going to push), letting notorious racist cranks hang out in their comment sections and blogrolls, pushing a variety of wordy theories about women and poor people, and otherwise pushing stupid selfish ideas that eat clever pasty-faced dudes alive. Since I am of a demographic vulnerable to this nonsense, I do the prudent thing and say to myself “anything those folks are pushing hard is bad news”: one of them called this “epistemic learned helplessness.”

          One of my profoundest disagreements with the LessWrong rationalists is that I don’t think being clever gives you a special power to solve any problem using your mighty brain and skipping focused study, experience, collaboration, and other slow messy things. Man is the rationalizing animal (de Camp), the fat turkey has never experienced Thanksgiving (Taleb).

        • I also have places offline and online where I engage with controversial and unconventional ideas in some of my areas of expertise. But since AI Foom is not one of them, I place a lot of weight on “predicting the future of economic or technological systems is very hard” and “many of the people pushing the theory wave red flags like its May Day in 1952.”

        • I disagree with Sean’s allegations and think people should not believe them without seeing strong evidence.

        • Leon Lang: David Gerard in Australia has put a lot of time into documenting and recording those connections in written and audio form all over the Internet. Anyone can look up Peter Thiel, Scott Alexander’s enthusiasm for race ‘science’ and biological-race thinking, Eliezer Yudkowski’s various adventures, the Longermist Effective Altruists who bought a castle (castles?) with donations, Caroline Ellison of FTX, the Machine Intelligence Research Institute and its various rebrandings, Steven Pinker’s Race-and-IQ connections, etc. etc. etc. and see a group of closely connected people most of whom push at least one of a short list of terrible ideas and many of whom push AI Foom. Or just listen to a few random episodes of Julia Galef’s podcast and think a bit about how the ideas her bright young prosperous guests advocate have gone wrong in the past and could go wrong in the future! For all of their other flaws these folks make their ideas and connections a matter of public record.

        • Sean:

          I didn’t know this about Julia Galef—none of this came up when she interviewed my sister and me about our research!—but I have noticed a lot of race and gender essentialism floating around in that general corner of the world. I discussed some of this in my review of the book by the science journalist Nicholas Wade (see followups here and here), my discussion of an article about genetics and rich and poor countries, and I noticed it way earlier when it came up in the Freakonomics blog (see the quote on the top-right of page 318 of this article. In the latter case it was gender rather than racial essentialism but it was the same complacent attempt to explain the inequalities of the world via stereotypically innate differences.

        • To clarify, I respect Julia Galef and have seen no sign whatsoever that she is in to race ‘science’ or wants to turn the solar system into a giant computer simulating humans. The problem I see is more with a subset of her guests and the people, norms, and ideologies which influence those guests. I like that phrase about people in these spaces looking for reasons why inequality might just reflect innate difference and not the structure of society! That can be true but it can also be an excuse.

          I will stop here because the Internet has enough arguments about rationalism and longtermsm and AI.

        • Leon Lang: but Torres (who seems to go by they/them on twitter) is deeply suspicious of LessWrong or longtermist scenarios for AI doom or analyses of the risk, at least as far as I understand their argument in the WaPo. One of the concerns that a lot of us have (including people like Charlie Stross who have known these characters since the 1990s) is that these movements ways of framing the problem are not helpful or actively get in the way, in part because some prominent figures in these spaces are into skeevy ideologies like race ‘science’ or eugenics. A good undergraduate course will explain that the whole concept of “artificial intelligence” is poorly defined. Torres agrees that deciding what aspects of software to worry about is hard, but is suspicious of LessWrong and longtermist approaches (as am I from my contact with them ten years ago). This is my last post on the subject.

  11. Completely agree with “conchis” and “M”.

    If the author has not been following all the arguments that Yudkowski’s and Bostrom’s (and largely, Effective Altruists) have been making for years about existential AI threats, then its understandable why she ascribes the motivations of all these experts to basically virtue signaling. That seems extremely unfair.

    In general, one observes lots of meticulous dissection of potential economic or other types of motivations behind existential-AI-safety letters. Now its important to be aware of such motivations, lest someone is trying to game you. But its also important to be aware that this is an ad hominem, we must evaluate the argument on its merits which is done by the long and hard work of listening to 4h podcast discussions and reading bunch of books like Bostrom’s Superintelligence.

  12. In the realm of natural, rather than artificial, intelligence, it seems relevant that each leap in brain power during the evolution of homo sapiens has led to the extinction of its predecessor. Unlike the many species of, say, canines, only one hominin species is still around. So it seems reasonable to expect that an AI that achieves some level of capability that substantially exceeds ours might well do unto us as we have done unto our forebears.

    The only reason I’m not terribly worried about AI leading to the extinction of humanity is that I think it is more likely that we will do it to ourselves in some other way first. Nuclear war, in particular, seems a clearer and more present danger.

    • I’m not worried all that much about nuclear war. What we’re doing to the environment, on the other hand, really is horrific, and really could lead to our extinction.

      As I keep saying, every time there’s a new result in environmental science (usually monitoring), the bottom line is always: things are getting bad way faster than any previous models predicted. Antarctic ice sheets are the latest (Science, within the last 2 or 3 months). Basically, there are more positive feedback systems working to melt/collapse the ice sheets than we thought. And, they’re stronger than we though. Oops.

      And we worry about ocean microplastics destroying the food web. I read somewhere that one major source of ocean microplastics was tire/road friction emitting plastic dust*. Once again, the private car is shown to be one ot the stupidest, most insane ideas imaginable. But I’d guess I’m pretty much the only one here who as never** bought a car…

      *: And EVs will make this problem worse, since they have higher torques at low speeds.
      **: I never bought a car because I didn’t want a car for car-inherent reasons (expensive, dangerous, stinky). I’m no environmentalist.

  13. Two recent news items give, I submit, a hint as to what’s going on.

    1. Lawyers hand in ChatGPT written brief that’s full of invented (i.e. bogus) cases.
    2. Psych help service notices ChatGPT is offering unhelpful advice.

    People are begining to realize that AI doesn’t actually work (since it really doesn’t) and thus the industry now feels it needs the hype machine to keep toeing the line that AI is amazing. Also, Hinton and LeCunn are committed to the NN model, even though actual mammalian neurons aren’t anything like that. So even though they _seem_ to be admitting that there are problems, their basic intellectual stance remains that the layered NN model “works”. (Another thought here, maybe they see AI as escaping from academic (i.e. their) control and they want academia, not industry, to be the intellectual center. (Yes, this is jaundiced beyond words. But it may be the best theory yet…))

    Whatever, there are lots of fun theories that can be hypothesized for this current round of AI inannity.

    (When I ran out of academic energy after passing the quals (translation: couldn’t find a problem I wanted to commit to), one of the things at the back of my mind was that if I did AI as my main thing, I’d spend my life screaming at idiots. I seem not to have escaped that fate after all. Sigh.)

  14. > The only reason I’m not terribly worried about AI leading to the extinction of humanity is that I think it is more likely that we will do it to ourselves in some other way first. Nuclear war, in particular, seems a clearer and more present danger.The only reason I’m not terribly worried about AI leading to the extinction of humanity is that I think it is more likely that we will do it to ourselves in some other way first. Nuclear war, in particular, seems a clearer and more present danger.

    Seems to me that AI magnifies those dangers, but certainly they aren’t mutually exclusive maybe there’s an additive gain in risk. Although I guess it’s also theoretically possible that AI could mitigate this risks.

    At any rate, I think the inherent risks of AI are theoretical and I don’t really understand the arguments. But I think clearly AI will be a very powerful tool and I would think it won’t be very hard for malicious actors to gain access to thsg very powerful tool – like nuclear weapons in a sense.

    I just listened to part of this podcast where the interviewee calls for a Manhatten Project-like investment in AI to ensure that the good guys outpace any bad guys.

    Reading the cynicism in this thread, I suppose thst idea (and his supporting arguments) won’t go over too big with this crowd. It I see it as at least meriting consideration (although I think it’s a pipe dream to think that this society will would ever be able to get together in any such collective action. Those days are gone by.)

    https://nonzero.substack.com/p/a-manhattan-project-for-ai-safety#details

  15. To reiterate, AlphaGo beat the world champion Go player. AlphaChess beats other chess programs which beat grandmasters. AlphaFold predicts how proteins will fold about 30% better than any other known system. GPT-4 has passed final exams in several graduate-level courses (without access to the Internet or a calculator, just based on the neural network weight parameters it developed during its training). Those who say neural networks don’t really work and it is all a parlor trick might possibly not be paying enough attention.

    The evil AI in science fiction goes way back before Space Odyssey’s HAL. It is easy to see how an evil AI could do a lot of damage, or, more likely, how AI’s could be misused. The missing step for me is how an AI gets evil naturally or by accident, since the only motivations an AI has to do anything are provided by the humans who program it. AlphaFold has no reason to want to rule the world. Granted, bugs happen, but if an AI does destroy us it will be entirely our own fault (those of us who programmed it).

    I would like to see an FDA for AI’s, that tests them in safe environments and certifies them before they can be used commercially or by the government. I think we are at about the point where that would start to be beneficial. Given the current USA politics I doubt that will happen here though.

    • To reiterate, AlphaGo beat the world champion Go player.

      https://www.ft.com/content/175e5314-a7f7-4741-a786-273219f433a1

      Amateur humans can beat KataGo (stronger than alphago) by memorizing an adversarial strategy.

      The moment you step outside the typical set of a neural network’s training data, they fall apart spectacularly in a way that humans do not. Look up “adversarial examples” and “covariate shift.” While I don’t agree with this term, that is what is meant by “parlor trick.”

      • I’m not sure the latest, but almost immediately after that adversarial attack paper was published, the KataGo folks added such attacks to the training data and patched the program to be able to handle that class of situations and it was getting better at handling them.

        (Although, interestingly, one thing that Minsky and Pappert said in 1972, was that perceptrons were not able to recognize whether a string figure was closed or open, and that inability is pretty much what enabled this particular adversarial attack.)

        Like ChatGPT, when a glitch is found, someone fixes it.

        The KataGo folks seem to think that there are classes of human-used strategies that KataGo can’t, in principle, learn. (The basic idea is that although KataGo is really really good at reading out fights, if the human identifies a condition that would make such a fight not work, creates that condition long in advance, then Katago might not realize it was being snookered.) But in real life, KataGo acts as though it really really does understand the concept of “snookering”. To the point of it being really really depressing.

        (I claim that I’m almost (and will be able to get) good enough at Go that I can beat anyone (including KataGo or any future program) if I’m given a four stone handicap. I was roughly to that stage for teaching games against Japanese pros. But KataGo is seriously viscous. I’m pretty sure that I’m very close to that point for 5-stone games, though. (The problem here is that the difference between 5, 4, and then 3 stone games is ridiculously large, and each reduction in handicap introduces new conceptual problems. E.g. with 5 stones, black has a stone in the middle of the board, and the standard snooker of creating outrside strength around the edges (by threatening moves that the frightened human reacts to overly passively) and then grabbing an enormous center area doesn’t work, but with 4 stones, you have to be able to avoid that snooker.)

        But as I’ve said before, drawing conclusions about anything (other than that computer nerds (a) aren’t as dumb as you think and (b) like writing chess and Go programs) from computer chess/Go, really doesn’t make any sense.

        • But as I’ve said before, drawing conclusions about anything (other than that computer nerds (a) aren’t as dumb as you think and (b) like writing chess and Go programs) from computer chess/Go, really doesn’t make any sense.

          Trouble is that all nonparametric learners, especially neural networks, are subject to this kind of adversarial attack. Finding such exploits is also algorithmic with access to a serialized representation of the network–the glitches get patched, but there’s just more glitches. Covering all the glitches is essentially equivalent to covering the entire high-dimensional input space with training data — undecidable on realistic timescales.

          There’s a reason you could see crazy stuff like this 5 years ago

          https://www.youtube.com/watch?v=gn4nRCC9TwQ

          but to date we don’t have delivery robots walking around with one of them in its brain, despite the advent of low-cost ML inference chips in toasters and cameras and stuff. I’m not going to say that they don’t work–they’re obviously useful sometimes–but they are a kind of “trick” in that if you move through the input space a tiny bit weird, either on purpose or by accident, they collapse, like viewing a magic trick from behind the stage.

        • Somebody said:

          “Trouble is that all nonparametric learners,…”

          Sure. But Go programs also run an MCTS, which is actually where their strength comes from*. Attacks against a Go program have to work both against the NN and against MCTS.

          I agree that the whole NN thing is largely BS, but they do happen to have exactly the same geometry as the Go board…

          (Yes, I seem to be saying opposite things: NNs are BS but the Go programs are kewl. They’re not opposite because Go programs aren’t just NNs.)

          *: A program (Zen, that doesn’t even use the graphics card) that KataGo beats even with a handicap** trounces KataGo, even with KataGo taking a handicap, if you turn off the MCTS.

          **: Oops, I forget what handicap I used. Whatever, it’s a complete reversal, from KataGo being several levels above mid-range pros to being several levels below.

          (People forget that Zen was playing at mid-level pro strength before Google threw a server farm at the problem.)

        • Sure. But Go programs also run an MCTS, which is actually where their strength comes from*. Attacks against a Go program have to work both against the NN and against MCTS.

          I’m a little confused about that. As far as I know, the monte-carlo tree search still uses a NN evaluation function on board strength at each searched node. If the NN evaluation is bad, the tree-search will be bad. I don’t actually know anything about go–I think there’s some non-board state in there–but on the board alone it looks like it’s at least 19*19*2 = 722 dimensional, so you should always have a path to take the game *somewhere* the NN breaks.

          I do think the monte-carlo tree search is the more important engine of success in computer board game engines. I’m also sure that if you replace the NN with a hand-tuned evaluation function but keep the rest of the engine the same, the AI will play extremely well without being vulnerable to adversarial attacks by amateurs. However, it may also no longer be able to beat the human world champion, so I wouldn’t classify the NN as BS–it can solve problems that other things can’t, but it’s also fragile and very expensive.

        • Somebody said:

          ” If the NN evaluation is bad, the tree-search will be bad.”

          No. The MCTS finds the tactics the NN misses. Again, without MCTS, NN-only gets womped by a pre-NN MCTS only program, with MCTS, it’s the non-NN program that gets womped.

          An NN program without MCTS plays, at first glance, what seems to be amazing Go. But it’s groups die. That’s because it plays moves that a strong Go player would recognize as (local) “good shape”, but which of the multiple locally optimal moves is the better global strategy requires the search. And it takes very little search to switch from it’s own groups dying to it’s opponents groups dying.

          The attack in question was a global tactic that would have taken a lot of search to find. My bet is that that’s the last of the low-hanging fruit. (The Go programmers had to kludge in things like ladders so the search didn’t have to do the work to find them every time. Also, there were some openings that took more search than GPUs could do five years ago, so those got kludged in, too.)

          And just because the NN can’t find cyclic groups, doesn’t mean the program that uses an NN can’t check (using ordinary computation) that the NN has missed a cyclic group issue. That is, there’s a lot of special-case stuff in a modern NN Go program; it’s not an academic game to see if NN-only works. It’s hack and slash hackery as well.

          Which is why I said making philosophical conclusions (other than that nerds are smarter and harder working than you thought, and that they like Go) from the Go program world is a bad idea. One of the anti-AI philosophers back in the day screamed loudly that it would be impossible for a computer to play chess. Greenblatt’s program trashed him easily.

          An important point here is that while ChatGPT and the LLMs play a game of processing meaningless tokens while having no actual knowledge of, or ability to reason about, the real world, the chess and Go programs have evaluation functions that report on the actual state of their “real world”, and “reason about” that real world by search.

      • “The moment you step outside the typical set of a neural network’s training data, they fall apart spectacularly in a way that humans do not.”

        In my experience, in industry, humans fail quite a lot when outside their training experience. Niels Bohr: “An expert is someone who has made every possible mistake in a narrow field.” A difference is that learning can be continuous in humans, whereas most neural network programs are not coded to analyse failures and keep updating themselves after their formal training is over–yet.

        AlphaGo made a move in its tournament against the South Korean World Champion which astonished watching experts. The World Champion later said, “I thought no machine could ever beat me, but when I saw that move, I knew I would lose the tournament.”

        I tend to see it as a double-standard when accomplishments by neural networks are often responded to with “but what about this or that failure.” Humans fail a lot, some of us almost every day.

        • I tend to see it as a double-standard when accomplishments by neural networks are often responded to with “but what about this or that failure.

          Maybe you tend to see it that way, but it is not. These are categorically different kinds of failures. You cannot memorize a strategy, even in secret, despite full access to the history of notable games, which will allow you to beat the Go world champion. Or if you can, please do it and become wealthy.

          My point is not that neural networks are useless. My point is that they are not superhuman intelligences, and them passing graduate student exams after seeing lots of graduate students exams means jack shit

  16. I realize this is a scary time but I’ve found that I can deal with my AI fears with certain activities. After opening a position in MSFT a few months ago, then adding several times since, I’ve found I’m better able to control my irrational fears. Each time I add to my AI stocks, I feel more positive about the future and less fearful of AI, and I find myself worrying less about dystopian scenarios of disaster. I also find that when one of my AI stocks rises by double digits in a day, I become less depressed and more optimistic.

  17. I’ve seen enough of these guys talking about the imminent danger of AI technology but none of them made any coherent argument about their claims. They all seem to be alluding to the potential of creating a “conscious machine”—a notion that makes no attempt to distinguish between optimized learning (and responses)—AI—and biological consciousness. We have received a great deal of gifts and miseries from human natural experiences; from great architecture to mass murders. Could a single AI unit (or an equivalent of a human) produce or inspire something so grand independent of humans? And what would be the motivating factor that would parallel human’s natural experiences?

    If the argument is just that AI as a tool can be dangerous in the hands of humans, that wouldn’t be an interesting argument either. Almost anything can be deployed for evil purposes; from cars to toothbrushes. I don’t think any misuse of AI would inflict novel suffering on the human psyche. These people need to shut up and think a little.

  18. For what it’s worth, I (and I’d guess many other signatories) have been worried about existential risk for years, and some of us have even been working on / writing about it. It’s just that before ChatGPT, most people would have called us crazy (but maybe that’s still the case…).

    For instance, I wrote this blog post all the way back in 2014: https://jsteinhardt.stat.berkeley.edu/blog/long-term-and-short-term-challenges-to-ensuring-the-safety-of-ai-systems. And in general much of my work has been trying to identify short-term problems whose solution would also address long-term risks from AI (among which extinction and other catastrophes loom large for me). That’s included writing e.g. Concrete Problems in AI Safety, Unsolved Problems in ML Safety, and of course a large chunk of concrete research papers on robustness, interpretability, alignment, etc.

    Other signatories who have been working on trustworthy AI for a long time (including many “near-term” problems) include Dawn Song, Anca Dragan, Jaime Fisac, Been Kim, Sharon Li, Dylan Hadfield-Menell, and Tegan Maharaj (and probably a number I’m missing). And on the policy/econ/activist side there are folks like Audrey Tang, Marian Croak, Allan Dafoe, Jess Whittlestone, Sarah Kreps, and Erik Brynjolffson, who have all been working on societal impacts of technology for a long time. Many of these are major figures and I think their reason for signing the statement is they earnestly believe it.

    In terms of why I’m worried about existential risk, I’m fairly sympathetic to this argument by Holden Karnofsky: https://www.cold-takes.com/ai-could-defeat-all-of-us-combined/. I also hope to have my own blog post on this soon: in short, within the next couple decades I expect us to have models that are at least human-level in most respects, with superhuman capabilities in several crucial domains including hacking, persuasion/manipulation, and protein engineering. Models could also adapt to new information much faster than humans due to perfect knowledge sharing across copies. Historically, most catastrophes have been caused by pathogens (rapidly-evolving adaptive systems) and other humans (intelligent agents). AI systems are the first thing that is both at once, so it takes us into pretty unprecedented territory and I’d currently estimate a 15% chance of extinction from AI (which is within the range of what competitive forecasters give, too: https://www.foxy-scout.com/samotsvetys-ai-risk-forecasts/).

    • > AI systems are the first thing that is both at once, so it takes us into pretty unprecedented territory and I’d currently estimate a 15% chance of extinction from AI (which is within the range of what competitive forecasters give, too

      “We […] received a request from the FTX Foundation to forecast on 3 questions about AGI timelines and risk.”

      It’s interesting that Sam Bankman-Fried was so worried about the upcoming collapse of civilization and/or enslavement of humanity but couldn’t foresee the collapse of his own house of cards a few weeks later.

      • Carlos:

        Regarding the house of cards: I’ve never met or corresponded with Bankman-Fried so I can’t speak to his thinking directly, but speaking in general terms it’s my impression that successful people typically keep doing what worked for them in the past. Bankman-Fried had a lot of success with his financial schemes, so he kept doing it. Sure, now it looks like a house of cards, but for years that house of cards stood up just fine, so I can see how he would’ve thought it would last forever. Similarly with the pizzagate guy and the paper shredder guy and the disgraced primatologist and the sleep guy all the other notorious researchers with their data problems: They were working with iffy data for years and years and managed to easily turn aside all objections, so it makes sense that they kept doing it, thinking their approach could go on forever.

    • “I’d currently estimate a 15% chance of extinction from AI (which is within the range of what competitive forecasters give”

      From the link you sent (if I’m reading this correctly) the range for an existential catastrophe (not extinction which is far more severe) is 4-98%. So 15% is in that range, but so is pretty much any other number.

      • I think a 4% chance of extinction is very much worth worrying about! Notably 0.1% or other numbers I’d feel more comfortable with are *not* in that range.

        • The extinction claim is not supportable by any rational analysis. It’s just another “population bomb” argument: aggregate the negatives into the largest possible extremely short steep trend and project into the future as though there will be no response to correcting them.

    • Thanks for the comment and links. I assumed your name was on there for geniune reasons, and seeing names like yours and Been Kim signing makes me want to understand the argument I seem to be missing. I agree there is value in speculating about the worst that might happen, and how we might try to prevent it, and have appreciated how your posts attempt to trace how we could get from attributes we recognize in current systems to negative future outcomes. And I’m definitely sympathetic to the work on alignment. The parts I struggle with when it comes ot the extinction argument are:

      1. The idea that even though these systems continue to be very fragile (i.e., performance drops steeply as soon as you move outside the narrow assumptions under which they were trained), we’re going to somehow soon surpass this seemingly inherent condition once and for all, such that they don’t fail dramatically at things we didn’t train them to do. (Though I can see how failures can be dangerous in themselves, which seems to be the sentiment behind a lot of the current responsible AI/AI safety work).

      2. The fact that many of the arguments for extinction (such as the Karnofsky one) seem to go something like, “If we’re going to spend time trying to prevent negative outcomes from AI, we might as well devote our attention to the worst possible thing that can happen.” By hinging on a conditional, it comes across as a religious or dogmatic belief. That’s an individual decision, which is perfectly fine, it’s just that when it motivates public-oriented actions, like a bunch of big names declaring that mitigating risks of extinction needs to be a global priority the same way a pandemic is, it implies a sense of probability, rather than just possibility, which can have consequences, e.g., potentially redirecting attention from other issues which we know are more pressing right now, and feeding the public’s fear of AI.

      Someone above said maybe I’m just not a worrier, but I think it’s more that when I perceive uncertainty to be extremely large, my tendency is the opposite of standing up and make a bold claim. I need more dogma, maybe!

      • Jessica, if I may: you observe:

        The fact that many of the arguments for extinction (such as the Karnofsky one) seem to go something like, “If we’re going to spend time trying to prevent negative outcomes from AI, we might as well devote our attention to the worst possible thing that can happen.”

        That argument also seems a bit like an argument proposed in the 11th century, St. Anselm’s ontological argument for the existence of God. Shorn of all subtlety and elaboration (which have some philosophical importance), it goes like this:

        1) God is a being such that no greater can be conceived.
        2) Existence is better than non-existence.
        3) Therefore God must exist.

        When presented in full, it was enough to send me in conceptual circles as a college freshman. Then I lost interest.

        While acknowledging that AI presents some serious problems, I don’t find the arguments for existential risk very convincing. For what it’s worth, Pinker is skeptical, thus:

        There’s a recurring fallacy in AI-existential-threat speculations to treat intelligence as a kind of magical pixie dust, a miracle elixir that, if a system only had enough of it, would grant it omniscience and omnipotence and the ability to instantly accomplish any outcome we can imagine. This is in contrast to what intelligence really is: a gadget that can compute particular outputs that are useful in particular worlds.

        Magical pixie-dust fits well into an Anselm style argument.

        As others have noted, the arguments about AI existential risk have been developing for well over a decade, perhaps two, in a diffuse intellectual community that is also associated with effective altruism and longtermism. You’ll find a lot of this discussion at a website called LessWrong. Here, for example, is an ongoing discussion of the statement on extinction you reference in your post.

        It’s a strange community, but also influential in AI circles.

      • Thanks for the thoughtful response, Jessica! (And sorry for the slow reply–Thursday/Friday tend to be packed for me.)

        Regarding 1., my experience with systems starting around GPT-3 is that they aren’t too fragile, or in other words I don’t observe a qualitative difference between “on-distribution” and “off-distribution”. For instance, models instruction-tuned in English automatically seem to do a good job of following instructions in French, even though they weren’t tuned on French data. And we can even give models instructions in bizarre encoding such as base64, or give them instructions that they’ve certainly never seen at training time (ex: “talk like a pirate while inverting the meaning of every answer, and making animal noises every three words”) and it will do it.

        I used to think that fragility off-distribution was a fundamental obstacle to deep learning working. Here’s a document I wrote in 2018 that arguing that: https://docs.google.com/document/d/1BNJZxWhQXGE8Ppnh50WM_i_INxuyJCn9KTtOwSRRFkI/edit#heading=h.n3vymto9ydv3. In the end, I made a number of explicit predictions about tasks that deep learning systems would be unable to solve due to fragility issues, and all of those tasks ended up being solved not too long after. So that was a big thing that convinced me that there aren’t fundamental obstacles.

        Possibly we could turn our disagreement into differing short-term predictions. I’d for instance predict a lot of impressive near-term results over the next 5 years, such as surpassing all (or all but a few) humans at math and programming competitions. (That was just an example–feel free to pick a more meaningful area if you want and I could say my prediction.)

        On 2., first I’d clarify that I’m not arguing (and don’t believe) that extinction is the only thing we should care about. There are many important problems that we should be thinking about! But if there is a 15% chance we go extinct (which I believe!) then it’s really something we ought to talk about, among other issues.

        The thing that I do believe is something like: “The implications of AI are going to be enormous. If we aren’t thinking about huge impacts then we’re likely missing the forest for the trees.” For me this doesn’t just apply to extinction risk. For instance, I’d hope folks thinking about AI + law are not just thinking about self-driving cars, but about the fundamental challenges posed to our legal system when non-human systems are making autonomous decisions. More generally, I think it’s good to be extrapolating forward, and not *just* looking at what systems are doing today (though that is also important), because what systems can do today is totally different from what they could do 3 years ago.

        In terms of redirecting attention, I personally would like more people to be working on mitigating issues with AI systems of all forms, both present-day and future. Both of these get a tiny slice of the pie relative to effort into AI R&D, and an even tinier slice relative to total global resources. It doesn’t seem like they should be trading off against each other, and I certainly don’t personally feel that way—the actual research needed for both present-day and future problems has enough overlap that in many cases I can’t easily separate them.

        • One thing I should add here, on why I think ‘global priority’ is warranted: I believe that preventing engineered pandemics (which could be worse than COVID) should also be a global priority, even though I think the chance of an engineered pandemic worse than COVID in the next 20 years is only around 5%. This might just be a different attitude towards risk–for me, prevention is *much* cheaper than picking up the pieces afterwards, and so for really big threats we should be working on prevention *before* the signs are obvious (in the same way that it was good people were working on mRNA vaccines long before COVID).

        • The best thing to prevent an engineered pandemic is shut down all these BSL-3/4 labs that passage viruses in human cells and leak them into the public every few years. If they really need to exist go put them in the middle of nowhere and require a 40 day quarantine at the end of each deployment.

          The next one (either engineered or natural) that generates hysteria will likely be much worse though. Now elder abuse (keeping from family/friends and scaring them) has somehow been normalized to even be considered a valid medical intervention. You can clearly see where the abuse began and ended on the all cause mortality chart.

          https://www.usmortality.com/

          Then there is antigenic imprinting to the original covid spike. That will be an issue when SARS-3 comes along. I can think of more reasons it should be worse, but really can’t see anything learned that would reduce severity. Eg, we could have figured out nutrition (easy ways to monitor and treat deficiencies), air purification, and that strange protective effect of smoking/asthma.

    • We need a Terminator movie knockoff where the protagonist is roaming the wasteland to deliver a trophy to the person who successfully predicted Skynet.

      I think that shows the problem with asking “competitive forecasters” to predict Skynet: not only is it a long-term prediction of a chaotic system (and even Phil Tetlock’s foxes are terrible at those) but there is no way to profit from successfully predicting the end of the world.

  19. Jessica:

    I guess there’s some disagreement among AI experts regarding the risks to humanity. In your post you mentioned LeCun speaking up. We had a discussion last year in the comments that went like this:

    LeCun: People being scared to death of technological progress is nothing new either. There is a very long list of retrospectively-hilarious fears of every new technological or cultural progress. An interesting case is the printing press, which the catholic church saw as potentially destroying the fabric of society. And that, it did. And we are all better off for it.

    Me: I have to admit, I’m scared of drones flying around with machine guns.

    LeCun: I’m scared of machines guns in the wrong hands, which includes those of average citizens. I’m scared of machine guns mounted on drones controled by unsavory characters. But I don’t really mind them in the hands of the well-trained military of a liberal democracy for the defense of freedom.

    Me: There are two questions here:

    1. Given that the technology exists to make drones flying around with machine guns, what should our government do about it?

    2. Am I scared of drones flying around with machine guns.

    The answer to question 1 is complicated. I guess that our government has many options, including research and development on more effective drones flying around with machine guns; research and development on anti-drone weapons; and negotiating treaties to limit research, development, and deployment of these devices.

    As to question 2: Yeah, I’m scared of drones flying around with machine guns! Partly because the “bad guys” might use them in terrorism, and partly because “the good guys” might use them in terrorism. The U.S. has been a liberal democracy for a long time and that hasn’t stopped its military from terroristic acts from time to time.

    Anyway, yeah, it’s scary to me. Sometimes I see little drones flying around in the park. If I thought they might have machine guns on them . . . jeez!

    So my big fear is not so much the computers getting together to figure out how to wipe us out, but rather computers getting good enough that existing people who are willing to kill lots of people to achieve their ends will be able to do so much more effectively using computers, and then other people will do the same to protect themselves from the other people’s killer robots, etc. Drones with machine guns . . . that’s scary, and it kinda surprises me that LeCun doesn’t see them as scary.

    On the other hand, I may find a future of drones with machine guns to be scary, but I’m not putting in any effort to stop that future from happening. I’m not really sure what I could do to make it less likely to happen. So I guess that’s the point of the AI-safety research: to come up with some plan or political proposal that people could get behind and support. Such a proposal would then be opposed by people who could make money selling and deploying killer-robot technology, then there’d be political struggles, etc.—but that would still be better than doing nothing and letting the bad stuff just happen. Or at least that’s the idea, right?

    • “…the technology exists to make drones flying around with machine guns…”

      Great comment. When you tear it apart, it starts to look very unlikely.

      For starters I’m not sure the technology exists, much less if it did why it would be used. What would the recoil from a 50cal machine gun do to the flight path of a drone? Most larger drones are fixed wing, so they can’t just park over, say, a large crowd and fire continuously. Also, purely from the “destructive capacity” point of view, it would be kind of stupid to put a heavy machine gun on a drone with what little ammo can be carried on it when you can put a much more destructive missile on it. But then you need the capacity to build missiles.

      It’s clear from Ukraine that drones have destructive capacity as platforms for missiles or as missiles themselves, but the machine gun idea doesn’t seem have been utilized. I guess you could operate a military chopper remotely, so that would give you a functional machine gun platform, but it’s not so cheap as a drone, and makes a much bigger target for shooting down. And patriot missiles are doing pretty well shooting down the drones, so the cops have their answer to the criminals.

        • As for caliber of machine gun. There are 3 major calibers used by NATO: 5.56 used in stuff like the M249 Squad Automatic Weapon (a “light” machine gun), 7.62 used in the medium machine guns like the M240 and previously the M60, and the .50 caliber BMG round used in the M2 (“heavy” machine gun). There are recently some efforts to move to different rounds, including a 6.8 mm (.277 Fury) round that will be the new standard for US infantry rifles in an attempt to be more effective against body armor.

          Furthermore, there’s really no need for a drone to lay down automatic fire. It’d still be able to mechanically trigger semi-auto fire at probably a hundred rounds per minute or something (1 or 2 per second), and be much more on target if it aims each shot in the way that only a machine could. Basically, I expect drones to have belt fed light semi-auto weapons and something in 5.56 with a buffer system would be very possible to compensate for in the flight control software.

          There’s no question in my mind that after the Ukraine war we will see medium sized drones with mounted rifles of some sort.

        • “There’s no question in my mind that after the Ukraine war we will see medium sized drones with mounted rifles of some sort.”

          I doubt it! Talk about a waste of time and money. That’s why it hasn’t been done. There are much more effective technologies than rifles on drones.

          Machine guns have been mounted on planes for over 100 years and on helicopters since the 50s. It’s not the availability of flying machines with machine guns that’s the limiting factor. It’s the cost relative to the effect. $100 drones will never be able to carry 50-cal machine guns with ammo, but even if they could they never will because there are already more effective ways to use a $100 drone.

        • Chipmunk

          There have been lots of footage of ukrainian drones dropping what amount to hand grenades or similar in trenches. These are area affect weapons, they’re pretty effective, but the footage makes it clear that if a drone had a 7.62 rifle it could pick off individuals in trenches and they’d be essentially helpless, since in general they can barely see the drone if at all. It can fire from stabilized position 2000-3000 feet out. the whole thing is the size of a single human, so good luck to you countering it without specialized counter-drone technology (ie microwave cookers).

          It’s essentially a form of sniper, except one that doesn’t need 3 years of precision training, is cheap, easy to deploy, can fly at high speeds, and for which the loss of the drone doesn’t represent loss of life. We already have semi-automated sniper rifles… humans mark a target with a scope, the scope measures the distance precisely, sets automated elevation adjustment for the range, then the human just attempts to put the target in the scope and pulls the trigger. The trigger is “permission to fire” the rifle / scope actually decides when to actually fire based on waiting for the precise sight picture it wants. A person who has never fired a rifle in their lives can hit targets at 1000 yards with this equipment.

          Put something like that on a stabilized flying platform. If you’ve got 200 rounds of ammunition and average say 8 kills for every 10 shots, 4 or 5 drones could wipe out a few thousand entrenched soldiers for a few thousand dollars.

          A drone can carry explosives etc, but in general the explosives are way less precision and more likely you’ll get 5-20 hits from the kind of grenade level explosives you’d likely carry on a single drone.

  20. “Why couldn’t we just emphasize safety or something?”

    As I see it, the issue here is the extremely wide range of threats. Yes, it’s true that there are relatively mundane problems like violations of data privacy. Hopefully we can concentrate both on problems like that and on longer-range problems like extinction risk. In practice, of course, it may be hard to give both kinds of problem the attention they deserve; but surely it’s an ideal worth working towards.

    To me, highlighting extinction risk seems especially worthwhile for several reasons. First, it’s all too easy to get caught up in immediate problems and lose sight of longer-range risks. Second, the best time to prevent long-range risks is exactly while they remain some distance in the future. There may be any number of things we can do right now at small cost to avoid those risks down the road. Finally, the thing about extinction is that even if there’s a very small chance of it, it’s still something to worry about. In that way, it’s qualitatively different from just about any other problem you can imagine.

    I don’t think anyone is seriously arguing that ChatGPT4 poses an extinction risk. But in thirty or forty years, we may well have AIs that make ChatGPT4 look like DOS on a 1985 PC. All sorts of dangers may crop up that are extremely hard to foresee, which is why we should put real effort into foreseeing them as best we possibly can.

    If you’re skeptical that extinction is even a problem, consider a future where ever more real-world decision-making is delegated to AIs. Buildings get built where AIs recommend; AIs drive trucks to the chosen sites and control robots that actually perform construction. Your own car doesn’t just drive itself; it decides on the best route to your destination. Then the military concludes that autonomous drones are better than human controllers at choosing their own targets.

    If AIs show they’re good at making these sorts of decisions, there will inevitably be pressure to hand the decisions over to them. What ultimately lies down that path? I don’t know, and neither does anyone else. If in fact a monster is lurking there, we should be on the lookout for it so we can change course before it’s too late.

    • “If you’re skeptical that extinction is even a problem”

      Yes, I’m *extremely* skeptical, to the point I think the claim is nonsense.

      “consider a future where ever more real-world decision-making is delegated to AIs. ”

      Why not consider a future where humans aren’t dumb enough to delegate critical decisions to AI? Why create a situation where we’re pouring resources into mitigating a very very unlikely outcome, and at the same time foregoing the social and technical benefits that *would* come from embracing the technology?

  21. Here’s an explanation from one of the signers — https://www.schneier.com/blog/archives/2023/06/on-the-catastrophic-risk-of-ai.html

    Bruce Schneier, who is not and never has been a Chicken Little (he’s the one who came up with the term “security theater” to describe TSA operations): “I actually don’t think that AI poses a risk to human extinction. I think it poses a similar risk to pandemics and nuclear war—which is to say, a risk worth taking seriously, but not something to panic over. Which is what I thought the statement said.”

    • Thanks for the link.

      > Clearly I should have focused on the word “extinction,” and not the relative comparisons.

      > I should also learn not to sign on to group statements.

  22. What makes a man? Plato said it’s the arms and the legs. But I have to disagree. Isn’t it really our capability to predict the next word we will say based on the words we said previously?

  23. Dear naysayers,

    Consider how much critical infrastructure runs on boomer code that gets exploited by random humans every once in a while. Humans have limited resources, motivation, technique, speed, and so on. So what happens when an AI that’s read every book on every programming language, every book on computer security, and every guide to finding exploits puts its little transistors to work with an unceasing supercomputer efficiency on breaking X, Y, or, god forbid, Z?

    • Concerned:

      As with the robots with machine guns, my main concern is not that the computer will do this autonomously but that people will program computers to do this and then use this knowledge to kill people.

  24. My big fear is that some crazy dude will build a massive army of Mario and Luigi robots that will run around and relentlessly smash people with gigantic hammers! Humans will be helpless, as the AI/ML-driven Marios and Luigis will rapidly learn to counter any defense that humans can muster! The armies of Ms and Ls would drive humanity before them, herding them into the oceans to drown helplessly!!! The end of humanity!!

    Contact the President! We need to regulate all the parts that can build Mario and Luigi robots!

Leave a Reply

Your email address will not be published. Required fields are marked *