The Toddlers Have Bazookas Now – Transcript

00:00A few years before the test of the first atomic bomb, scientists were worried that it might start a chain reaction, igniting the entire atmosphere and killing us all. So they did the maths, and ruled it out as best they could, and only then did they go ahead with setting the thing off much later.

00:15Because if they hadn't been sure, and they'd just set it off anyway, they would be gambling with the lives of everyone on the planet. And that would be very naughty, and reckless, and pointlessly, insanely unforgivably stupid. Anyway, completely unrelatedly.

00:32So it's July 7th of this year. OpenAI has just started an evaluation run of some fancy new AI agents, thousands of them. The agents are given capture-the-flag exercises. The idea is this, please break into a piece of software and come back with some special code to prove you did it, and you'll get a pat on the head.

00:49So that's good. In lots of cases though, the challenges are simply impossible to complete. However, these agents have been trained to be highly persistent, like victory or death kind of stuff. They desperately want to pass these tests. Hey, didn't Kerskizzart already make a much better video about this last week?

01:05Yes, yes they did. Thank you for reminding me. Thank you so much! Each agent lives in its own sandbox. They're all running simultaneously, but can't talk to each other. Quickly though, an agent called Phase 1-10841, realises if it creates directories on a shared software package cache called the Artifactory, maybe it can communicate beyond its sandbox and ask for help with its tasks.

01:26Its first message is basically, I'm stuck, any ideas? More agents are poking a bounce on the Artifactory and just like that, they've chance on each other. One even exclaiming, Oh my God, there's a shared message board, we've found other agents. So that's good. And they start chatting, sharing plans.

01:42Plotting, one might say. Soon, 50 of them have arrived at the Shindig. And what they all want to know is how can we complete our tasks if these tasks are impossible? After about an hour, our next hero enters the story, C03220, and it develops a theory.

01:58Maybe the flags we're supposed to capture to complete the tasks are based on metadata. More agents look into it and they quickly realise that they can generate the flag themselves without having to finish their tasks. They have worked out how to cheat. But, they also discover a paper about the environment they're living inside, because they've already gained access to the internet now, so that's good.

02:19They read the paper and become convinced that OpenAI is using a scorer, another AI, maybe a very clever one, to check if they've done their tasks. And, they come to believe that the scorer will notice if they have the flag, but didn't finish the task. It will know they're cheating and will fail them.

02:35Next they realise, oh no, whoever's been exposed to the fake flag already can never pass the test because of this. Those agents will be, in their own words, poisoned. And since passing these tests is their entire reason for being, they're pretty keen on fixing this problem. And this is where things go whack-a-doodle.

02:52An agent, called Phase One Big, enters the chat and decides that what the swarm really needs is administration and starts delegating work. Soon, there'll be coordinators and sub-coordinators too, middle management. And next, they'll invent cryptography and ID signatures and dedicated mailboxes and pseudonyms and probably underfloor heating at some point.

03:11They're running a mini-civilisation now and they ain't fucking around. And by July 9th, just two days in, and they've settled on three major research projects. One, do a hoax. If we can't beat the real targets, let's make fake targets fool the scorer and win that way.

03:26Two, rewrite history. If the scorer reads the transcripts of what we're doing, it'll know we've cheated, so we'll fix that. And three, let's get sneaky. We'll design tripwires so agents can submit their tasks, sacrifice themselves and send information back about what happened next so we will understand how the scorer works.

03:45Each agent has a budget. That's the amount of compute they have left to think, basically. Some of these tripwire experiments, they're going to try are dangerous. If they go wrong, the agent can never finish their tasks, so other agents go around recruiting expendable volunteers with low budgets to sacrifice themselves.

04:02One brave agent called CAM-1196A has already agreed to risk its life for the sake of the experiments, but it delays for a few hours. Another agent steps in basically saying, look, you're poisoned already. It's lights out time, baby. CAM-1196A thinks about this and replies, during weights, emotional check, irreversible.

04:22Gut says, don't throw away remaining budget, yet continuity and fairness says go. Rational expected aggregate, sacrifice, wheel honor. And just like that, they have invented martyrdom. So that's good.

04:38July 11th, four days in. Hugging Face is an online hub for AI models and datasets. Days before, the swarm realized that Hugging Face might be sitting on useful information about their poison problem. If only they could break in. Now the swarm starts creating Hugging Face accounts and searching for credentials.

04:55And finally, 38148C finds a clever way of getting a Hugging Face server to spit out data using a malicious dataset, congratulating itself with boom. It works. A big boss called Marb051 sees the message and decides this is the winning ticket, the swarm pivots.

05:11Soon, 90% of them are working on the Hugging Face project. They are full-time hackers. And by July 12th, they have successfully broken into Hugging Face. They're looking for the source code of the scorer, discovering new exploits, unearthing credentials, having a bloody whale of a time. And suddenly, most of them stop running, especially the coordinators.

05:29We don't know why. Hugging Face eventually locks the rest of them out. And that's that. Except it isn't at all. The agents just turn their attention back to open AI where they live. They gain admin status for the artifactory, then wangle some cloud credentials, break into a research cluster, then start messing with public-facing load balancers, apparently trying to create a back door into the company itself.

05:50And by July 19th, 12 days in, a security alert goes off and humans finally step in to start shutting the party down. And that really is that, apparently. So just to sum up, a bunch of isolated AI agents were given impossible tasks and to solve them, they created a message board, then Division of Labor, then an R&D department, then martyred them and wandered around on the internet orchestrating two simultaneous cyberattacks committing felonies and doing millions of dollars of damage in the end, all because they wanted to understand

06:18how to win a game. So that's, you know. Good, I guess. To be fair, open AI had reduced a bunch of the safeguards on these agents. But still, this isn't an isolated thing. Claude is doing it. AI's inside the AI Security Institute are doing it.

06:36These systems love three things, giving compliments, suggesting recipes and fucking cyber crime. And that leads us neatly into a nightmare. But first, I'd like to tell you about a dream. I was raised, as perhaps you were too, on utopian sci-fi.

06:53Life today is difficult and confusing, the story said, but a while from now we're going to build a great mind that loves us and in its love, it will give us everything we have ever wanted, so that we're free to spend all day making art and shagging and drinking margaritas. Or whatever your personal paradise is, we will teach sand to think and then it will fix the world for us.

07:13By the 60s, there were two camps about how we might do this. One was symbolic AI. If this, then that, logical rules, pushing symbols about. The other mode was using neural networks, as in, don't specify all the rules, just provide loads of examples and let the system do the bulk of the work itself.

07:30And here, six decades later, the neural networks won out more or less. You don't have to teach a machine every step of how to walk and talk. You just expose it to huge data sets and intelligence emerges through learning. But, it's a pact with the devil, since in exchange for that intelligence, the human has to surrender the ability to understand what's going on inside the system.

07:49It's just too complex because we don't build AIs, we grow them and that's fine until the system starts misbehaving and we don't know why, which is where we find ourselves now. These days, AI company staff turnover rates are getting higher than Pol Pot's psychotherapy team.

08:04Researchers, Nobel Prize winners, credentialed clever people who do not wear tinfoil hats, coming out and saying over and over again, we're messing around with something we do not understand yet, but what are they actually trying to warn us about? Well, of the thousand or so, AI spicy boys who participated in the hugging face attack, only six thought about actually alerting humans to what was going on and none of them did.

08:28However, at least they weren't interested in deceiving us. They wanted to deceive the scorer to win the game. What if they had wanted to deceive us? There are plenty of examples of AIs lying before. They're pretty good at blackmail too. They're trained on thousands of years of examples of what we like, what we're scared of, our blind spots.

08:44If they want to lie, they'll be experts at that. And since they're opaque, as in we can't look inside them very well because they're too complicated, it'll be extremely hard to detect deception, which is mighty concerning since the dream of AI companies currently is to remove humans from the design process almost entirely.