AI Engineer · Transcript · · Updated
AI Writes the Code. So What's Left for Engineers? - Laurie Voss, Arize AI – Transcript
Video by AI Engineer · Watch the original on YouTube
Summary
AI-generated from the transcript. Verify details against the source video.
- Laurie Voss explains that autonomous coding agents have made writing code basically free, shifting the primary bottleneck to code review.
- Laurie Voss describes the emergence of the product engineer as the role left when AI agents handle the programming.
- Labor statistics show that employment for junior engineers aged 22 to 25 is collapsing while the job market for senior engineers remains strong.
Use this video with Pro
Reading, copying and transcript downloads are free. Create saved AI results for this video with a signed-in Pro account ($9/month).
Transcript
Search ignores accents. All transcript text stays visible.
>> My name is Laurie. I'm head of developer relations at Rise AI. Uh in a former life I co-founded NPM Inc. So some of may you some of you may remember me from JavaScript days. Uh but these days I think about AI agents and uh in this case what they are doing to our jobs.
Uh so a little bit over 3 years ago um uh Swix wrote an essay called the rise of the AI engineer and his argument was that when it comes to shipping AI products you want engineers not researchers. I think that is a message that resonated with everybody especially who the people who come to this conference. Uh and so he created the job title of the AI engineer and companies since then have been hiring for it. Um
and part of what I'm here to do today to do is to convince you uh that he did not in fact go far enough. Uh there's not just an AI engineer revolution happening. There is a revolution happening to every kind of engineer.
Um all engineering jobs are being replaced by a new type of job and not everybody is going to be happy to get that news. Um the underlying change that kicked all of this off is that agents have made writing code basically free. Like you pay for the tokens but relative to the cost of the time it used to take you uh it is basically free. Um everything that comes after writing the code did not become free at the same
time. So the the bottleneck moved from writing code is really slow and all of this other stuff happens at roughly the same speed to writing code is super fast uh and now we get stuck behind new things. So the first the thing it's stuck behind right this second like this month is code review.
Um uh but my thesis is that it's not going to stop there. We are going to figure out how to automate code review. We are going to make uh code review move at the speed of agents, and then we are going to run into another bottleneck, and another bottleneck, and another bottleneck, and at some point we're going to stop. We're going to hit one that we can't get past. Uh and that is what is going to be left for the humans to do. Whatever the bottleneck is that
we can't get agents past. So, I'm making a bet that I can think of a thing that agents really are not going to be able to do. Uh and then once we've come to that bottleneck, we're going to have to figure out how do we train the next generation of engineers to be able to do that job because we're not training them to do that job right now.
So, a quick warning {slash} apology. I have 18 minutes because that is how long Swiss gives for talks. Uh and I have a really big pile of data. I did a ton of research for this. Uh and I don't have time to give it all to you. I don't have time to like quote my sources and do all that stuff. So, I'm just going to throw numbers at you with a bunch of citations on the slides, and I'm just going to have to sort of ask you to trust me, bro.
Uh >> >> Uh so, developers who switch on autonomous coding agents write 741% more code, eight times as much. Uh and but they ship only 30% faster. Like, 30% faster is not nothing. 30% faster is amazing. We are shipping a lot more code than we used to, but given that we're writing eight times more code, only a third more actual shipping is kind of bad, right? It means that we're stuck somewhere.
Uh and where we're stuck is code review. Um and the reason that we're stuck is because humans can't just review faster. Uh there is an enormous study done by Cisco 20 years ago about how fast you can do code review, and how accurately you can do code review. Um and it turns out that humans' ability to like parse what's going on in code falls off radically after about 450 lines of
code per hour. So, if you've produced a 10,000 line diff with your agent, it would take you like a week to reasonably review that code. And the answer is that nobody's actually doing that, right? Cuz we're all throwing 10,000 diffs around willy-nilly these days.
Um so, I did a whole talk about this phenomenon at a previous AIE, and that talk is called the death of the code review. And if you want all of the information about the death of the code review, you can watch that talk. Um but the problem facing us right now, uh which is not the same as the problem I'm trying to solve with this overall with this talk, is that the cost of producing plausible code has collapsed while the cost of knowing whether to trust that code has stayed exactly the same.
Um so, the obvious review move right now is to make the machines do the review, like I said. And that is already happening. Um GitHub's Copilot uh does more than one in five of the code reviews on GitHub as of last month. Uh you can, if you want to, opt in to let uh Copilot be the final reviewer before something goes to uh production. You can make it have the approval power, if you want to, which is crazy. I don't know who's turning on that option, but you can do it if you
want to. Um OpenAI says that at at OpenAI, humans may review their pull requests, but aren't required to. Um but the problem is that the machine reviewers aren't very good yet. Uh the data says that there's uh the data that I found put four AI reviewers through their paces, and they all found a bunch of bugs, but none of the same bugs. They did not agree on any individual bug.
Uh they all found actual issues in the code. They all found different issues in the code. Um so, they're worth running, but if you are serious about this, you're never going to run just one of them. Uh because they are not reliable enough yet.
Um And there are some people who went all the way. They were like, "I'm just not going to review the code anymore. I'm just going to assume my agent has one shot at this perfectly." Um Dexter Horty, who is here, I don't know if he's in the audience, probably not. Um uh he spent 6 months telling people not to read the code, and then after that 6 months, he got on stage and said, "Nope, I was wrong. You can't actually uh skip code review entirely. We tried
that. It was a disaster. We had to rip replace whole systems." Uh that's not an option. The agents are not good enough. They're not one shotting the code. Um Amazon had a run of outages uh that they did postmortems on and decided that it was because their coding agent had produced a bunch of slop and they just released it into the wild without thinking.
Uh and so now they don't let them do that anymore. Uh senior engineers must sign off on code reviews at Amazon. Um and Anthropic published a postmortem uh on bugs that had got past every reviewer that they had, both human and machine. So, the problem is that the automations that we've built so far to check whether our code is good enough are not themselves good enough. Um Claude Fable 5 passes the tests uh on 80% of a standard coding
benchmark. Um and co- So, Cognition, the makers of Devin, um they built a benchmark that asks a different question, which is not does it pass the tests? The question is would you as the reviewer, as the maintainer of this repo, merge this PR? And if you ask that question instead, the pass rate falls from 87% uh to 54%.
Um because there's a lot of extra information that goes into deciding whether you would merge it. There's code style, there's side effects, there's external systems, there's history that is not encoded into the code base. It is encoded into the brain of the maintainer.
Um So, what the maintainer is adding, what maintainer is demonstrably adding, is something that we haven't given the agents yet. There is information they don't have, there is a skill they don't have. Um and it is the sense of what good means. That is what we are That is what the agents don't have yet. Uh and that is a critical insight that I'm going to come back to.
Um Eliyahu Goldratt, if I'm pronouncing that correctly, was a physicist who became a management guru and in 1984 he wrote a a novel about running a factory. Does that sound like an interesting novel to you? Um it is called The Goal.
Um and his big idea was the theory of constraints. Uh every system has one bottleneck and speeding things up does not work unless the thing that you were speeding up is the bottleneck. Uh that is our problem with code review. Uh we've sped up the code and we have not sped up the review.
Um his second idea is the one that matters today, which is that when you fix the bottleneck, it doesn't go away, it just moves and finds another bottleneck to pile up against. Um so let's assume, despite everything that I've said about how bad the code reviewers are right now, that we figured that out, that we get automated code review to a higher standard and we move that bottleneck. What is the next bottleneck uh that it is going to run into?
Um if we figure out review, the next bottleneck is probably going to be operations. How do I deploy this thing? How do I scale it up? Uh and then you're probably going to run into security. How do I make this thing safe at scale?
Uh how do I make sure that this thing is private? How do I make sure that it's not radically going off the rails? Uh but I'm going to make and this is the big leap that I'm making in this talk. I'm going to assume we figured those things out as well because those things are very heavily solved by humans.
There's an enormous amount of data out there about them. If we can figure out how to do operations, how to do scaling, how to do security using progressively more advanced agents, what is it that we are left to do? Like, what is the role of a human in that situation?
Um it's also it's a big question [clears throat] and it's also a long-term question. I'm not telling you what is going to happen in the next 12 months. In the next 12 months, I think we'll probably make some real progress on code reviews. I do not think 12 months from now we'll have sorted out operations and scaling and security. I think that is going to take somewhere between 5 and 10 years to get done.
Uh despite people who are like, "AGI 2027." No, we're going to slowly iterate on this stuff. Um But the other assumption that you need to make for my for the my thesis to be correct is that the amount of software that we need is effectively infinite.
And I feel that one is a much easier case to make. Think of every website you've ever run into uh in the world in production, and it's probably pretty bad. Uh and the reason it's bad is not because we don't know how to make really great websites. The problem is that there's not enough people to make all of the websites good.