Designing & Building PR Review Multi Agent System (3 Hours Build) – Transcript

00:00Okay, today we are going to build an agent that reviews your pull requests. And here's the plan. What we will do, we will take the difference of your code which you have added or deleted. We'll give it to maybe OpenAI or Entropic API key with a completion prompt. And in the prompt, we will write, okay, tell me what's wrong with this code and hopefully expect some good comments out of that LL. And okay, okay, we can do something else as well. We can also add rag. We can pull some relevant files

00:25from the repository. We can stuff them into the nice little prompt and then we can call that production ready. Right? If that's what you came here for, please close this video. I'm seriously please close this video. There are 400 tutorials that do exactly like that. And none of them is the production ready agentic system. Here's what we will do instead. And frankly speaking, it does not starts with tech. It starts with a real senior reviewer who has 20 PRs to

00:51review today. And the quality on the first PR is not equals to the quality on the 20th one. And you might imagine some reason why. It might be fatigue. It might be inconsistency. It might be some human judgment. It might be some missed things in the 20th PR. Or it can be anything which relates to a human. And I want you to think about it. The problem is not to have an agent that can review your code automatically or in automated

01:17way. But instead the problem is the selectivity which is which findings does the agent takes and which ones does a human judgment actually get spent on. Now that one sentence of selectivity is going to be the whole architecture and everything will fall around it. Here's the part which I will also care about is I'm not going to give you a finished game or finished diagram of your architecture that this is how we're

01:44going to build it. I'm going to show you how I thought about this process and frankly speaking it's kind of a loop that runs once per component. What we do is first we'll find a human system that has already solved it which means watching how a senior actually reads a difference of the code which you added or deleted. Then we'll bring the codebased context which a senior reviewer actually thought about building in that particular moment. Then what we will do we will reason across separate

02:10concerns. For example, a reviewer might think about security. Maybe they think about quality, they think about correctness, they think about testing, they think about documentation. Every one of them is coming from a different mindset. And frankly speaking, they are very skeptical. They're providing evidence of the everything which they are talking about so that if at any point of time they get wrong, they can get back to it. And if you read those four statements from the engineering point of view, the codebase context means retriever. Separate concerns means

02:38it's not just one reasoner. It has multi- aent architecture. Every site which is having evidence which means every finding which my agent will make must carry a rational and a confidence and the four agents which will design will not come from okay let's design for agents but actually thinking from what eventually Howard human reviews and then we will take one component of entire architectures one by one and then we'll

03:04ask a single question to every component is what can go wrong and what we will do we'll think from two different perspective ive two different directions. The first one is the engineering direction where it all can fail. For example, let's say you're working on the component of GitHub which is GitHub retries the same delivery all the time. Maybe the security problem that somebody else is sending your request of PR which is not GitHub officially and then you have just 10 seconds to acknowledge to GitHub but

03:31your agent takes 90 seconds. All of that is engineering direction failure modes. In the LM direction we have the let's say the model hallucinates the retrieval pulls the wrong slices every single thing which our LLMs can go wrong and then what we will do for every component we'll put those failures into 2x2 matrix for example what you know you don't know what you will recognize when you see it and what you have never ever considered and we'll do that and the design of the

03:58component will start to write itself and then we'll move to the next one for example orchestrator retriever events gate states five components will run the same loop five times and by the end we'll be having an architecture which is fall tolerance we have designed already till there and only after designing the whole system we will code and frankly speaking we'll not vip code it like other people we'll run our genesis ritual genesis is in harnesses which

04:25actually give you an ability to maintain a state and to be able to consciously work with AI to be able to develop your project or to be able to implement your architecture for example Example, a context craft with your invariance. How does a definition of a done which looks like what each milestone will eventually contain and exactly what will the output of each milestones and then the loop will drive the agent against that spine.

04:49We'll be having five different gates to build that particular loop and an independent verifier that actually verifies whether it has done the work or not. Along with it, we'll integrate systems where it literally asks questions whether you have understood whatever it has written or not. And we'll truly show you how AI native engineering actually works when you actually be mindful about what your AI is doing. And by that point, you're not depending upon your agent to think and code. You're actually directing your

05:15agent and still having the full harness of how you're controlling the agent and what your agent is doing. You'll ship a real system. You'll have a multi- aents running in parallel. You'll be having a human approval queue, a full trace viewer, and a cost dashboard that we're actually going to show you the economics of your LM. But more than that, you will have the loop and the loop moves. We'll point it at a correct previewer and incident agent and anything of that kind. And more than that, you'll also

05:42learn how to natively work with AI and still be able to understand what your AI is doing at every single point of time. So there's a road map in the description. So I'll end this entire course not by building the entire system, but giving you a baseline to build on top of it so that you can actually extend this project and natively work with it. So let's get started with our system design. All right guys, so let's get started with our actually designing the system. So

06:08first of all before we even get towards telling you about hey this is how to think about this is how to write the design the idea of this project is to of course tell you that okay this is how you can design an AI PR review agent but more importantly I want you to be able to extend this to other problem I want you to be able to extend this to other ideas by your own and that's why I first of all first of all want to share you

06:33that this is the lens which is behind the way we think about you know uh this particular whole system EIP or agent which you can further generalize to any system which you're working on now so we usually have this template and which I have created and the first uh template and the first move is mapping the mess so whenever we will go ahead and when we

06:58design the system as well we want to document what actually happens today I mean if you I'm designing an agent which reviews a PR. I want to see that what if that particular AI agent is not there.

07:13What is happening right now? What is happening without that particular particular solution without that particular AI agent? Now let's assume why I do this because as I said that if you remove a component from it, if you remove that okay you want to review a PR. So instead of an AI which is reviewing the PR what you're doing you're having an human to review it.

07:34That's what you're doing right now. Given that we assume that there is no other AIP review agents uh in the world. So if you remove that component what is happening what you want to do you want to observe everything which a person is going to do. You want to write those micro decisions which a particular person is going to do. It can be literally anything. It can be okay these micro decisions for for example let's

07:59say I will take a very simple example in this case let's say you we we want to build this agent u a how a particular senior reviewer will go ahead will look at what code the microactions he will take the knowledge base he will recall the knowledge base he will see the architecture decision records he will see the documentation he will see etc etc so what is he doing at this

08:24particular point of time to review a particular PR and then you write every single minute steps of that particular person doing it because when you do that and then you'll figure out okay this is the mess which we need to organize in terms of agentic design and that's the first step there is no concrete step and I and and I always tell tell this to people is AI system design is just that there's a manual work going on you need

08:50to look at every possible steps which a human is doing even unconsciously and then you pick that up and then what you say okay fair enough now what I'm going to do I'm going to go ahead and figure out which is mechanical where human is doing his judgment and where we can you know put LLMs where we can put deterministic ML where we can put anything so we need to figure out that this is what we want to do so the first step which we're going to do in this project is figuring out what happens

09:17today right and more importantly I want to understand okay what triggers it basically if my if if what triggers a senior engineer or some some engineer to review your PR it's like okay as soon as the PR is made he will get some notifications on his GitHub or probably discord alert or slack alert or probably some person is going to say hey I made a PR can you review it and then he will make a comment and then so we want to

09:43write all those minute decisions and we're going to do that and we'll explicitly say okay this is where the human is required and this is the mechanical stuff is happening and we'll also figure out where it breaks. Again, I'm not yet on AI. I'm just on humans reviewing the PR and figuring out what it does, how it does, why it does, and where human fails. That's the first step. Second move, I want to make it more explicit to talk about what is

10:11going to my be my trigger and what is going to be my output. And more importantly, I want to just talk about one thing is if a person if a particular uh person wants to review a particular PR, what happens? He gets a Slack notification. He get a GitHub notification. He gets an email notification. He gets some some person saying, "Hey, can you review my PR? Can you u can you u you know tell me that okay, I have made a PR can you please go

10:38ahead and review it?" That's that's what the trigger which I want to give to my agent is I want my agent to be triggering its workflow at a very precise trigger and it should be a very precise for example the the the way I mean by precise is I want to take one example let's say that you're building some insurance approval agent which approves whether to give the insurance

11:02to this person or not right what happens is any email which comes in what will trigger the agent. We don't want that claims comes in. That's not a that's a very vague trigger. We want a very precise trigger. So in that case, I would be a claim which arrives at this particular email ID and has a word known has a keyword such as claims. Similarly in this we want that okay we want

11:28anytime a PR gets posted we should be able to create some GitHub web hook and use the GitHub web hook stuff to be able to get this on our Slack and that particular PR by that particular person probably can be one of our trigger now it's not just about having a trigger we want to output for example into the our insurance claim example we want to be able to say okay it should be able to produce some output right even in this

11:55what it will do it will say okay code eventually a structured PR review is posted the way a human would do is a structured PR will eventually get posted now let's assume like there are several types a person will eventually think about a particular PR or or maybe any GitHub PR which eventually gets made the first point is that it should be able to think in terms of okay because my PR is posted I should be able to test across

12:21the verticals whether it is readable whether it follows a security principles whether it is u you know that the the the the test cases has been added whether it uh whether it actually confirms and works based on architectural style there can be multiple parameters on this particular engineer might be testing the PR might be reviewing the PR and then he will make a comment and then the person will come the one who makes makes the code and then he will come to this engineer and say that hey this is and clarify if