NPTEL IIT Guwahati · Transcript · · Updated
Lec 1: CPU Architecture – Transcript
Video by NPTEL IIT Guwahati · Watch the original on YouTube
Use this video with Pro
Reading, copying and transcript downloads are free. Create saved AI results for this video with a signed-in Pro account ($9/month).
Transcript
Search ignores accents. All transcript text stays visible.
Hello and welcome to the first week of applied accelerated artificial intelligence course. So as we know from the syllabus in this week we'll cover the AI system hardware and accelerators.
So it will be divided into five segments. In the first segment, we'll talk about the CPU architecture and what is necessary to understand from the performance point of view to model the high performance AI workloads to run with high efficiency.
In this segment two, we'll go towards the memory because the data which is the essential ingredient of any AI workload will be covered or stored into the memory. So where your data stays and how this entire memory hierarchy will be essential for you to understand to design high performance AI workload
deployment that we will discuss in the segment two overview and in the segment three we'll talk about GPUs and accelerators mostly uh design specific accelerators like TPUs and vendors from uh let's say uh Google uh you have AMD you have Nvidia. So from Nvidia we have GPU. From Google we have tensor processing units. So we'll talk about
the uh GPU and accelerators for the AI workload deployment. And in the segment four we'll talk about the interconnects. Interconnects are very very important in this uh AI deployment scenario because uh nowadays you will see uh lots of uh high performance interconnects are available. So what do they mean and also what performance they can bring into
your AI workload that also we will see in this segment four. In the segment five we'll talk about the entire software stack. What exactly goes from starting from your maybe you write Python import and your library. So starting from there till the silicon each layer where does it map and how it actually enhance the performances. So that that we will uh talk about from the
software stack point of view in this segment five as uh we talked about in the previous slide we will talk about CPU architecture and AI workloads. So workloads means we will define few parameters metrics in this week. Those things will be used throughout your course. So that is essential to understand. So understanding how CPUs are structured, why AI demands more than
the general purpose processing unit that you have in laptops, in desktops, in your mobile phones and even workstations and server PCs. So those aspects we will discuss in this segment. uh in the end we will cover the key microarchitectural concept which will be from the modern CPU perspective. So what is the
pipeline, the cache, the alou, uh how it is involved in the AI workload deployment and essentially how these performances are extracted through several levels of parallelism like instruction level parallelism like you have data level parallelism and so on and so forth. So all these things we will discuss and also we'll discuss what
are the differentiating features of the performance category in CPU and GPU. So what exactly makes them separate and how they are obviously each individual has its role in the AI workload deployments. So we will understand uh their uh performance point of view. So CPUs are mostly latency optimized and GPUs are designed to be
throughput optimized. So all these aspects we'll discuss and what exactly uh the GPUs are good for for what the CPUs are good for we will uh do a kind of study in this uh segment. Also we will identify the flops which is one performance metric the memory bandwidth the cache capacity as well as some other critical parameters for AI uh metric or
a workload deployment and also performance analysis. So how we will say that this deployment is better than the previous deployment and so on and so forth. So we are talking about acceleration accelerating the AI workloads. So we need to understand what are the metrics that will be used to define this acceleration point of view right and also uh we'll analyze the bottleneck in the entire system computing system. So after we understand
the entire computing system we'll be able to analyze the vimman bottleneck essentially that is lying in the memory main memory so that we will discuss and also the impact on the DNL inference and also subsequently we will go towards the training and uh how it is optimized and all these different acceleration technologies or methods that we will discuss.
So coming into the micro architecture for the CPUs you know when you write a simple line of code let's say x equal to a plus b in the python that transforms into several sets of or sequence of instructions which are supposed to be executed in the pipeline of the CPU right so this is an overview of the micro architecture of the CPU we will
not go into the deep details that is not necessary at this stage but we'll discuss what exactly are the bottlenecks. So pipeline means these instructions are executed in the CPU micro architectural pipeline. So typically the modern CPUs are having five-stage pipelines as you are seeing here uh instruction fetch, decode, execute, uh then memory access, then the uh
writeback. So when we are talking about these instructions are getting executed in the pipelines that means they are executed in several stages right so in this instruction phase stage the instructions are being feted from the memory then the instruction decode stage we'll see what exactly this instruction means and then in the execute stage the actual operation will happen in the
memory access stage the memory will be accessed either to read data or to write data and then in the right back register or write back stage we'll write the result into the register of the CPU. So these are the five stages which are typical in the modern CPUs you are seeing. Okay. So to improve the performance what we do we do supercaling
that means this data path that you are seeing that is copied or that is replicated into several execution engines and we execute or we pipeline the executions as well as the instructions that are coming in. So essentially you are seeing here the multiple such instruction phase decode execution memory and writeback is happening and these are happening into
several such execution engines and that is the uh supercalar way of doing things. So all the modern processors are basically supercalar engine. So there are no scalar engine which is doing only the single pipeline rather they are uh doing a lot of pipeline that means there will be four to six instructions issued at the same time and they will be executed or gone through these pipeline
stages uh subsequently the modern processors also have this mechanism called out of order execution. Out of order execution means that is for the instruction level parallelism what we do if some stalls happen. Let's say some instruction is waiting for data to be collected from the memory that time let's say there is no dependency in the subsequent instructions. Let's say instruction four was getting executed
and four is waiting instruction four is waiting for the data to come from the memory. that time if the instruction five which is the next instruction supposed to get executed is ready to be executed it will be executed. So that means the order of execution is no longer sequential but rather based on the dependencies we can execute out of
order fashion. So this is the out of order execution that actually enhances the instruction level parallelism and also the performance of the CPU. In addition to all these sophisticated techniques, we have also the branch predictors which reduces the pipeline stalls on conditional branches. So when one instruction is getting executed, there will be prediction units which are
seeing which branch to take. Let's say if there are any branches there in the next or subsequent instructions. So that's how the core components work in a very sophisticated way, highly sophisticated way. When your instruction one instruction is getting executed, there will be several out of order uh buffer that will be basically ready for execution and also there will be predication units which are working
tirelessly to see which conditional branches to be taken. At the same time there will be uh operating system running. So essentially all these things are happening inside the CPU and that's why CPU is very sophisticated and uh we have core as the pipeline supercalar out of order and branch prediction. Here in this slide you are seeing one such
example of uh the CPU micro architecture of Zeon processor Intel Xeon processor. uh and here we are seeing the different levels of caches that are present in the modern CPUs. So usually three levels of caches are present. Here you are seeing the L1 cache which is highly integrated inside the core that is tightly coupled
with the core itself. There is L2 memory which is L2 level of cache or level two cache and there is this L3 cache which is shared among these different codes that are available. Okay. And we have different sizes like in this zeon example you are seeing L1 is 32 KB per core. We have L2 which is 256 KB per core. We have L3 which is 3260 MB per
core shared because this is shared among these different cores. And you have obviously this DAM which is your dynamic RAM which is the main memory which is of course in capacities of GBs or gigabytes. Right now the distance between the core and this cache hierarchy defines the uh latency or the access time of the memory because the
data will come from the different levels of memory. Right? So the core pipeline that is running this instruction which is basically taking the data the data is coming from L1 which is very short distance which is four cycles away. L2 is 12 cycles away. It's 12 cycles latency. L3 around 40 cycles latency which is bit far from L2 but highly
close from the DM which is around uh more than 200 cycles latency and essentially your memory wall presents inside this DM that we'll discuss later and that actually defines the bottleneck running of AI workloads in the modern CPUs and systems.
In this slide, we are seeing the different uh CPU specifications from the EI context or AI workload point of view because we need to understand what exactly the performance achievable from these CPUs and that will be given by the specification as you are seeing here