AI & SAP Basis Wizard · Transcript · · Updated
AI Explained: What Does the Number of Parameters in an LLM Mean? – Transcript
Video by AI & SAP Basis Wizard · Watch the original on YouTube
Summary
AI-generated from the transcript. Verify details against the source video.
- A neural network processes data using layers of interconnected neurons that perform simple multiplication and addition.
- Most modern large language models are built on a transformer architecture containing self-attention and feed-forward blocks.
- Comparing model parameters to biological brains shows that current language models may be reaching the complexity of a mouse brain.
Use this video with Pro
Reading, copying and transcript downloads are free. Create saved AI results for this video with a signed-in Pro account ($9/month).
Transcript
Search ignores accents. All transcript text stays visible.
You've probably heard about large language models with 7, 70, or over 600 billion parameters. Have you ever wondered what it means? We'll break it down step by step, and even see if we can compare it to living organisms. But first, let's dive into neural networks.
A neural network is a computational model inspired by the structure and functioning of the human brain. It consists of layers of interconnected neurons that process and transform input data to produce an output. A simple neural network consists of an input layer that receives data, such as numbers, images, or text.
Then, we have hidden layers that perform calculations, extracting patterns from the input data. Finally, we have an output layer that produces final predictions or classifications, usually as probabilities.
It sounds complicated, but it isn't. Data processing involves simple multiplication and addition. Each connection between neurons in successive layers is responsible for multiplication. The results of these multiplications are then summed up, and a bias is added to them.
Let's explain this with an example. For simplicity, we will use only natural numbers. This is a representation of the simplest neural network. It has two nodes in the input layer and one neuron in the output layer. Each connection visible in our diagram has an assigned weight.
It is used to multiply the input value to calculate neuron activation. In our example, each of the two nodes connects to a neuron in the output layer, so we end up with two weights. We'll assign these weights the values 1 and 3.
Let's assume that the first input value is 4, then after multiplying 4 and 1 we get 4. Let's set the second input value to 2, and multiply it by 3, which gives us 6. Then, by adding that 6 to our current output neuron value we end up with 10.
Each neuron also has a bias parameter that we add to the final result. So if the bias is 2, the final result is 12. Unfortunately, the final value is further processed through a so-called activation function, but for simplicity we will omit it in these considerations.
This neural network therefore has three parameters, two weights and one bias. Let's take a look at a slightly more complex neural network. This one has 5 neurons per layer, and only two layers.
This translates to 30 parameters. We have 25 connections between neurons, as well as 5 bias parameters. Together it makes 30. We've added one hidden layer, and now we have 60 parameters in total.
There are 25 weights between the input and the hidden layer, and another 25 weights between the hidden and the output layer. Plus, we have 10 bias parameters. There can be more hidden layers. In this example we've got 3 hidden layers, and a total of 120 parameters.
There are 25 parameters between each consecutive layer, plus an additional 20 bias parameters. This neural network already has 631 parameters. This one is already 2405 parameters.
And this here is only 14,526 parameters. Is that a lot? Well, it depends. A language model with 14 billion parameters is actually pretty average LLM, and that's a million times more.
Is it really that simple though? Is an LLM just a big neural network? Most of today's large language models are built on transformer architecture. The short scientific paper, Attention is All You Need, has truly revolutionized the world of AI.
And the world in general. In this architecture, a large language model is made up of many modules. But the most critical components are the self-attention and feed-forward blocks that appear repeatedly. When we add together the parameters from all the neural networks in a large language model, we get the total number of parameters for the entire model.
When we draw a comparison between a large language model and a brain, we can think of the number of parameters as being similar to the number of synapses in the brain. It's not a perfect comparison, but it helps us grasp the complexity of today's AI models.
A mouse brain is estimated to have around 500 billion synapses. We can say that the cat has over a trillion parameters. A dog can have up to 2.5 trillion of them. The number of synapses in the human brain is estimated to be between 100 and 500 trillion.
So we can say that the complexity of today's language models might be reaching the level of a mouse's brain. Doesn't look too impressive, does it? Then try teaching a mouse how to program in Python.