tubetranscript.io

NPTEL IIT Guwahati · Transcript · · Updated

Lec 25: Transformers - II – Transcript

Video by NPTEL IIT Guwahati · Watch the original on YouTube

Summary

AI-generated from the transcript. Verify details against the source video.

  • The speaker explains how alignment scores measure how well a query matches a key in transformer attention computations.
  • Masked self-attention ensures that each token can attend only to itself and tokens appearing before it in auto-regressive transformers.
  • Multi-head self-attention allows the transformer to simultaneously process different contextual viewpoints and relationships using multiple attention heads.
Video thumbnail: Lec 25: Transformers - II
Source video on YouTube · Transcript generated automatically and may contain errors.

Use this video with Pro

Reading, copying and transcript downloads are free. Create saved AI results for this video with a signed-in Pro account ($9/month).

Mindmap Pro · Sign in requiredTurn this transcript into a topic map you can review and download as PNG.Continue with this video →Flashcards Pro · Sign in requiredCreate questions and answers from this video to review and export.Continue with this video →X thread Pro · Sign in requiredCreate a source-based X thread draft to review before publishing.Continue with this video →Action items Pro · Sign in requiredFind explicit tasks and commitments mentioned in this video. Some videos have none.Continue with this video →

Transcript

Search ignores accents. All transcript text stays visible.

Hello and welcome to the lecture Transformers part two. In this lecture, we are going to first discuss how attention is computed in Transformer.

Then, we introduce the self-attention and how the self-attention computed in Transformer architecture. Then, two variant of self-attention.

One is must self-attention and multi-head self-attention being discussed. Now, we try to understand how attention is computed in Transformer for image captioning problem.

So, as you know that this image captioning problem is a a computer vision problem where input in the image and output is a sequence of words describing this image.

Now, what is the input here? You can see that input is basically the features. This features is extracted by a means of a CNN.

So, this is the features. Now, here we introduce another terms query. Initially, we in special context this model we have seen this H is using MLP we can get H.

For timing we can say this is a similar but to understand the attention computation for transformer we need to understand these three uh matrices.

One is query. Another is key and the value. Now, what is the basic thing? We'll come detail in the subsequent slide about query, key, and value.

But for the very initial understanding the query and key vectors are used to compute alignment scores in the self attention. Or for the timing attention.

While the value vectors are used to construct the final vector. Okay? So, this is this vectors are used to compute attention or self attention that we'll see later.

So, from where this query is? So, that H we say query now. It's of dimension D. So, it's in a dimension of D elements. Okay? So, that is our input to this particular uh when we are trying to compute the attention for this particular machine task.

So, having understand or having understood this particular background information, we try to find out that what do you mean by alignment score?

The next step in attention mechanism computation, the alignment score is being measured. Or alignment score being obtained.

So, what is the alignment score? The alignment score means that how well a query matches a key. Now, that is the very interesting thing here. That there is a white is query vectors. Query and key vectors are used to compute the alignment score in self-attention.

Now, these vectors, query, key, and value, are being obtained by using a separate fully connected linear transformation.

This applied to the input to generate this vectors. So, these vectors, this particular this input that being used to give different transformation matrix, one for key, value, key, and for value.

That we'll see in details later. But for time being, we are Now, having saying this, that we say the attention with alignment score is basically that how well a query matches with So, it indicates that importance and relevance of each key to the given query.

Okay. So, that is the basic alignment score. So, how alignment score is being uh computed that E of I G is equal to this if all this basically a neural net that H that is H and J of I J. So, this Okay? So, these two things will be taken

and we can find the alignment score. Okay? The details thing when will come to the actual computation when I understand the problem. But overall, this is the meaning of this alignment score this So, one important thing you have to understand here, the alignment score measure how well a query matches.

The next step is the computation of this attention. So, what we're going to do? We put a soft max layer over this alignment score. And you can see the the in the previous step, we generate E I G is equal to F of alignment this of H {comma} J of I J.

So, alignment is being obtained. And now this alignment is being given to a soft max to get this attention. values.

So, once the alignment score are calculated between a query and all the keys they are passed through a soft max function to convert them into the attention weights. So, attention weights are being computed in this manner. So, for the time being if we if we consider that this is nothing but a key after definitely a

FC layer is used to that and we can get this is the query and this is the alignment score. This alignment score basically measure how well this query and key vectors are matching. And then this alignment score put to the softmax and we get that attention. Afterwards.

So this is that way we get that attention weights. In this particular problem. So our next thing the context vector. But it's actually we try to find out.

Context vector is obtained by taking the weighted sum of the value vectors using the attention weights. Now you can see again this is coming. So one thing we see that H is generated from this.

This is again come here. And so what is important to understand, this is there is a key vector here.

This is query vector. And again this is coming here as a value vector. So same input are being as we said in a very first slide that the in transformation separate fully connected linear transformation that applied to the input to generate three distinct representation.

Query, key and value. So here you can see that first time it is with using a fully connected layer we can get H. This is the key and this is the value.

We'll definitely show exactly where but it is just a uh kind of hypothetical example, that's why we are showing in this one. Now it's a value vector here feature because we have not shown. These are features only.

But it is being transformed by a separate fully connected layer. Okay? So, first of all, the context vector is obtained by taking the weighted sum of the value vectors. These are the value vectors.

Frequently asked questions

Alignment scores measure how well a query matches a key and indicate the importance and relevance of each key to the given query.

Masked self-attention is commonly used for auto-regressive transformers to ensure that each token can attend only to itself and preceding tokens.

Multi-head self-attention performs the attention process simultaneously across multiple attention heads so the model can capture different types of relations and contextual information.