Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/components/Attention.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -139,8 +139,8 @@
</div>
{/each}
<Tooltip class="popover" triggeredBy={'.step.attention .out .cell'} placement="right"
>Attention Out, Head 1, vector({$modelMeta.dimension /
$modelMeta.attention_head_num})</Tooltip
>Attention Out before concatenation and projection, Head {$attentionHeadIdx + 1},
vector({$modelMeta.dimension / $modelMeta.attention_head_num})</Tooltip
>
</div>
</div>
Expand Down
11 changes: 6 additions & 5 deletions src/components/article/Article.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -328,11 +328,12 @@
<h4>Step 4: Output and Concatenation</h4>
<p>
The model uses the masked self-attention scores and multiplies them with the
<span class="v-color">Value</span> matrix to get the
<span class="purple-color">final output</span>
of the self-attention mechanism. GPT-2 has <code>12</code> self-attention heads, each capturing
different relationships between tokens. The outputs of these heads are concatenated and passed
through a linear projection.
<span class="v-color">Value</span> matrix to get each head's attention output. GPT-2 has
<code>12</code> self-attention heads, each capturing different relationships between tokens.
The outputs of these heads are concatenated into one vector and then passed through the
attention block's learned output projection matrix (<code>c_proj</code>) before the result
flows to the next layer. The projection matrix is part of the model computation, but it is
not drawn as a separate matrix in this visualization.
</p>
</div>
</div>
Expand Down
2 changes: 1 addition & 1 deletion src/utils/textbookPages.ts
Original file line number Diff line number Diff line change
Expand Up @@ -333,7 +333,7 @@ export const textPages: TextbookPage[] = [
id: 'output-concatenation',
title: 'Attention Output & Concatenation',
content:
'<p>Each head <span class="highlight">multiplies its <span class="purple">attention scores</span> with the <span class="green">Value</span> embeddings to produce its attention output</span>—a refined representation of each token after considering context.</p><p>GPT-2 (small) has 12 such outputs, which are concatenated to form a single vector of the original size (768 numbers).</p>',
'<p>Each head <span class="highlight">multiplies its <span class="purple">attention scores</span> with the <span class="green">Value</span> embeddings to produce a head-specific attention output</span>—a refined representation of each token after considering context.</p><p>GPT-2 (small) concatenates the 12 head outputs, then applies the attention block\'s learned output projection matrix (<code>c_proj</code>) before sending the result to the next layer. This projection is part of the computation even though it is not drawn as a separate matrix here.</p>',
on: function () {
this.timeoutId = setTimeout(
() => {
Expand Down