Theoretical Foundations of Language Modelling

Almost every state-of-the-art language model is a decoder-only transformer – a neural network design based on the architecture described in "Attention Is All You Need" [1Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 30). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf]. In this chapter, we will focus on text, as it is the easiest modality to follow. However, please keep in mind that nothing in the architecture is specific to text.

Neural networks

On the map of AI from the introduction, the transformer sits on the subsymbolic side. It is a neural network that autoregressively generates probability distributions (Figure 1).

To follow this chapter, you should therefore have an understanding of how a neural network works. If you are not yet familiar with neural networks, we suggest watching this short introduction by Grant Sanderson of 3Blue1Brown [2Sanderson, G. (2017, October 5). But what is a neural network? | Deep learning chapter 1. 3Blue1Brown, YouTube. https://www.youtube.com/watch?v=aircAruvnKk].

Conceptual components

In order to perform autoregressive sequence modelling, a model needs to grasp the following concepts:

• What each element of the given sequence means on its own. cow is an animal; bank is money, or a river.
• How the elements relate. In "the cow says ___", cow shapes what says means.
• Where each element stands. "The cow licks the calf" is not "the calf licks the cow".
• What the world knows. "The Eiffel Tower is in ___": no word in the text says Paris.
• Given the understanding of the sequence: how to predict what comes next. A probability for every token of the vocabulary.

Embeddings: What each element of the given sequence means on its own

As we learned in the previous chapter, the tokenizer parses raw text into discrete tokens. Each token has an integer ID, which acts as a lookup index in the embedding table, a learned matrix with one row per token ID (Figure 2). Word2vec showed that, during training, the positions of such vectors change so that similar words end up close together, and some directions come to represent natural relations [3Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. In Y. Bengio & Y. LeCun (Eds.), 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings. http://arxiv.org/abs/1301.3781]; the embedding tables of today's models behave the same way (Figure 3). bank has one row, for money and river alike; which one is meant is left to the next step, attention.

The cow mooed

↓ tokenizer

The 785

·cow 19362

·moo 45871

ed 291

↓ each ID picks its row of the embedding table

Embedding table

0 ⋯ 1 ⋯ 2 ⋯ ⋮ 291 ⋯ →ed⋮ 785 ⋯ →The⋮ 19361 ⋯ 19362 ⋯ →·cow19363 ⋯ ⋮ 45871 ⋯ →·moo⋮ 151935 ⋯

151,936 rows, one per token, of 1,024 numbers

The text, as vectors

The ⋯·cow ⋯·moo ⋯ed ⋯

4 × 1,024, a row per token

Attention: How the elements relate

When thinking about a cow and a small purple cow, one rarely imagines the same cow. It seems that the context in which a token occurs changes its meaning, even though the token itself is unchanged. cow alone could be any cow. In a sentence, every word in front of it sharpens the understanding of what the token cow refers to in its context: "the cow", "the small cow", "the small purple cow" (Figure 4). By the end, the cow in question is a long way from an ordinary Holstein and close to the one on a bar of Milka chocolate.

(a) cow

(b) small cow

(c) small purple cow

The goal of the so-called attention mechanism is to compute how much, and how, each token should shift the vector representation of every other token in a given sequence. These shifts thereby capture the contextualisation.

Each token in our example sequence moves cow's vector; the figure shows the shifts one at a time, but attention adds them up in a single step (Figure 6). Using the same tokens in front of the token flower moves it in the same directions but by different amounts. Intuitively, we might say that the colour purple changes the meaning of flower less than the meaning of cow. The token purple is the same in both cases, but purple tells us more about a cow than about a flower. This raises two questions: How can we quantify how a token shifts the meaning of another? And how strongly should each token shift another?

How much: queries and keys

The dot product is a common tool for quantifying how vectors relate to each other, as it measures how well they align: $a \cdot b = \lVert a \rVert \lVert b \rVert \cos \varphi$. It is largest when they point the same way, 0 at right angles, and negative when they point apart. On embeddings themselves, it measures similarity; however, similarity is the wrong question. purple and spotted both describe how a thing looks, so they point nearly the same way and are about equally (un-)similar to cow. Yet purple changes a model's understanding of a cow a lot, and spotted hardly at all, as spotted is what we already expect of a cow. Therefore, we need different representations of the original tokens that better capture the magnitude of influence. Attention first projects every vector into a second space, through two learned matrices. $W^Q$ turns a token into what is known as its query, what it looks for; $W^K$ turns it into its key, what it offers. There, what cow looks for lines up with what purple offers, and spotted's key points elsewhere (Figure 7).

(a) Meaning, the embeddings

(b) $\times W^Q$, $\times W^K$, learned

How: values

The influence of one token on another could simply be its embedding, $x_\text{cow} + x_\text{purple}$ [4He, B., & Hofmann, T. (2024). Simplifying Transformer Blocks. International Conference on Learning Representations (ICLR).]. Most transformers nevertheless learn a third matrix, which computes the so-called values $v_\text{purple} = x_\text{purple} W^V$ (Figure 8). Queries and keys quantify how much a token $a$ should shift a token $b$; the values decide how $a$ shifts $b$ [5Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., & others. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread. https://transformer-circuits.pub/2021/framework/index.html].

In matrices

Typical implementations of the attention mechanism handle all tokens at once. Their embeddings are the rows of $X$, and three matrix products give every token its query, key and value in parallel (Figure 9).

query

d_model

X

×

d_k

d_model

W^Q

=

d_k

n

Q

key

X

×

W^K

=

K

value

X

×

d_v

W^V

=

d_v

V

$Q K^\top$ compares every query with every key in one product. Row $i$ of $Q$ and column $j$ of $K^\top$ meet in the score of how much token $j$ matters to token $i$ (Figure 10).

d_k

n

Q

×

n

d_k

K^⊤

=

n

n

scores

Attention scores and weights

The dot product of each query with each key gives the attention scores. In principle, the scores can be any real numbers. A token's attention, however, should be split into shares that add up to 1 – no token can spend more than 100 % of its attention. Softmax turns a row of scores $s$ into such shares. Every share is positive, they add up to 1, and a higher score always gets a larger share:

$$\mathrm{softmax}(s)_j = \frac{e^{s_j}}{\sum_i e^{s_i}}$$

A score 1 higher gets $e \approx 2.7$ times the share, and one 10 higher gets 22,000 times. As dot products spread further apart the more dimensions they have (by about $\sqrt{d_k}$, where $d_k$ is the length of a query and a key), one token can easily receive a share of nearly 1 and the rest almost 0. Dividing by $\sqrt{d_k}$, here $\sqrt{64} = 8$, keeps their order and brings the values back into a range softmax can work with (Figure 11).

1. ·purple 100 %
2. ·cow 0.3 %
3. ·small 0.0 %
4. The 0.0 %

(a) softmax of the scores

1. ·purple 60 %
2. ·cow 28 %
3. ·small 10 %
4. The 1.1 %

(b) softmax of the scores $\div \sqrt{64}$

These shares are the attention weights $\alpha_{ij}$.

Output

Each token's new vector after the attention mechanism is the weighted sum of the values, $\sum_j \alpha_{ij} v_j$: every value counts as much as its weight says. For all tokens at once, that is one matrix product (Figure 12):

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{Q K^\top}{\sqrt{d_k}}\right) V$$

softmax(

n

scores

÷ √d_k )×

d_v

V

=

d_v

new vectors

In code it is a few lines:

import mathimport torchfrom torch import nn​d_model, d_k = 512, 64n_tokens = 4x = torch.randn(n_tokens, d_model)        # a row per token​W_Q = nn.Linear(d_model, d_k, bias=False)W_K = nn.Linear(d_model, d_k, bias=False)W_V = nn.Linear(d_model, d_k, bias=False)  # d_v = d_k here​Q, K, V = W_Q(x), W_K(x), W_V(x)         # [n, d_k] each​scores = Q @ K.T / math.sqrt(d_k)         # [n, n]: every query against every keyweights = scores.softmax(dim=-1)          # each row sums to 1out = weights @ V                         # [n, d_k]: weighted sums of the values

Masked attention

The chain rule from the introduction writes a text's probability as a product of conditional probabilities, each of one token given the ones before it:

$$p(x_1, \ldots, x_n) = \prod_{t=1}^{n} p(x_t \mid x_1, \ldots, x_{t-1})$$

In training, the desired text exists in its totality. Without a stop, the model could simply look at the token it is asked to predict. A mask stops that. It adds $-\infty$ to every score above the diagonal, where a token would look ahead. Softmax turns $e^{-\infty}$ into 0, so each token's weights fall on itself and the tokens before it (Figure 13). This is also called causal attention, and the mask the causal or look-ahead mask. It also lets training check all $n$ predictions of a text in one pass.

scores

+

0−∞−∞−∞−∞−∞00−∞−∞−∞−∞000−∞−∞−∞0000−∞−∞00000−∞000000

mask

→ softmax →

weights

Multi-head attention

Conceptually, one set of queries and keys asks one kind of question, such as which adjective describes this noun. A text holds many relations at once: who does what, what a pronoun refers to, which word came just before. Multi-head attention runs $h$ attentions side by side. Here $W^Q$, $W^K$ and $W^V$ are $d_\text{model}$ wide, and their results are cut into $h$ slices of $d_k = d_\text{model} / h$ columns, one per head. The $h$ results are put back side by side and projected by one more learned matrix, $W^O$ (Figure 14). The original transformer uses $h = 8$ heads of $d_k = 64$, so $d_\text{model} = 512$.

d_model

Q

→

d_model / h

Q₁

d_model / h

Q₂

d_model / h

Q₃

d_model / h

Q₄

concat(

head₁ … head₄

) ×

W^O

=

output

Positional encoding: Where each element stands

Leave the mask aside for a moment. A token's new vector is a sum over pairs, $\mathrm{out}_i = \sum_j \mathrm{softmax}_j(q_i \cdot k_j / \sqrt{d_k})\, v_j$. It depends on which tokens co-occur in a sequence, never on where they stand; $i$ and $j$ only label them. Shuffle the tokens, and the same results come out, shuffled the same way:

$$\mathrm{Attention}(PX) = P \cdot \mathrm{Attention}(X)$$

for any permutation $P$. So "the cow licks the calf" and "the calf licks the cow" give every word the same new vector.

There are two families of positional encodings. Absolute positional encodings give every place its own number, 0, 1, 2 and on. Relative positional encodings indicate, for every pair, how far apart the two tokens are (Figure 15). Rotary positions, the third subsection below, are relative too, but encode the distance by rotating.

Absolute positions

The simplest way is a second learned table, one row per position, added to the token's embedding. The smallest GPT-2 has 1,024 such rows of 768 numbers [6Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners.]. The model learns what a position should look like, but it cannot read a text longer than its table, and it has no notion of positions it never saw.

The original transformer computes the positions instead [1Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 30). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf]. Each dimension of the position vector follows a wave down the positions, fast in the first dimensions and slower and slower in the later ones:

$$PE_{(pos,\, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d}}\right), \qquad PE_{(pos,\, 2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d}}\right)$$

0 The1 ·cow2 ·says3 ·moo4 ·to5 ·the6 ·small7 ·calf

X

+

PE

=

X + PE

The coloured columns are three waves, read down the positions, and the outlined row is what position 5 adds to its token (Figure 17). The first wave turns 1 radian a position and has almost gone round once by position 6; the third turns a hundred times slower and has hardly moved.

(a) A column, one dimension down the positions, slower and slower to the right

01234567

PE

(b) A row, position 5's vector, added to the row of the token there, ·the

Nothing has to be learned, and there is a vector for any position. Each pair of dimensions, a sine and a cosine of the same speed, is a point on a circle. Going $k$ positions further along the text rotates that point by an angle that depends only on $k$, whatever position it starts from. So the distance between two tokens is contained in their position vectors, and the model can learn to read it. It does have to learn this, because the distance is never given to attention directly; it is only mixed into the sum $X + PE$.

Relative positions

Relative positions, first proposed by Shaw et al. [7Shaw, P., Uszkoreit, J., & Vaswani, A. (2018). Self-Attention with Relative Position Representations. In M. Walker, H. Ji, & A. Stent (Eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) (pp. 464–468). Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-2074], skip the detour through the embeddings and change the attention scores instead. A matrix $B$ with one learned number per distance, the same all along each diagonal, is added to $Q K^\top$ before the softmax (Figure 18). T5 learns these numbers per head [8Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140), 1–67.]. ALiBi does not learn them at all and simply lowers a score in proportion to the distance, so far-away tokens count less [9Press, O., Smith, N. A., & Lewis, M. (2022). Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. International Conference on Learning Representations (ICLR).].

The·small·purple·cow·says·moo

scores

+

b₀b₁b₀b₂b₁b₀b₃b₂b₁b₀b₄b₃b₂b₁b₀b₅b₄b₃b₂b₁b₀

B

=

scores + B

Rotary positions

Most of today's models use rotary positions, RoPE for short [10Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., & Liu, Y. (2024). RoFormer: Enhanced transformer with Rotary Position Embedding. Neurocomputing, 568, 127063. https://doi.org/10.1016/j.neucom.2023.127063]. They take the numbers of each query and key two dimensions at a time, read each pair as an arrow in a plane, and turn it by an angle that grows with the position. At position $m$, a pair turns by $m\theta$:

$$\begin{pmatrix} x'_1 \\ x'_2 \end{pmatrix} = \begin{pmatrix} \cos m\theta & -\sin m\theta \\ \sin m\theta & \cos m\theta \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix}$$

The dot product of a query and a key depends on the angle between them. Turn both by their own positions, and that angle changes only with how far apart they are. ·cow at position 2 and ·says at 3 meet at the same angle as at 12 and 13 (Figure 19), so attention sees relative distances.

Every pair turns at its own speed. With a single speed of 1 radian a position, the hand comes round again: position 44 is almost exactly 7 full turns, and looks like position 0. With many speeds, from fast to slow, every position gets its own set of hands, the way a clock's second, minute and hour hands tell every time of day apart (Figure 20).

The speeds are fixed before training and never learned. Commonly, pair $i$ of the $d/2$ pairs turns by

$$\theta_i = b^{-2(i-1)/d}$$

radians a position. With Llama 2's $b = 10{,}000$ and $d = 128$, the fastest pair turns 1 radian a position and goes round every 6 positions; the slowest turns 0.0001 radians and goes round about every 54,000 [11Touvron, H., Martin, L., Stone, K., & others. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv Preprint arXiv:2307.09288.]. Each pair is 0.87 times as fast as the one before, so every scale of distance, from neighbours to a long text, has pairs that tell it apart.

The speeds also set how long a text the model can read. Llama 2 was trained on 4,096 positions, so its slow pairs only ever saw a small part of their turn. Past 4,096 they reach angles the model has never seen, and its attention breaks down. Position interpolation halves every $\theta$, so 8,192 positions fit into the angles seen in training [12Chen, S., Wong, S., Chen, L., & Tian, Y. (2023). Extending Context Window of Large Language Models via Positional Interpolation. arXiv Preprint arXiv:2306.15595.]. YaRN stretches only the slow pairs and leaves the fast ones, which have seen every angle anyway [13Peng, B., Quesnelle, J., Fan, H., & Shippole, E. (2024). YaRN: Efficient Context Window Extension of Large Language Models. International Conference on Learning Representations (ICLR).]; Qwen3 uses it to go from 32,768 to 131,072 tokens [14Yang, A., Li, A., Yang, B., & others. (2025). Qwen3 Technical Report. arXiv Preprint arXiv:2505.09388.]. Newer models also start with a larger base, 500,000 in Llama 3 [15Grattafiori, A., Dubey, A., Jauhri, A., & others. (2024). The Llama 3 Herd of Models. arXiv Preprint arXiv:2407.21783.] and 1,000,000 in Qwen3.

Feed forward: What the world knows

Attention only combines what is already in the text. In "The Eiffel Tower is in ___", no token says Paris (Figure 21), and yet Qwen3-0.6B-Base puts 75 % on it (Figure 22). What the model knows about the world has to sit in its own weights.

1. ·Paris 75 %
2. ·France 6.0 %
3. ·the 4.3 %
4. other 15 %

Most of it is thought to sit in the feed forward network after the attention block [16Geva, M., Schuster, R., Berant, J., & Levy, O. (2021). Transformer Feed-Forward Layers Are Key-Value Memories. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 5484–5495.]. It works on every token's vector on its own, the same small network for every token; nothing passes between tokens there. It widens the vector, usually to four times $d_\text{model}$, applies ReLU, and narrows it back (Figure 23):

$$\mathrm{FFN}(x) = \mathrm{ReLU}(x W_1)\, W_2$$

d_ff = 4 * d_model​feed_forward = nn.Sequential(    nn.Linear(d_model, d_ff, bias=False),   # up: 512 → 2,048    nn.ReLU(),                              # the negatives to 0    nn.Linear(d_ff, d_model, bias=False),   # down: 2,048 → 512)​x = feed_forward(x)                         # every token, on its own

The feed forward holds most of a block's weights. Attention has four $d \times d$ matrices, $W^Q$, $W^K$, $W^V$ and $W^O$; the feed forward has two of $d \times 4d$, twice as many weights. Conceptually, its first layer can be read as keys that recognise patterns in a token's vector, and its second as the values written back when they match [16Geva, M., Schuster, R., Berant, J., & Levy, O. (2021). Transformer Feed-Forward Layers Are Key-Value Memories. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 5484–5495.]. Most of today's models use a gated variant, SwiGLU, instead of the plain ReLU [17Shazeer, N. (2020). GLU Variants Improve Transformer. arXiv Preprint arXiv:2002.05202.].

Add & Norm

Two small parts keep a deep network trainable. Each comes after every attention and every feed forward.

Layer norm rescales each token's vector to a steady range [18Ba, J., Kiros, J. R., & Hinton, G. E. (2016). Layer Normalization. ArXiv, abs/1607.06450. https://api.semanticscholar.org/CorpusID:8236317]. It subtracts the mean $\mu$ of the vector's $d$ numbers, divides by their spread $\sigma$, and lets the model scale and shift the result back with two learned vectors, $\gamma$ and $\beta$:

$$\mathrm{LayerNorm}(x) = \gamma \odot \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$

$\epsilon$, about $10^{-6}$, keeps it from ever dividing by 0, and $\odot$ multiplies element by element. Take the first six numbers of ·cow's vector after Qwen3's first layer, $(-3.77, 39.92, -0.13, -13.9, 0.93, 0.81)$, and, for illustration, normalise just these six; the real layer uses all 1,024. Their mean is 3.98 and their spread 16.9, so they become $(-0.46, 2.13, -0.24, -1.06, -0.18, -0.19)$. This way, every part gets its input in the same range, however deep the stack. Qwen3 itself uses RMSNorm, which skips the mean [19Zhang, B., & Sennrich, R. (2019). Root Mean Square Layer Normalization. Advances in Neural Information Processing Systems, 32.], and normalises before each part instead of after it [20Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., & Liu, T.-Y. (2020). On Layer Normalization in the Transformer Architecture. Proceedings of the 37th International Conference on Machine Learning, 10524–10533.]; it also normalises each query and key (QK-norm) [14Yang, A., Li, A., Yang, B., & others. (2025). Qwen3 Technical Report. arXiv Preprint arXiv:2505.09388.].

Add is the residual connection [21He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778.]. Attention does not replace a token's vector; it adds what it found, $x + \mathrm{Attention}(x)$, and the feed forward does the same, $x + \mathrm{FFN}(x)$. Each token's vector runs straight up through the model, the residual stream, and every part writes a change into it. Each part only has to learn a correction, and in training the gradient has a straight path down through the sums.

Decoder blocks

Attention and feed forward, each with its Add & Norm, together are a decoder block. A model stacks many of them, each with its own weights, each taking $n$ vectors of $d_\text{model}$ numbers and handing on the same shape (Figure 24).

Linear and softmax: What comes next

After the last decoder block, we have a sequence of enriched vectors. Each represents a token of the input sequence, but contextualised by its preceding tokens. The final step in the model translates these vectors into a probability distribution over the model's vocabulary. First, a last learned matrix, a simple linear layer, turns each vector into a score per token of the vocabulary, the logits; softmax turns the logits into a distribution (Figure 25). In Qwen3-0.6B, this matrix is the embedding table itself, used the other way round. In training, every row (i.e. every token in the sequence) predicts the token after it. In generation, only the last row is used.

used for training: should be ·cowtraining: should be ·saysgeneration: draw , ↑↑↑ next token 3 × 151,936 ↑↑↑ the head

Linear → Softmax

↑↑↑ H, out 3 × 1,024, the same shape ⋯ ⋯ ⋯ ↑↑↑ the model

28 decoder blocks

↑↑↑ embeddings X 3 × 1,024 ⋯ ⋯ ⋯ ↑↑↑ tokens The·cow·says

That completes the decoder-only transformer (Figure 26).