Sequence Models & Transformers

What you’ll be able to do

1. LO1Factorise sequence probabilities with the chain rule, build and smooth n-gram language models, and evaluate any language model by cross-entropy, perplexity or bits per byte on held-out data.
2. LO2Derive, implement and evaluate word embeddings (PPMI with SVD, skip-gram with negative sampling, GloVe) and explain what static embeddings capture and miss.
3. LO3Implement recurrent networks and backpropagation through time from first principles, and analyse vanishing and exploding gradients through Jacobian products.
4. LO4Derive, implement and choose between gated recurrent cells (LSTM, GRU) and recurrent architectures (stacked, bidirectional, encoder-decoder) for a given sequence task.
5. LO5Derive and implement attention mechanisms (additive, multiplicative, scaled dot-product and multi-head) with padding and causal masks, and explain how attention removes the encoder-decoder bottleneck.
6. LO6Build a Transformer from its components (positional encodings including RoPE, pre-LN residual blocks, feed-forward layers), and compute its parameters, FLOPs and memory.
7. LO7Compare encoder-only, decoder-only and encoder-decoder models with their pretraining objectives (causal and masked LM), and explain how tokenisation choices affect cost, quality and comparisons.
8. LO8Implement and judge decoding strategies (greedy, beam, temperature, top-k, top-p) and KV caching for a given generation task.
9. LO9Analyse the cost of attention at long context, explain efficient alternatives (local and linear attention, IO-aware exact attention, grouped-query attention, state-space models), and apply Transformers beyond text with sound baselines.

Chapters

12 chapters · each section ends with a short check

1. 01Sequences, probability and language models
2. 02Word embeddings: meaning from context
3. 03Recurrent neural networks and backpropagation through time
4. 04Long-range memory: vanishing gradients, LSTM and GRU
5. 05Encoder-decoder models
6. 06Attention: learning where to look
7. 07Self-attention in depth
8. 08The Transformer architecture
9. 09Pretraining objectives and tokenisation
10. 10Decoding: from probabilities to text
11. 11Efficient attention and long context
12. 12Transformers beyond text

Graded labs

1. L1

   Language models and word vectors from counts and from prediction

   After chapter 2 · about 9 hours · Railway-announcement corpus · CC BY 4.0 (own work) · Runs on a CPU

2. L2

   Recurrent networks from scratch: BPTT, vanishing gradients and gated cells

   After chapter 4 · about 10 hours · Optical recognition of handwritten digits · BSD-3-Clause (scikit-learn) · Runs on a CPU

3. L3

   Normalising dates with an attentive encoder-decoder

   After chapter 6 · about 10 hours · Synthetic date-normalisation pairs · CC BY 4.0 (own work) · Runs on a CPU

4. L4

   A small Transformer language model from scratch

   After chapter 9 · about 10 hours · Railway-announcement corpus · CC BY 4.0 (own work) · Runs on a CPU

5. L5

   Decoding, KV caching and efficient attention

   After chapter 11 · about 10 hours · Railway-announcement corpus · CC BY 4.0 (own work) · Runs on a CPU

Before you start

Where it leads

How it’s taught

You read short sections written for the page. Each ends with a check, and in-browser exercises let you practise. Graded labs run on your own cloud lab machine, CPU-friendly.

No videos. No live classes. No AI tutor.

How it works→