Semester 4 · Language models
Post-training, Alignment and Evaluation of LLMs
https://epochschool.com/subjects/llm-post-training-alignment
Turn a pretrained language model into an assistant that follows instructions, prefers better answers, reasons on checkable problems and declines harmful requests, then measure honestly whether it worked.
You derive and implement supervised fine-tuning, LoRA and quantised fine-tuning, Bradley-Terry reward models, policy gradients, PPO, GRPO and DPO, train each on a tiny model on one CPU core, and red-team the result. The last third is evaluation: paired statistics, contamination, human and model judges, calibration and hallucination, so you can defend every claim you make about a model.
- Prerequisites
- Required: Finish this subject before you start. The subject builds directly on it.Recommended: Helpful background. You can start without it if you are willing to fill the gaps.
What you’ll be able to do
- LO1Explain what post-training changes in a pretrained model, and implement supervised fine-tuning with chat templates, loss masks and well-chosen instruction data.
- LO2Derive, implement and budget parameter-efficient and quantised fine-tuning (adapters, LoRA, and LoRA on a 4-bit base), including LoRA's gradients and merging rule.
- LO3Derive Bradley-Terry and Plackett-Luce models of preference, train and evaluate a neural reward model against the ceiling set by annotator noise, and diagnose reward over-optimisation.
- LO4Derive the policy-gradient, KL-regularised and PPO objectives for language models, and implement RLHF on a small model with a critic and generalised advantage estimation.
- LO5Apply RL with verifiable rewards (rejection sampling, GRPO and its variants) and test-time compute to reasoning tasks, and judge from pass@k evidence what such training does and does not change.
- LO6Derive direct preference optimisation from the KL-regularised objective, implement it and its variants (IPO, SimPO, ORPO, KTO), diagnose its failure modes, and choose between offline and online preference methods.
- LO7Train and measure safety behaviour: refusals, the trade-off between harmful compliance and over-refusal, jailbreak mechanisms, and red-team evaluation on held-out attack families.
- LO8Design statistically sound evaluations of post-trained models: verifiable instruction following, paired comparisons with adequate power, contamination checks, human and arena ratings, and validated model judges.
- LO9Measure and improve the calibration of generated answers, explain and reduce hallucination through abstention, and apply knowledge distillation and model merging.
Chapters
The chapter and lab list for this subject is being finalised and will be published here.
Before you start
Required
Finish this subject before you start. The subject builds directly on it.
Responsible AI, Ethics & Governance
Recommended
Helpful background. You can start without it if you are willing to fill the gaps.
NLP & Language Model Foundations
Where it leads
Subject 3 of 4 in the Language models track.
Previous: Building and Training LLMs from ScratchNext: LLM Applications: Retrieval, RAG and Agents
How it’s taught
You read short sections written for the page. Each ends with a check, and in-browser exercises let you practise. Graded labs run on your own cloud lab machine, CPU-friendly.
No videos. No live classes. No AI tutor.