Semester 3 · Language models
NLP & Language Model Foundations
https://epochschool.com/subjects/nlp-language-model-foundations
How computers read and model human language beyond the Transformer core: Unicode and the world's writing systems, the Unigram and WordPiece tokenisers, smoothed n-gram models and their mixture with neural models, word vectors across languages and scripts, including morphologically rich ones, text classification, sequence labelling and extraction, sentence encoders, and honest evaluation with the right metric, a significance test and a contamination check.
Many languages and scripts run through the whole subject, among them English, Spanish, Arabic, Hindi, Chinese and Swahili: Devanagari and other abugida scripts as one family of writing systems, transliteration, code-mixing, cross-lingual transfer and machine translation, including for low-resource languages. It builds on Sequence Models & Transformers, which teaches language-model basics, static word embeddings, the BPE concept, pretraining objectives and decoding, and recalls them instead of re-teaching them.
- Prerequisites
- Required: Finish this subject before you start. The subject builds directly on it.Recommended: Helpful background. You can start without it if you are willing to fill the gaps.
What you’ll be able to do
- LO1Process raw multilingual text correctly: encodings, Unicode normalisation, the structure of different writing systems, segmentation and corpus statistics.
- LO2Implement and train the WordPiece and Unigram tokenisers (EM, pruning and Viterbi segmentation), explain SentencePiece and subword regularisation, and measure a tokeniser's effect on sequence length, cost and parity across languages.
- LO3Smooth n-gram language models in depth (Good–Turing, Katz backoff, Kneser–Ney), and compare and combine count and neural language models fairly.
- LO4Evaluate and extend word and sentence representations across languages and scripts: subword vectors for morphology, hubness, bias measurement and contrastive sentence encoders trained with InfoNCE.
- LO5Build and diagnose text classifiers for real tasks, from TF-IDF linear models to subword-embedding models and fine-tuned encoders.
- LO6Build sequence labellers (HMMs, CRFs, neural taggers) and extract entities, relations and dependency structure from text.
- LO7Design honest evaluations of NLP systems: choose, compute and validate metrics for classification, translation and generation, test significance, measure annotator agreement, and detect contamination and dataset artefacts.
- LO8Handle multilingual and low-resource-language settings: language identification, transliteration, code-mixing, cross-lingual alignment, transfer and machine translation.
Chapters
The chapter and lab list for this subject is being finalised and will be published here.
Before you start
Required
Finish this subject before you start. The subject builds directly on it.
Neural Networks & Deep Learning
Recommended
Helpful background. You can start without it if you are willing to fill the gaps.
Optimisation & Training at Scale
Probability & Statistics for ML
Where it leads
Subject 1 of 4 in the Language models track.
Building and Training LLMs from Scratch
Post-training, Alignment and Evaluation of LLMs
LLM Applications: Retrieval, RAG and Agents
Capstone Project
How it’s taught
You read short sections written for the page. Each ends with a check, and in-browser exercises let you practise. Graded labs run on your own cloud lab machine, CPU-friendly.
No videos. No live classes. No AI tutor.