Michał Chromiak's blog Language models: From n-gram counts to neural next-token prediction
Probability, smoothing, cross-entropy, perplexity, and the assumptions behind the numbers
A language model assigns probabilities to token sequences by predicting each token from its context. This guide derives the chain rule, explains n-gram estimation and smoothing, connects cross-entropy to perplexity, and shows what changes when a neural Transformer replaces the count table.
more ...