Foundations of diffusion language models
Read a method section and identify the state space, corruption process, training objective, and sampler.
Source-linked starting points, not an exhaustive survey or a current leaderboard. Reading exercises below are suggestions, not experiments run by this site.
Before you start
Conditional probability, cross-entropy, and the difference between a token and its embedding. No diffusion implementation is required.
Forward process · Reverse process · Absorbing mask · Discrete score / score entropy · ELBO / negative ELBO · Continuous versus discrete state
1. Define what gets corrupted
Structured Denoising Diffusion Models in Discrete State-Spaces — Austin et al. · NeurIPS 2021
Sections 2–3: forward/reverse processes and categorical transitions.
D3PM specifies corruption using transition matrices over discrete states; masking is one choice, not the definition of all diffusion.
Check your understanding
For a three-token sentence, sketch two corrupted states. Which transitions are legal under masking but not token replacement?
Reasoning to check against
An absorbing-mask process can replace a token with MASK; after that it stays masked in the forward process. A general replacement process can move between vocabulary tokens.
2. Separate training from decoding
Simple and Effective Masked Diffusion Language Models — Sahoo et al. · NeurIPS 2024
Section 3: parameterization and the masked-language-model objective.
MDLM expresses its objective as a mixture of masked-LM losses. A training loss and a sampling schedule are different parts of the method.
Check your understanding
Does one masked-token prediction pass automatically define a complete text generator?
Reasoning to check against
No. Specify initialization, the noise/time schedule, how predictions become tokens, and termination. The same trained model may admit several samplers.
3. Understand the discrete score
The score-entropy formulation and language-model evaluation setup.
SEDD estimates probability ratios between discrete states. Do not treat a discrete token ID as a continuous coordinate to differentiate.
Check your understanding
Can a continuous-time model still have a discrete state space?
Reasoning to check against
Yes. Time and state are separate axes: time may be continuous while each state remains a sequence of categorical tokens.
Apply the reading
For one paper, write four lines: state → corruption → objective → sampler. Mark anything the paper does not specify as unknown.