L19: Transformer Pretraining

Slides

Readings

Extra Readings

Coding Resources

Category

LLM/Agents

Tags

6LLMAgents Deep Generative Training

Att: the following markdown text was generated from the corresponding powerpoint lecture file automatically. Errors and misformatting, therefore, do exist (a lot!)!

In this lecture, we cover:

(Turning a pretrained base model into an assistant (SFT, LoRA/QLoRA, RLHF, DPO) is covered in “Transformer Post-training”.)

Study Guide: Transformer Pretraining


Quiz: Short-Answer Questions

Instructions: Provide a concise answer (2-3 sentences) for each of the following questions.

  1. How is BERT pretrained, and how does its training differ from GPT’s?
  2. What is the training target for each position of a GPT input sequence, and why can all T positions be trained in one forward pass?
  3. Why is pretraining with next-token cross-entropy the same as maximum likelihood?
  4. Why do we use subword tokenization (such as BPE) instead of whole words or single characters?
  5. How does Adam differ from SGD with a single learning rate, and what does AdamW change?
  6. What are learning-rate warmup and decay, and why are they used when training Transformers?
  7. Besides masked language modeling, what second pretraining task does BERT use, and on what data is it pretrained?
  8. How much text were Llama 3 and DeepSeek-V3 pretrained on, and how big is that?
  9. Classical bias–variance says more parameters eventually overfit. Why does an LLM’s test loss keep falling as parameters grow?

Answer Key

  1. BERT is pretrained as a masked language model: about 15% of tokens are picked (80% replaced by [MASK], 10% by a random token, 10% kept), and the model predicts them from both left and right context. Unlike GPT, it uses no causal mask, has a loss only at the picked positions, adds a task head for fine-tuning, and does inference in one forward pass with no generation.

  2. The target at position t is the token at position t + 1: the targets are the inputs shifted left by one. The causal mask stops position t from seeing any token after t, so all T predictions can be computed in parallel from the true prefix (teacher forcing), with no step-by-step loop.

  3. The loss is ℒ(θ) = −(1/N) ∑ log pθ(xt | x<t), the negative log-likelihood of the text under the chain rule. Maximizing ∏ pθ(xt | x<t) is the same as minimizing −∑ log pθ(xt | x<t), and the labels are free because the next word is already in the text.

  4. Whole words give a huge vocabulary and cannot handle unseen words, while characters give very long sequences. Subword units keep the vocabulary at a fixed size (about 32K to 200K), split rare words into pieces, and never fall back to an unknown token.

  5. Adam keeps two running averages per weight, the gradient (momentum) and the squared gradient (scale), with a bias fix, so each weight gets its own step size: steep directions are scaled down and flat directions scaled up. AdamW adds the weight decay λθ directly to the update instead of to the gradient, so it is not divided by √ŝ and every weight decays equally; it is the usual choice for Transformers.

  6. Warmup starts with a small learning rate and increases it linearly over the first steps, while gradients are noisy and Adam’s running averages are unsettled; decay (e.g., cosine) takes smaller steps near the end. Debug with a fixed learning rate first, then add the schedule.

  7. BERT also uses next-sentence prediction: given a sentence pair A, B, it predicts IsNext or NotNext, with the two cases 50/50. It is pretrained on BooksCorpus (800M words) and Wikipedia (2.5B words) with WordPiece tokens, adding the next-sentence cross-entropy to the masked-LM loss.

  8. Llama 3 was pretrained on about 15T multilingual tokens (vs. 1.8T for Llama 2), and DeepSeek-V3 on 14.8 trillion tokens. Printed at 1,000 tokens per page and 100 sheets per cm, 15T tokens would make a stack about 1,500 km (1,000 miles) thick.

  9. The training data grows along with the model and is seen only about once, so there is little to memorize. As a result, test loss keeps falling as a power law in parameters, data, and compute (Kaplan et al., 2020), and the lesson is to scale data and parameters together.


Essay Questions

Instructions: Prepare a detailed, essay-format response for each of the following prompts. (Answers not provided).

  1. Compare how an encoder-only model (BERT) and a decoder-only model (GPT) are pretrained. For each, describe the data, the training targets, the mask, the loss, the optimizer and schedule, and what the pretrained weights are then used for.

  2. Explain why next-token prediction on web-scale text works so well. Discuss the chain rule and maximum likelihood, why next-word prediction is implicit multi-task learning, and how scale (data, parameters, compute) and scaling laws shaped GPT-1 → GPT-2 → GPT-3 and the models of 2024.


Glossary of Key Terms

Term Definition
Masked Language Model (MLM) BERT’s pretraining task: predict picked (masked) tokens from their left and right context.
Byte-Pair Encoding (BPE) A tokenizer that starts from characters (or bytes) and repeatedly merges the most frequent adjacent pair into a new token.
Next-Token Prediction The pretraining task of a GPT: predict token t + 1 from tokens 1, …, t, trained with cross-entropy (= maximum likelihood).
Teacher Forcing Training on the true previous tokens instead of the model’s own outputs, so all positions are trained in one pass.
Adam / AdamW Gradient descent with momentum and a per-weight step size; AdamW decouples weight decay from the adaptive scaling.
Warmup and Decay Increasing the learning rate from near zero over the first steps, then lowering it (e.g., cosine) toward the end.
Next-Sentence Prediction (NSP) BERT’s second pretraining task: predict whether sentence B follows sentence A (IsNext / NotNext, 50/50).
Scaling Laws Test loss falls as a power law in data, parameters, and compute: straight lines on log-log axes.