Reinforcement Learning and for Post Training LLMs

Slides

Readings

Coding Resources

Category

structured

Tags

RL

Att: the following markdown text was generated from the corresponding powerpoint lecture file automatically. Errors and misformatting, therefore, do exist (a lot!)!

In this lecture, we cover:

(SFT, LoRA, and an introduction to RLHF and DPO are in “Transformer Post-training”; hands-on RL environments and tooling are in “deep RL Gym”.)

Study Guide: Reinforcement Learning and for Post Training LLMs


Quiz: Short-Answer Questions

Instructions: Provide a concise answer (2-3 sentences) for each of the following questions.

  1. Why is next-token cross-entropy not enough to make an LLM helpful, correct, and safe, and why does that call for reinforcement learning?
  2. Name the four elements of RL, and explain the difference between a reward and a value.
  3. What is credit assignment, and why is it a problem both for a game player and for an LLM writing a 500-token answer?
  4. Compare Monte Carlo and temporal-difference (TD) learning. Which one needs the episode to end?
  5. On the grid world (γ = 0.5, deterministic moves), after rounds 1 and 2 the only non-zero entries are Q̂(<2,1>, north) = 100 and Q̂(<2,2>, north) = 50. Round 3 starts at <1,1> and moves east to <2,1> with reward 0. What is the new Q̂(<1,1>, east)? What would Q̂(<1,1>, south) be if it had moved south to <1,2> instead?
  6. In the same example, why did <2,2> get a non-zero value only in round 2, not in round 1?
  7. What two tricks make deep Q-learning (DQN) stable, and what problem does each one fix?
  8. Why does LLM training use policy gradients rather than value-based methods such as Q-learning?
  9. For a fixed state s, why does subtracting a baseline b(s) that does not depend on the action leave the expected policy gradient unchanged? A group of 4 answers gets rewards 1, 1, 1, 0; with the mean as the baseline, what weight does each answer get?
  10. In PPO, what do the ratio ρt(θ) and the clip do when  > 0 and when  < 0?
  11. In RLHF, why is a reward model trained, and why is a KL penalty to πref added?
  12. How does DPO remove the reward model and the RL loop?
  13. GRPO samples G = 4 answers to one prompt with rewards 1, 0, 0, 1. Compute each answer’s advantage, and say what GRPO no longer needs compared with PPO.

Answer Key

  1. Cross-entropy rewards copying the data token by token, but helpfulness, correctness, and safety are properties of the whole answer. There is no single “right next token” for them, only a score after the answer is written, and learning from a score that comes after a sequence of choices is reinforcement learning.

  2. The elements are a policy π(a|s), a reward r, a value function V(s) or Q(s, a), and (optionally) a model of the environment P(s′|s, a). The reward says what is good now; the value is the total expected reward from a state (or state-action pair), so it says what is good in the long run.

  3. Credit assignment is deciding which of many earlier actions deserves the credit for a reward that arrives later. Actions affect the immediate reward and all later rewards, so when a game is won after 100 moves, or an answer is judged after 500 tokens, the single final score must be spread back over all the choices.

  4. Monte Carlo waits until an episode ends and averages the observed returns, so it only works for episodic tasks; it is unbiased but has high variance. TD learning, like DP, learns from the values of successor states and updates after every step, so it works for continuing tasks and is usually faster than MC; like MC, it needs no model.

  5. Q̂(<1,1>, east) = 0 + 0.5 × maxa′ Q̂(<2,1>, a′) = 0.5 × 100 = 50. Moving south to <1,2> gives Q̂(<1,1>, south) = 0 + 0.5 × 0 = 0, because every entry for <1,2> is still 0.

  6. The value of the chest reaches <2,2> only after <2,1> already has a non-zero Q value. Round 1 started at <1,1> and never visited <2,2>; it set Q̂(<2,1>, north) = 100 at its last step. In round 2 the agent moved north from <2,2> to <2,1>, so the update read that 100 and set Q̂(<2,2>, north) = 0 + 0.5 × 100 = 50. Values spread backward one step per visit.

  7. Experience replay and a target network. Experience replay stores transitions (s, a, r, s′) in a buffer and trains on random mini-batches, breaking the correlation between consecutive frames; the target network w− is a frozen copy of w refreshed every C steps, so the regression target does not move every step.

  8. An LLM is already a stochastic policy over tokens, with a huge action space of ~50k tokens. Value-based RL acts by an argmax over Q(s, a) and has no explicit policy, which is hard with many actions; policy-based RL learns πθ(a|s) directly and can learn stochastic policies, at the cost of local optima and high-variance gradient estimates.

  9. Writing the expectation as a sum over actions, Σa π(a|s) ∇log π(a|s) b(s) = b(s) ∇ Σa π(a|s) = b(s) ∇1 = 0. So the baseline lowers variance without changing the expected gradient (if b depended on a, it could no longer be pulled out of the sum, and this argument fails). With rewards 1, 1, 1, 0 the mean is 0.75, so the weights are +0.25, +0.25, +0.25, and −0.75.

  10. ρt(θ) = πθ(at|st) / πθold(at|st) measures how far the new policy has moved, and the clip limits it to [1 − ε, 1 + ε] (ε ≈ 0.2). When  > 0 PPO raises the action’s probability, but by at most a factor 1 + ε; when  < 0 it lowers it, but by at most 1 − ε. This lets PPO reuse each batch for several steps without a too-large step wrecking the policy.

  11. Asking humans for every score is expensive, so a separate reward model rφ(x, y) is trained on human comparisons to score the winning answer yw above the losing answer yl. The policy then maximizes this reward minus a KL penalty to the reference model πref, which stops RL fine-tuning from drifting to gibberish that fools the reward model.

  12. The RLHF objective (maximize reward minus a KL penalty to πref) has a closed-form solution, so the reward can be written in terms of the policy. Because the reward-model loss only needs the difference of rewards, r(x, yw) − r(x, yl) = β[log(πθ(yw|x)/πref(yw|x)) − log(πθ(yl|x)/πref(yl|x))], and plugging this into −log σ(·) gives a classification loss that connects preference data to the LM parameters directly, with no reward model and no RL loop.

  13. The mean reward is 0.5 and the standard deviation is 0.5, so  = (r − 0.5) / 0.5 = +1, −1, −1, +1. All tokens of an answer share its advantage. Because the group mean serves as the baseline, GRPO needs no value model (critic), which PPO must train and run; this makes it much more efficient for scaling up LLM training.


Essay Questions

Instructions: Prepare a detailed, essay-format response for each of the following prompts. (Answers not provided).

  1. Trace the grid-world example from policy evaluation to Q-learning. Define the MDP (states, actions, rewards, γ = 0.5), evaluate the given policy, run Q-learning by hand until the values converge (100, 50, 25, 12.5, 6.25), and explain how ε-greedy exploration and, for large state spaces, deep Q-learning extend the method.

  2. Explain the single policy-gradient template behind REINFORCE, actor-critic, PPO, GRPO, and supervised fine-tuning. Derive the gradient with the log-derivative trick, and compare the weight Wt that each method multiplies ∇log π by, the extra model it needs, and how it controls variance and step size.

  3. Map reinforcement learning onto an LLM and walk through the three phases of RLHF. Define state, action, policy, transition, episode, and reward for a language model; then describe instruction tuning, reward model training on preference pairs, and PPO with a KL penalty, and discuss the practical difficulties of RLHF.

  4. Compare RLHF with PPO, DPO, and GRPO with verifiable rewards. For each, describe the data it needs, the models it trains or keeps in memory, and its loss; then explain how DeepSeek-R1-Zero used format and accuracy rewards with GRPO to elicit reasoning, and how that reasoning was distilled into a smaller model.


Glossary of Key Terms

Term Definition
Reinforcement Learning (RL) Learning to map states to actions, by trial and error, so as to maximize a numerical reward over many steps.
Policy π(a|s) A map from states to actions (or a distribution over actions); may be stochastic.
Reward r A real number for a state or state-action pair: what is good now.
Value Function V(s) / Q(s, a) The total expected (discounted) reward from a state, or from a state-action pair, when following a policy: what is good in the long run.
Markov Decision Process (MDP) States, actions, rewards, and transitions P(s′|s, a) in which the future depends only on the current state.
Discount Factor γ A number 0 ≤ γ < 1 that weights a reward i steps ahead by γi in the return.
Credit Assignment Deciding which earlier actions (moves or tokens) deserve credit for a delayed reward.
Bellman Equation A recursive equation relating a state’s value to the reward and the values of its successor states; the optimality version defines V* and Q*.
Policy / Value Iteration Dynamic programming methods that need a full model of the environment; value iteration uses the Bellman optimality equation as an update.
Temporal-Difference (TD) Learning Learning from experience after every step by updating toward the reward plus the discounted value of the successor state.
ε-greedy Take the greedy action with probability 1 − ε and a random action with probability ε.
Q-learning A model-free TD method that learns Q(s, a) toward r + γ maxa′ Q(s′, a′) and acts by argmax.
Deep Q-learning (DQN) Q-learning with a neural network (a CNN on pixels) for Q(s, a; w), trained with experience replay and a target network.
Policy Gradient Directly raising the log-probability of sampled actions in proportion to their reward, using ∇p = p ∇log p.
REINFORCE Monte Carlo policy gradient that weights ∇log π by the observed return; unbiased but high variance.
Baseline / Advantage A value subtracted from the return to lower variance; the advantage  ≈ Q(s, a) − V(s) says how much better than average an action was.
Actor-Critic A policy (actor) updated in the direction suggested by a learned value function (critic), with bootstrapped updates at each step.
PPO Proximal Policy Optimization: a policy-gradient method that clips the probability ratio to [1 − ε, 1 + ε] so the policy moves only a little per update.
RLHF Reinforcement learning from human feedback: SFT, then a reward model from preference pairs, then PPO with a KL penalty to πref.
Reward Model A model rφ(x, y) trained on human comparisons to give the preferred answer a higher scalar reward.
KL Penalty A term that keeps the fine-tuned policy close to the reference model πref, so it does not drift into outputs that fool the reward model.
DPO Direct Preference Optimization: rewrites the RLHF objective as a classification loss on preference pairs (yw, yl), with no reward model and no RL loop.
RLVR RL with verifiable rewards: a checker (e.g., a math answer or unit tests) gives reward 1 if correct and 0 otherwise.
GRPO Group Relative Policy Optimization: PPO without a critic, using the reward normalized over a group of G answers to the same prompt as the advantage.
DeepSeek-R1-Zero A reasoning model trained by pure RL from DeepSeek-V3-Base with format and accuracy rewards.
Distillation Training a smaller student model to match a larger teacher’s output distribution, as in distilling DeepSeek-R1’s reasoning into Qwen-32B.