LLM basics - Preference Alignment
- SlideDeck: 2026-SP-W1.2-basic-llm-alignment.pdf
- Version: current
- Notes: Basic Preference Optimization
In this session, our readings cover:
Reading
Here are the related slide deck from the previous two course offerings:
| Topic | Slide Deck | Previous Semester |
|---|---|---|
| Introduction to Deep NLP Basics | W1.1-deepNNtext | 25course |
| LLM Basics - Emergent Ability and GenAI Platform | W1.2-IntroLLMv3 | 25course |
| More LLM Basics - A Survey | W2.1-moreLLM | 25course |
| LLM Basics Foundation | S0-Intro | 24course |
| Survey: LLMs and Multimodal FMs | S1-LLM | 24course |
| Recent LLM Basics | W13-RecentLLMbasics | 24course |
| Advanced Transformer Architectures | W14_LLM_advanced_arch | 24course |
A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More
- [Submitted on 23 Jul 2024]
- Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Zixu (James)Zhu, Xiang-Bo Mao, Sitaram Asur, Na (Claire)Cheng
- With advancements in self-supervised learning, the availability of trillions tokens in a pre-training corpus, instruction fine-tuning, and the development of large Transformers with billions of parameters, large language models (LLMs) are now capable of generating factual and coherent responses to human queries. However, the mixed quality of training data can lead to the generation of undesired responses, presenting a significant challenge. Over the past two years, various methods have been proposed from different perspectives to enhance LLMs, particularly in aligning them with human expectation. Despite these efforts, there has not been a comprehensive survey paper that categorizes and details these approaches. In this work, we aim to address this gap by categorizing these papers into distinct topics and providing detailed explanations of each alignment method, thereby helping readers gain a thorough understanding of the current state of the field.
Extra Readings: