2025 Fall UVA CS - Machine Learning



TuTh 3:30pm-4:45PM

Bias Variance Tradeoff

5Theory Local

Att: the following markdown text was generated from the corresponding powerpoint lecture file automatically. Errors and misformatting, therefore, do exist (a lot!)!

Study Guide: The Bias-Variance Tradeoff

This guide is designed to review and reinforce key concepts related to the bias-variance tradeoff in machine learning. It includes a quiz with an answer key to test comprehension, a set of essay questions for deeper exploration, and a glossary of essential terms.

Part I: Short-Answer Quiz

Instructions: Answer the following ten questions in 2-3 sentences each, based on the provided source material.

  1. What is the ultimate goal in machine learning, and why is minimizing training error not always sufficient to achieve it?
  2. Explain the decomposition of Expected Prediction Error (EPE). What does each component represent?
  3. Describe the characteristics of a model that is “underfitting.” How does this relate to bias and variance?
  4. Describe the characteristics of a model that is “overfitting.” How does this relate to bias and variance?
  5. Why are bias and variance not directly computable in a real-world machine learning problem?
  6. Compare the properties of a linear classification model with a 1-Nearest Neighbor model in terms of bias and variance.
  7. According to statistical decision theory, what is the best theoretical estimator for a regression problem under L2 (squared error) loss?
  8. Identify two distinct methods for reducing high model variance (overfitting).
  9. Identify two distinct methods for reducing high model bias (underfitting).
  10. What is the difference between model selection and model assessment?

Part II: Quiz Answer Key

  1. The ultimate goal is generalization, which is the model’s ability to perform well on new, unseen data. Minimizing training error is not always sufficient because a model can become too complex and fit irrelevant noise in the training data (e.g., 1-NN), leading to poor performance on test data.

  2. Expected Prediction Error (EPE) decomposes into three parts: EPE(x) = noise² + bias² + variance. Noise (or Irreducible/Bayes error) is the unavoidable error inherent in the data. Bias is the error from incorrect assumptions made by the model. Variance is the error due to the model’s sensitivity to the randomness of the training samples.

  3. An underfitting model is too “simple” to represent the relevant characteristics of the data. This condition is defined by high bias and low variance, resulting in both high training error and high test error.

  4. An overfitting model is too “complex” and fits irrelevant characteristics or noise in the training data. This condition is defined by low bias and high variance, which leads to low training error but high test error.

  5. Bias and variance cannot be computed directly because their calculation relies on knowing the true underlying distribution of the input vectors (x) and output variables (y), which is unknown in practice.

  6. A linear classification model is a global model that is stable but can be inaccurate, exhibiting high bias and low variance. In contrast, a 1-Nearest Neighbor model is a local model that is accurate on training data but unstable, exhibiting low bias and high variance.

  7. Under L2 loss, the best theoretical estimator to minimize EPE is the conditional mean (also called conditional expectation). This is expressed as f̂(x) = E(Y X = x).

  8. To reduce high variance, one can choose a simpler classifier, regularize the parameters, get more training data, or use a smaller set of features.

  9. To reduce high bias, one can get additional features to provide more information or try a more complex learning model with more flexibility.

  10. Model selection is the process of estimating the performance of different models to choose the best one. Model assessment is the subsequent process of taking the chosen model and estimating its prediction error on new, unseen data.

Part III: Essay Questions

Instructions: The following questions are designed to encourage a deeper, more synthesized understanding of the topic. Formulate comprehensive answers based on the source material.

  1. Discuss the Bias-Variance Tradeoff in detail. Explain how model complexity influences bias, variance, training error, and expected test error. Use the examples of a highly regularized linear model and a 1-Nearest Neighbor classifier to illustrate the opposing ends of the complexity spectrum.

  2. You are presented with a learning curve where the training error is unacceptably high and there is only a small gap between the training error and the test error. Diagnose the problem, explain the underlying issues in terms of bias and variance, and propose at least two concrete remedies to improve the model.

  3. Explain the concept of Expected Prediction Error (EPE) and its decomposition into noise, bias, and variance. Describe why the noise component is considered “irreducible” and provide the theoretical justification for why the conditional mean is the optimal estimator for EPE under L2 loss.

  4. Compare and contrast the Frequentist and Bayesian interpretations of probability as they relate to model parameters. How does the Bayesian approach handle parameter estimation differently from the Frequentist approach?

  5. Describe the role of cross-validation (CV) in the machine learning pipeline, particularly when data is not abundant. Discuss the practical considerations for choosing the value of K in K-fold CV, including the bias-variance tradeoff inherent in this choice.

Part IV: Glossary of Key Terms

Term Definition
Bias The error component resulting from incorrect assumptions or simplifications made by a model. It measures the accuracy or quality of an estimator; low bias means the estimator will, on average, accurately estimate the true parameter.
Variance The error component resulting from a model’s sensitivity to the specific training data sample. It measures the precision or specificity of an estimator; low variance means the estimator does not change much as the training set varies.
Bias-Variance Tradeoff The principle that models with few parameters (low complexity) tend to have high bias and low variance, while models with many parameters (high complexity) tend to have low bias and high variance. The goal is to find a balance that minimizes total error.
Underfitting A state where a model is too “simple” to represent the relevant characteristics of the data. It is characterized by high bias, low variance, high training error, and high test error.
Overfitting A state where a model is too “complex” and fits irrelevant noise in the training data. It is characterized by low bias, high variance, low training error, and high test error.
Generalization The ultimate goal of a machine learning model, referring to its performance on new, unseen data.
Expected Prediction Error (EPE) The expected value of a loss function over the joint probability distribution of inputs and outputs. It decomposes into noise² + bias² + variance.
Irreducible Error (Bayes Error / Noise) The component of EPE that is unavoidable, resulting from inherent randomness or noise in the data itself.
L2 Loss (Squared Error Loss) A loss function used in regression that calculates the error as the square of the difference between the true output and the predicted output: (y - f(x))².
Conditional Mean The optimal estimator for minimizing Expected Prediction Error under L2 loss, mathematically expressed as E(Y \| X = x).
Model Selection The process of estimating the performance of different models (or a single model with different hyperparameters) to choose the best one.
Model Assessment The process of taking a final, chosen model and estimating its prediction error on new data.
Cross-Validation (CV) An efficient method for reusing samples for model selection and assessment, particularly when there is insufficient data for a simple train/validation/test split.
K-Nearest Neighbors (k-NN) A local, non-parametric method where k acts as a smoother. A k=1 model has very low bias and high variance, while a model with a larger k (e.g., k=15) is smoother, with higher bias and lower variance.
Frequentist Probability An interpretation of probability where probabilities are objective properties of the real world, referring to limiting relative frequencies. Model parameters are considered fixed, unknown constants.
Bayesian Probability An interpretation of probability that describes degrees of belief. Model parameters are treated as hidden random variables, and one can compute their probability distribution given data, P(θ \| data).