Sampling Distribution of OLS
The slope is a random variable
When you run reg y x, Stata hands back one number for the slope. It looks final. But that number is an accident of the particular sample you happened to draw. Draw a fresh sample from the same population and the line tilts to a slightly different angle; \(\hat{\beta}_1\) comes back a different number. The estimator \(\hat{\beta}_1\) is not a number — it is a recipe that turns whatever data you feed it into a slope. Because the data are random, its output is random too.
The sampling distribution of \(\hat{\beta}_1\) is the histogram you would get by running that recipe on every sample you could have drawn. Everything in this part of the course — unbiasedness, the variance formula, standard errors, \(t\)-stats — is a statement about that histogram. This page lets you build it.
Where the proof’s weights come from
Start from the slope formula you derived: \[\hat{\beta}_1 = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2}.\]
Because \(\sum_i (x_i - \bar{x})\,\bar{y} = 0\), the \(\bar{y}\) drops out and you can write the slope as a weighted sum of the \(y\)’s: \[\hat{\beta}_1 = \sum_i w_i\, y_i, \qquad w_i = \frac{x_i - \bar{x}}{SST_x}, \qquad SST_x = \sum_i (x_i - \bar{x})^2.\]
The weights \(w_i\) are fixed the moment you know the \(X\)’s — larger for points far from \(\bar{x}\), which is why outlying \(X\) values swing the line hardest. Now substitute the true model \(y_i = \beta_0 + \beta_1 x_i + u_i\). Two of the three pieces collapse (\(\sum_i w_i = 0\) and \(\sum_i w_i x_i = 1\)), leaving the one line that drives the whole of Topic 2: \[\hat{\beta}_1 = \beta_1 + \sum_i w_i\, u_i.\]
Read it in words: your estimate equals the truth plus a weighted average of the errors. Everything follows from here.
- Unbiasedness. Take expectations. With \(E(u\mid X) = 0\), each \(E(w_i u_i) = 0\), so \(E(\hat{\beta}_1) = \beta_1\). The estimate is centered on the truth — the histogram sits over \(\beta_1\).
- Variance. The only random part is \(\sum_i w_i u_i\). With independent, constant-variance errors, \(\mathrm{Var}(\hat{\beta}_1) = \sigma^2 \sum_i w_i^2 = \sigma^2 / SST_x\) — the width of the histogram.
Build it yourself
The simulator fixes a true line \(Y = 2 + 3X\) (so \(\beta_1 = 3\)), holds the \(X\)’s fixed, and redraws the errors \(u\) to generate one sample after another — exactly the “shuffling” from lecture, but you turn the knobs. The left panel shows a single sample: the true line (dashed) and the line OLS fits to that one draw (solid). They rarely coincide. The right panel collects the fitted slope from every sample into the sampling distribution.
Watch two things: the histogram centers on 3 (unbiasedness), and its width equals \(\sigma_u/\sqrt{SST_x}\) (the variance formula) — the red curve is that exact normal.
#| standalone: true
#| viewerHeight: 620
library(shiny)
ui <- fluidPage(
tags$head(tags$style(HTML("
.stats-box {
background: #eaf2f8; border-radius: 6px; padding: 14px;
margin-top: 12px; font-size: 14px; line-height: 1.8;
}
.stats-box b { color: #2c3e50; }
"))),
sidebarLayout(
sidebarPanel(
width = 3,
sliderInput("n", "Sample size (n):",
min = 5, max = 500, value = 250, step = 5),
sliderInput("sd_u", "Error SD (sigma_u):",
min = 1, max = 12, value = 6, step = 0.5),
sliderInput("sd_x", "Spread of X (SD of X):",
min = 1, max = 8, value = 3, step = 0.5),
sliderInput("reps", "Number of samples:",
min = 100, max = 3000, value = 1000, step = 100),
uiOutput("stats_box")
),
mainPanel(
width = 9,
fluidRow(
column(6, plotOutput("scatter_plot", height = "380px")),
column(6, plotOutput("sampling_plot", height = "380px"))
)
)
)
)
server <- function(input, output, session) {
b0 <- 2
b1 <- 3
# X is held fixed across repeated samples: we condition on the sample
# values X_1, ..., X_n. Changing n or the spread of X draws a fresh design.
xvals <- reactive({
rnorm(input$n, mean = 0, sd = input$sd_x)
})
sim <- reactive({
x <- xvals()
n <- input$n
sd_u <- input$sd_u
reps <- input$reps
xbar <- mean(x)
sst_x <- sum((x - xbar)^2)
# Redraw only the errors each replication; recompute the OLS slope.
slopes <- replicate(reps, {
y <- b0 + b1 * x + rnorm(n, 0, sd_u)
sum((x - xbar) * (y - mean(y))) / sst_x
})
list(x = x, slopes = slopes, n = n, sd_u = sd_u,
sst_x = sst_x, theo_se = sd_u / sqrt(sst_x), b0 = b0, b1 = b1)
})
# Left: one sample, true line vs fitted line
output$scatter_plot <- renderPlot({
s <- sim()
x <- s$x
y <- s$b0 + s$b1 * x + rnorm(s$n, 0, s$sd_u)
b1hat <- sum((x - mean(x)) * (y - mean(y))) / sum((x - mean(x))^2)
b0hat <- mean(y) - b1hat * mean(x)
par(mar = c(4.5, 4.5, 3, 1))
plot(x, y, pch = 19, col = "#6c8ebf80",
main = "One sample", xlab = "X", ylab = "Y")
abline(a = s$b0, b = s$b1, col = "#2c3e50", lwd = 2.5, lty = 2)
abline(a = b0hat, b = b1hat, col = "#e74c3c", lwd = 2.5)
legend("topleft", bty = "n", cex = 0.9,
legend = c(paste0("True line (slope ", s$b1, ")"),
paste0("Fitted line (slope ", round(b1hat, 3), ")")),
col = c("#2c3e50", "#e74c3c"), lwd = 2.5, lty = c(2, 1))
})
# Right: sampling distribution of the slope
output$sampling_plot <- renderPlot({
s <- sim()
par(mar = c(4.5, 4.5, 3, 1))
hist(s$slopes, breaks = 40, probability = TRUE,
col = "#dae8fc", border = "#6c8ebf",
main = paste0("Sampling Distribution of the Slope (n = ", s$n, ")"),
xlab = expression(hat(beta)[1]), ylab = "Density")
x_seq <- seq(min(s$slopes), max(s$slopes), length.out = 300)
lines(x_seq, dnorm(x_seq, mean = s$b1, sd = s$theo_se),
col = "#e74c3c", lwd = 2.5)
abline(v = s$b1, lty = 2, lwd = 2, col = "#2c3e50")
legend("topright", bty = "n", cex = 0.9,
legend = c("Normal approximation", "True slope"),
col = c("#e74c3c", "#2c3e50"),
lwd = c(2.5, 2),
lty = c(1, 2))
})
output$stats_box <- renderUI({
s <- sim()
tags$div(class = "stats-box",
HTML(paste0(
"<b>True slope β<sub>1</sub>:</b> ", s$b1, "<br>",
"<b>Mean of β̂<sub>1</sub>:</b> ", round(mean(s$slopes), 4),
" <small>(unbiasedness)</small><br>",
"<hr style='margin:8px 0'>",
"<b>Theoretical SE (σ<sub>u</sub>/√SST<sub>x</sub>):</b> ",
round(s$theo_se, 4), "<br>",
"<b>Observed SD of β̂<sub>1</sub>:</b> ",
round(sd(s$slopes), 4)
))
)
})
}
shinyApp(ui, server)Three levers on the variance
The formula \(\mathrm{Var}(\hat{\beta}_1) = \sigma^2 / SST_x\) has exactly three moving parts, and each is a slider:
| Slider | Effect on the histogram | Why |
|---|---|---|
| Error SD \(\sigma_u\) ↑ | wider | noisier errors (\(\sigma^2\) up) → harder to see the slope |
| Sample size \(n\) ↑ | narrower | more points → \(SST_x\) grows → more information |
| Spread of \(X\) ↑ | narrower | \(X\)’s more spread out → \(SST_x\) grows → line better pinned |
The last one is the least obvious and the most useful: you learn a slope best when your \(X\)’s are far apart. Two points at the ends of the range pin a line down better than a hundred bunched in the middle.
Things to try
- Defaults (\(n=250\), \(\sigma_u=6\), spread \(=3\)): the mean of \(\hat{\beta}_1\) lands on ~3.00, and the observed SD matches the theoretical SE almost exactly.
- Crank \(\sigma_u\) to 12: the histogram roughly doubles in width. That is noise, not bias — it is still centered on 3.
- Drop \(n\) to 20: wide and lumpy. The slope is barely identified from so few points.
- Push spread of \(X\) from 1 to 8 at fixed \(n\): the histogram tightens sharply, even though nothing about the errors changed. That is \(SST_x\) at work.
- Any setting: the mean of \(\hat{\beta}_1\) never leaves 3. Unbiasedness does not care about \(\sigma_u\), \(n\), or the spread of \(X\) — only about \(E(u\mid X) = 0\).
From here
- The assumptions behind all of this — SLR.1–SLR.4 for unbiasedness, SLR.5 for the variance formula — are on the Gauss-Markov page.
- The algebra that produces \(\hat{\beta}_1 = S_{xy}/S_{xx}\) in the first place is on The Algebra Behind OLS.
- In practice \(\sigma_u\) is unknown. Replacing it with an estimate — and the \(n-2\) degrees of freedom that makes that estimate unbiased — is what turns the theoretical SE above into the standard error Stata prints next to your slope.