From Normal to t

Every \(p\)-value and confidence interval you will compute rests on a small family of distributions built out of the normal. Textbooks usually define them in one line each and move on. Here is where they come from and why the \(t\) shows up the moment you run a regression.

Why you need anything past the normal

To test the slope you standardize it:

\[\frac{\hat{\beta}_1 - \beta_1}{\mathrm{se}(\hat{\beta}_1)}.\]

If you knew the error SD \(\sigma\), this would be a standard normal. But you don’t — you estimate it with \(s = \sqrt{SSR/(n-2)}\). Dividing by an estimated standard deviation adds a second source of wobble on top of the one in \(\hat{\beta}_1\), and that extra wobble gives the ratio fatter tails than the normal. That fatter-tailed distribution is the \(t\).

The three building blocks

  • Standard normal \(Z\) — mean 0, SD 1. The starting material.
  • Chi-square, \(\chi^2_k\) — the sum of \(k\) independent squared standard normals, \(Z_1^2 + \dots + Z_k^2\). Squaring makes it positive and right-skewed. This is what a sum of squared residuals follows: \(SSR/\sigma^2 \sim \chi^2_{n-2}\).
  • \(t_k\) — a standard normal divided by the square root of an independent, scaled chi-square: \(t_k = Z / \sqrt{\chi^2_k / k}\). That denominator is the “I had to estimate \(\sigma\)” wobble. When \(k\) is large the denominator settles near 1 and \(t_k\) becomes the normal.
  • \(F\) — a ratio of two scaled chi-squares, \(F = (\chi^2_{k_1}/k_1) / (\chi^2_{k_2}/k_2)\). It runs the joint tests (several coefficients at once); a squared \(t\) is just an \(F\) with one numerator degree of freedom.

Why degrees of freedom change the shape

Small \(k\) means the chi-square in the denominator of \(t\) is itself noisy, so the ratio swings more — fat tails, and critical values well above 1.96. As \(k\) grows, \(s\) homes in on \(\sigma\), the denominator stops wobbling, and the \(t\) collapses onto the standard normal.

The payoff: the regression \(t\)-test

Put the pieces together and

\[\frac{\hat{\beta}_1 - \beta_1}{\mathrm{se}(\hat{\beta}_1)} \sim t_{\,n-2}.\]

The \(n-2\) is the degrees of freedom of the \(SSR\) chi-square — the same \(n-2\) from estimating \(\sigma^2\). That is why regression output uses the \(t\) with \(n-2\) df, and why in a small sample the cutoff for significance is noticeably bigger than 1.96.

See it

Slide the degrees of freedom. At \(df = 1\) the \(t\) (red) has heavy tails and a two-sided cutoff far past 1.96; as \(df\) grows it settles onto the standard normal (black) and the cutoff falls toward 1.96.

#| standalone: true
#| viewerHeight: 500

library(shiny)

ui <- fluidPage(
  tags$head(tags$style(HTML("
    .eq-box { background:#f0f4f8; border-radius:6px; padding:14px;
              margin-top:14px; font-size:14px; line-height:1.8; }
    .eq-box b { color:#2c3e50; }
  "))),
  sidebarLayout(
    sidebarPanel(
      width = 3,
      sliderInput("df", "Degrees of freedom (n - 2):", min = 1, max = 60, value = 3, step = 1),
      helpText("Small df = fat tails. As df grows, the t curve approaches the standard normal."),
      uiOutput("box")
    ),
    mainPanel(
      width = 9,
      plotOutput("dens", height = "420px")
    )
  )
)

server <- function(input, output, session) {

  output$dens <- renderPlot({
    df <- input$df; par(mar = c(4.5, 4.5, 3, 1))
    xs <- seq(-5, 5, length.out = 400)
    plot(xs, dnorm(xs), type = "l", lwd = 2, col = "#2c3e50", ylim = c(0, 0.42),
         main = sprintf("t with %d df  vs  standard normal", df),
         xlab = "value", ylab = "density")
    lines(xs, dt(xs, df), lwd = 2.5, col = "#e74c3c")
    tc <- qt(0.975, df)
    abline(v = c(-tc, tc), col = "#e74c3c", lty = 2)
    abline(v = c(-1.96, 1.96), col = "#2c3e50", lty = 3)
    legend("topright", bty = "n", cex = 0.85,
           legend = c("standard normal", paste0("t (", df, " df)"),
                      "t 97.5% cutoff", "normal 1.96"),
           col = c("#2c3e50", "#e74c3c", "#e74c3c", "#2c3e50"),
           lwd = c(2, 2.5, 1, 1), lty = c(1, 1, 2, 3))
  })

  output$box <- renderUI({
    df <- input$df; tc <- qt(0.975, df)
    tags$div(class = "eq-box",
      HTML(sprintf(
        "<b>Two-sided 97.5%% cutoff</b><br>t (%d df): <b>%.3f</b><br>normal: 1.960<br><br>Extra width from<br>estimating &sigma;: %.3f",
        df, tc, tc - 1.96)))
  })
}

shinyApp(ui, server)

At \(df = 1\) the cutoff is enormous (over 12); by \(df = 30\) it is about 2.04, already close to 1.96. That shrinking gap is the price of not knowing \(\sigma\) — a price that fades as the sample grows.

Connections