Summation Algebra for Regression

Every OLS proof is really a few summation moves repeated. If those moves are automatic, the derivations stop looking like magic. Here are the only rules you need, then the three identities that do all the work.

The basic rules

  • A constant summed \(n\) times: \(\sum_{i=1}^{n} c = nc\).
  • Pull constants out: \(\sum_i a x_i = a \sum_i x_i\).
  • Split a sum: \(\sum_i (x_i + y_i) = \sum_i x_i + \sum_i y_i\).
  • The mean, rearranged: \(\sum_i x_i = n\bar{x}\) (this is just the definition \(\bar{x} = \frac{1}{n}\sum_i x_i\) turned around). Use it constantly.

The three identities that do the work

1. Deviations from the mean sum to zero.

\[\sum_i (x_i - \bar{x}) = \sum_i x_i - n\bar{x} = n\bar{x} - n\bar{x} = 0.\]

This is why \(\sum_i e_i = 0\) falls out of the normal equations, and why the fitted line runs through \((\bar{x}, \bar{y})\).

2. Sum of squared deviations has a shortcut.

\[\sum_i (x_i - \bar{x})^2 = \sum_i x_i^2 - n\bar{x}^2.\]

This is the denominator \(SST_x\) of the slope, and the same trick gives the sample variance.

3. The cross-term shortcut.

\[\sum_i (x_i - \bar{x})\,x_i = \sum_i (x_i - \bar{x})^2, \qquad \sum_i (x_i - \bar{x})\,y_i = \sum_i (x_i - \bar{x})(y_i - \bar{y}).\]

Both hold for the same reason: subtracting a constant (\(\bar{x}\) or \(\bar{y}\)) times \(\sum_i (x_i - \bar{x})\) subtracts zero, by identity 1. This is the step that turns the raw slope formula into \(\hat{\beta}_1 = S_{xy}/S_{xx}\).

Why these three

  • Identity 1 is what makes residuals and deviations “balance.”
  • Identity 2 is the denominator of every variance and slope.
  • Identity 3 is what lets you center a sum for free, which is how \(\hat{\beta}_1 = S_{xy}/S_{xx}\) appears in the derivation.

Once you can expand and cancel these in your sleep, the unbiasedness and variance proofs are just these moves in sequence.

Check them on real numbers

Pick an identity. The table shows every term for a small dataset; the box underneath adds the two sides up and confirms they match. Numbers first, symbols second.

#| standalone: true
#| viewerHeight: 520

library(shiny)

x <- c(2, 5, 3, 8, 6, 1)
y <- c(4, 7, 5, 9, 8, 3)
n <- length(x); xbar <- mean(x); ybar <- mean(y)

ui <- fluidPage(
  tags$head(tags$style(HTML("
    .eq-box { background:#eafaf1; border:1px solid #cdeed9; border-radius:6px;
              padding:12px 14px; margin-top:12px; font-size:14px; line-height:1.7; }
    .eq-box b { color:#1e7a46; }
  "))),
  sidebarLayout(
    sidebarPanel(
      width = 4,
      selectInput("id", "Identity to check:",
        choices = c(
          "1.  sum(x - xbar) = 0"                          = "one",
          "2.  sum((x - xbar)^2) = sum(x^2) - n*xbar^2"    = "two",
          "3.  sum((x - xbar)*y) = sum((x - xbar)*(y - ybar))" = "three")),
      helpText(sprintf("Data:  x = {%s},  xbar = %.2f", paste(x, collapse = ", "), xbar)),
      uiOutput("verdict")
    ),
    mainPanel(
      width = 8,
      tableOutput("tab")
    )
  )
)

server <- function(input, output, session) {

  output$tab <- renderTable({
    if (input$id == "one") {
      data.frame(i = 1:n, x = x, `x - xbar` = round(x - xbar, 2), check.names = FALSE)
    } else if (input$id == "two") {
      data.frame(i = 1:n, x = x,
                 `(x - xbar)^2` = round((x - xbar)^2, 2),
                 `x^2` = x^2, check.names = FALSE)
    } else {
      data.frame(i = 1:n, x = x, y = y,
                 `(x - xbar)*y` = round((x - xbar) * y, 2),
                 `(x - xbar)*(y - ybar)` = round((x - xbar) * (y - ybar), 2),
                 check.names = FALSE)
    }
  }, digits = 2, striped = TRUE, width = "100%")

  output$verdict <- renderUI({
    if (input$id == "one") {
      lhs <- sum(x - xbar)
      msg <- sprintf("sum(x - xbar) = %.2f  &rarr;  0 &check;", lhs)
    } else if (input$id == "two") {
      lhs <- sum((x - xbar)^2); rhs <- sum(x^2) - n * xbar^2
      msg <- sprintf("left = %.2f &nbsp;&nbsp; right = %.2f  &rarr;  equal &check;", lhs, rhs)
    } else {
      lhs <- sum((x - xbar) * y); rhs <- sum((x - xbar) * (y - ybar))
      msg <- sprintf("left = %.2f &nbsp;&nbsp; right = %.2f  &rarr;  equal &check;", lhs, rhs)
    }
    tags$div(class = "eq-box", HTML(paste0("<b>", msg, "</b>")))
  })
}

shinyApp(ui, server)

Identity 3 is the one to sit with: the two columns hold different numbers term by term, yet the column totals are identical — because the pieces that differ sum to zero by identity 1.

Connections