Chapter 5 Notes on OLS

Why the constant term?

The SLR model is a population model. When it comes to estimating \(\beta_1\) (and \(\beta_0\)) using a random sample of data, we must restrict how \(u\) and \(x\) are related to each other.

What we must do is restrict the way \(u\) and \(x\) relate to each other in the population.

Taking the simple linear regression (SLR) model as given, \[y = \beta_0 + \beta_1x + u\], we will make a simplifying assumption: \[E(u)=0\] That is, the average value of \(u\) is zero in the population.

Note then, that it is the presence of \(\beta_0\) in \(y = \beta_0 + \beta_1x + u\) that allows us to assume \(E(u) = 0\) without loss of generality.

If the average of \(u\) was different from zero, say \(\alpha_0\), we would just adjust the intercept, which leaves the slope unchanged: \[y = (\beta_0 + \alpha_0) + \beta_1x + (u-\alpha_0)\]
The new error is: \(u - \alpha_0\)
The new intercept is: \(\beta_0 + \alpha_0\)

Importantly, note that what we care about knowing (i.e., \(\beta_1\)) is invariant to \(\alpha_0\)

What matters more, though…

… what matters for the inference we can make is that the mean of the error term does not change across different values of \(x\) in the population: \[E(u | x)=E(u) \hspace{2mm}\forall \hspace{2mm} x\]

If \(E(u | x)=E(u)\) for all \(x\), then \(u\) is mean independent of \(x\).

For example

Suppose you are interested in some \[Income=\beta_0 + \beta_1(schooling) + u\]

In \(u\) would be “ability,” then, yes? Mean independence would require that, \[E(ability | x = 8) = E(ability | x = 12) = E(ability | x = 16)\]

That is, the average ability should be the same across levels of education.

But, surely people choose education levels partly based on ability, yes? (We’ll return to this.)

Combining \(E(u | x) = E(u)\) (the substantive assumption) with \(E(u) = 0\) (a normalization) gives \[E(u | x)=0 \hspace{2mm}\forall \hspace{2mm} x.\]

We refer to this as the zero conditional mean assumption.
Because the conditional expected value is a linear operator, \(E(u | x) = 0\) implies \[E(y | x)= \beta_0 +\beta_1 x,\] which implies that the conditional expectation function (or population regression function) is a linear function of \(x\).

How can we estimate the population parameters, \(\beta_0\) and \(\beta_1\)?

Let \(\{f(x_i ; y_i ) : i = 1, 2, ... ; n\}\) be a random sample of size \(n\) (the number of observations) from the population.
Plug any \(i^{th}\) observation into the population equation: \[y_i = \beta_0 + \beta_1x_i + u_i,\] where the \(i\) subscript indicates a particular observation.
Note: We observe \(y_i\) and \(x_i\), but we do not observe \(u_i\) (though we know it is there).

To obtain estimating equations for \(\beta_0\) and \(\beta_1\), use the two population restrictions: \[E(u) = 0\] \[Cov(x,u) = 0\]

We talked about the first condition. The second condition, that the covariance is zero, means that \(x\) and \(u\) are uncorrelated. Both conditions are implied by \(E(u | x) = 0\).

Fitted values: For any candidates \({\hat \beta}_0\) and \({\hat \beta}_1\), define the fitted value for each \(i\) as \[{\hat y}_i = {\hat \beta}_0 + {\hat \beta}_1x_i\] There are \(n\) of these fitted values… the values we predict for \(y_i\) given \(x_i\). \end{frame}

Residuals: The “mistakes” we make in predicting \(y_i\) is the residual, \[{\hat u}_i = y_i - {\hat y}_i = y_i - ({\hat \beta}_0 + {\hat \beta}_1x_i)\]

We want to track both overshooting and undershooting \(y\). So… we measure the size of the “mistakes” across all \(i\) in the sample by first squaring them, and then summing across \(i\):\[\sum_{i=1}^{n}{\hat u}_i^2 = \sum_{i=1}^{n}(y_i-{\hat \beta}_0 - {\hat \beta}_1x_i)^2\]

The OLS estimates are those \({\hat \beta}_0\) and \({\hat \beta}_1\) that minimize the sum of squared residuals (SSR).

Note… \({\hat \beta}_1\) is an estimate of the parameter \(\beta_1\), obtained for a specific sample.

Different samples will generate different estimates (of true \(\beta_1\)).

Unbiasedness: If we have an unbiased estimator, we could take as many random samples from a population as we wanted to, and the average (mean) of all the estimates from each of those samples would be equal to the true \(\beta\).

Zero Conditional Mean Assumption: In the population, the error term has zero mean given any value of the explanatory variable: \[E(u | x) = E(u) = 0\]

This is the key assumption implying that OLS is unbiased, with the “zero” not being important once we assume \(E(u | x)\) does not change with \(x\).

The problem with OLS?

The problem: We can compute OLS estimates whether or not the zero conditional mean assumption (i.e., \(E(u | x) = E(u) = 0\)) holds.

The bigger problem: We think the zero conditional mean assumption is a little hard to swallow.

Consider Anscombe’s Quartet, for example… four distinct data-generating processes, each yielding the same coefficient estimates in OLS (in this case, \(y=3+0.5x\)). In no way has the causal effect of \(x\) been identified in all four scenarios.

Assumed in each: \(n=11, \mu_x=9, \mu_y=7.5, \sigma_x=11, \sigma_y=[4.122,4.127], \sigma_{xy}=0.816\)

Omitted-variable bias

Consider another example: \[ \mathbb{1}(working_i) = \beta_0 + \beta_1 numkids_i + u_i\]

If family size is random, then \(Cov(numkids_i, u_i)=0\), in which case \({\hat \beta}_1\) is the causal effect of \(numkids\) on \(working\).

Q: How do we interpret \({\hat \beta}_1\) if \(numkids\) is non-random? That is, what if \(Cov(numkids_i, u_i)\ne 0\)?

A: \({\hat \beta}_1\) is biased, where the sign if the bias is determined by \(Cov(x_i, u_i)\) and \(Cov(u_i, y_i)\).

The rule:

If \(Cov(u_i, y_i)\) & \(Cov(x_i, u_i)\) are similar in sign, the bias is positive.

If \(Cov(u_i, y_i)\) & \(Cov(x_i, u_i)\) are opposite in sign, the bias is negative.

An example

Q: What if 1) those with more kids also tend to be married (\(Cov(numkids_i, married_i)>0\)) and, 2) married people tend to be more likely to work (\(Cov(married_i, working_i)>0\))?

A: \({\hat \beta}_1\) is biased upward. (The effect of \(married\) is loading onto \(numkids\).)

Q: What if 1) black families tend to have more kids (\(Cov(numkids_i, black_i)>0\)) and 2) black employment tends to be lower (\(Cov(black_i, working_i)<0\))?

A: \({\hat \beta}_1\) is biased downward. (The effect of \(black\) is loading onto \(numkids\).)

So, if family size varies non-randomly… can we get to a place where we “believe” our \({\hat \beta}_1\)? That is… be comfortable assuming that family size is conditionally random? What would you want to condition on?

Anything that correlates with \(numkids\) and with \(working\).

\[\mathbb{1}(working_i) = \beta_0 + \beta_1 numkids_i + \gamma_1 married_i + \gamma_2 black_i + u_i\]

Here, if we want to estimate the average causal effect of family size on labor supply, the identifying assumption is

\(Cov(numkids_i, u_i)= 0\), or,
\(Cov(numkids_i, u_i | \{married_i, black_i\}) = 0\)

OVB in one picture

An abstraction for sure, but sometimes helpful. Consider these shapes representative of variation, and overlap therefore co-variation. The source of OVB in attributing the overlap of \(Y\) and \(X_1\) entirely to \(X_1\), as though causal, is the three-way overlap of \(Y\), \(X_1\), and \(X_2\), having run the model \(Y=\beta_0 + \beta_1 X_1 + u\).

In these figures (Venn diagrams)

Each circle illustrates a variable.
Overlap gives the share of correlatation between two variables.
Dotted borders denote omitted variables.