Generalized Linear Models

• Our OLS regression equation is:
  $$y_{i}=\beta_{0}+\beta_{1}x_{i1}+\beta_{2}x_{i2}+\ldots+\beta_{p}x_{ip}+\epsilon_{i}$$

• For this model, we assume:
  1. The relationship between the independent variables and dependent variables is a linear function.
  2. The errors are distributed identically with a constant variance.
• These assumptions become problematic for categorical dependent variables because:
  1. Predictions can fall outside the range of the of the dependent variable (linearity problem)
  2. The variance of the error term is often related to the expected value ( $\hat{y_{i}}$) for each observation (true for both binomial and poisson distributed variables), meaning we have non-constant variance, or heteroskadasticity (i.i.d. problem). To put it more formally:
     $$V(y\vert x)=V(\hat{y})=f(\hat{y})$$

• The generalized linear model deals with this problem by two extensions of the linear regression model.
  1. We have a link function that relates our linear function of the independent variables not to $y_{i}$ directly, but rather to some function of $y_{i}$, given by $g(y_{i})$. The idea of the link function is to expand the range of $y$ from some limited range to the range of $(-\infty,\infty)$.
     • Take our titanic example. We are interested in modeling $p_{i}$, each individual's probability of survival. However, $p_{i}$ is bounded by 0 and 1.
     • We can improve this by looking at the odds of survival rather than the probability of survival. (we will discuss odds in more depth in the next module)
       $$o_{i}=\frac{p_{i}}{1-p_{i}}$$

     • While probabilities are bounded by 0 and 1, odds are bounded by 0 and $\infty$. This is an improvement, but we still could get results outside of the lower bounds. We can resolve this problem by logging the odds:
       logit$$(p_{i})=\log(o_{i})=\log(p_{i}(1-p_{i}))$$

     • This is called the logit transformation. Rather than related our x variables linearly to $p_{i}$, we relate them linearly to the logit transformation of $p_{i}$.
  2. Rather than assume i.i.d., we specify a particular error distribution for our errors. This distribution is based on what we think is driving the categorical distinctions we observe in $y$. (i.e. binomial for dichotomous variables)
     • Once again from our example. We believe each observation has an underlying $p_{i}$, but we actually only observe ones and zeroes (success and failures). Therefore, the error distribution from our predicted to actual values will be given by the binomial/bernoulli distribution.
       $$y_{i}$$    is given by binom$$(p_{i})$$
       $$V(p_{i})=p_{i}(1-p_{i})$$

• Once we do this we have a slightly different form for the equation:
  $$g(p_{i})=\beta_{0}+\beta_{1}x_{i1}+\beta_{2}x_{i2}+\ldots+\beta_{p}x_{ip}$$

• Generally, certain link and error functions go together.

  Type of category	link function	error dist.	name
  continuous	identity	normal/gaussian	OLS regression
  dichotomous	logit	binomial	logistic/logit regression
  	probit	binomial	probit regression
  discrete counts	log	poisson	log-linear models/possion regression
  polytomous (unordered)	logit	multinomial	multinomial logit
  polytomous (ordered)	logit	multinomical	ordered logit
  survival times/counts of events	log	poisson	continuous-time hazard model

  (discuss each of the items on the table)