Model Selection

• When two models are ``nested", we can formally compare them using the F-test.
  • Recall the F-test for one model, The null hypothesis is that none of the independent variables are related to the dependent variable (all $B_{j}$=0).
    $$F=MSM/MSE$$
    If the null hypothesis is true, then this term is distributed as an F-distribution with $p$ DF in the numerator and $n-p-1$ in the denominator.
  • Essentially the F-test is a test of the current model vs. the null model. We can also use the F-test to compare two non-null models. Consider two models, where model 2 is nested within model 1. then:
    $$F=\frac{(SSE_{1}-SSE_{2})/g}{SSE_{1}/(n-k-1)}$$
    Where $g$ is the number of parameters added in model 2 and $k$ is the number of parameters in model 1. Under the null hypothesis that none of the variables added in the second model is related to the dependent variable, this term is distributed as an F-distribution with $g$ DF in the numerator and $n-k-1$ in the denominator.
  • If model 2 adds only a single parameter then this F-test is equivalent to simply looking at the p-value for this new variable in model 2.
  • this technique is not reliable in deciding among a possible set of predictors because the effect size and standard error of each variable will differ depending on what other variables are included. There is an uncertainty about the appropriate model that we need to account for.

• We are often interested in a comparison between models, or in including the right control variables to get the best estimate of the net effect of a particular independent variable. Model selection plays a large role in social science research - where is the right model between the null model
  $$y_{i}=\beta_{0}+\epsilon_{i}=\mu_{y}$$
  and the saturated model?

• Example: state-level crime data from Ehrlich (1973). The question of interest is whether rational incentives like the probability of being caught and the length of imprisonment affect crime rates. In order to assess the effects of these variables we want a model that can control for other important predictors of crime rates. Our candidate variables are:
  1. percent of males aged 14-24
  2. south indicator variable
  3. mean years of schooling
  4. police expenditures in 1960
  5. police expenditures in 1959
  6. labor force participation rate
  7. M/F sex ratio
  8. population
  9. nonwhites per 1000 population
  10. unemployment rate of urban males 14-24
  11. unemployment rate of urban males 35-39
  12. GDP
  13. income inequality
  14. probability of imprisonment
  15. average time served in state prisons
  The full model with all variables and Ehrlich's chosen model are shown in the table. It is clear that the effects are much smaller in the full model than in the model Ehrlich chose, which makes us wonder if his effects are measured using the right model.

• We assess the goodness of fit of the OLS model generally by $R^{2}$, but we have to balance goodness-of-fit with parsimony.
• We can always get a better fit (or at least not a worse fit) by adding more variables to the model, so $R^{2}$ is not helpful in and of itself. One technique for balancing parsimony is to use the adjusted $R^{2}$:
     adj.$$R^{2}=1-(\frac{n-1}{n-p-1})(\frac{SSE}{SST})$$
  The expression before the sum of squares ratio will always be less than one when k is greater than zero, so this expression "penalizes" $R^{2}$ for including more variables into the model. We could try choosing the model which maximizes our adjusted $R^{2}$.

• Another technique is to proceed in a stepwise fashion. There are three basic approaches:
  1. Forward Selection. Begin with the variable with the highest single $R^{2}$ value. Next search for the variable to add which will lead to the next greatest increment in $R^{2}$, and continue this process until all possible variables are added or some stopping criterion is reached (typically that the next added value is not stat. sig. at some level).
  2. Backward Selection. Begin with the full model. Remove one variable based on some criterion (typically the smallest t-value). Continue this process until all remaining variables are above some pre-established threshold.
  3. Stepwise Selection. Begin like forward selection, but at the end of each step, remove any variables that have fallen below a certain threshold (typically on the t-value) before moving forward another step.
  • These methods can sometimes give different results.
  • Elimination is purely on statistical significance.
  • What about things like interactions or polynomial terms?

  
  
  

• Bayesian Information Criterion (BIC)
  • The mathematics behind BIC are fairly complex, but it is fairly intuitive to understand in practice - In addition it can be applied to a wide range of linear models besides OLS regression with the same interpretation, making it very versatile.
  • For OLS regression, BIC' (the punctuation indicates our comparison is the null model) is calculated as:
    $$BIC'=n \log(1-R^{2})+p \log(n)$$
    This equation balances parsimony and goodness-of-fit.
  • $BIC'$ is implicitly compared to the null model. The smaller $BIC'$ is, the better. If $BIC'<0$, then the current model is preferred over the null model.
  • $BIC'$ can also be used to compare two non-null models, even if they are non-nested. The model with a smaller (generally, more negative) $BIC'$ is preferred.

• The best approach depends on what you are doing with the model.
  • Its best to have a theoretical reason for fitting the model as you are, rather than just throwing in variables randomly and seeing what happens (monkeys and computers can do that-you are supposed to be the brains of the operation)
  • Generally we are interested in a particular variable or a small set of variables and the others are included as controls. Even if these controls look not statistically distinguishable from zero, it may still be good to include them in the equation. Show the full model and let your readers make their own decisions.
  • In some social science research, the key issue is a comparison of models which present competing explanations of the process. In these cases, model selection is very important, and you should carefully consider your options. Generally it is best here to present several techniques.