Model Selection

• Model Selection
  • Model selection generally plays a larger role in log-linear model than in other methods.
  • The saturated model for the log-linear model is a sensible model: it simply fits the table frequencies exactly. Therefore, all other models can be seen as attempts to derive the essential "structure" of the table without having to fit it exactly. If you can find a table with acceptable fit relative to the saturated model, then you have succeeded in this task.
  • Almost any model will be preferred to the null model. A more interesting comparison here is between models which fit some association and the independence model which assumes all of the action is in the marginal distributions.
  • For nested models, we can use our $LRT$ test to compare models. We can also compare any two models using the $BIC$ statistic.
  • The saturated model has $df=0$ and $G^{2}=0$, so it is always easy to compare any model to the saturated model using the $LRT$ test.

    The $LRT$ test versus the saturated model is just given by $\chi^{2}(df_{c},G_{c}^{2})$, where $c$ is the current model. In other words, for a model to be acceptable by the $LRT$ test, it must be reasonable that the model's overall deviance is a result of random chance.

    If two models are nested, their fit can be directly compared with $\chi^{2}(df_{1}-df_{2},G_{1}^{2}-G_{2}^{2})$, where model 2 is nested within model 1. Essentially, we need to show that the reduction in deviance resulting from the addition of parameters could not have resulted just by random chance.

    Let's look at how well different models fit to our mobility table.

    Model	Deviance ($G^{2}$)	DF	BIC
    Perfect Mobility	1425.3	16	1281.2
    Quasi-Perfect Moblity	151.6	11	52.5
    Corners	26.3	7	-36.8
    Uniform Association	484.7	15	349.6

    All of these models are nested within the perfect mobility model and it is clear at a glance that they are all a significant improvement over a model which assumes no association. For example, the LRT test between the perfect mobility and quasi-perfect mobility model is given by
    $$1425.3-151.6=1273.7$$
    The QPM model reduced the deviance by 1273.7 with the addition of five parameters accounting for the diagonal. This clearly is not a result of a sampling variability.

    The corners model is nested within the QPM model so we can also directly compare these two. The difference in deviance is $151.6-26.3=25.3$ with the addition of 4 parameters. On a $\chi^{2}$ test, the probability of observing a drop in the deviance of that size due to sampling variability is 0.000043. So the corners model seems like a clear improvement.

  • The problem with this technique is that in large samples it is often not possible to find models preferred to the saturated model because any deviation will be significant. For example, we could compare the Corners model to the saturated model which has a deviance of 0 and 0 degrees of freedom. The probability of getting a deviance of 26.3 or higher on 7 degrees of freedom is 0.00045. So by traditional tests, none of these models are preferred to the saturated model - even if we find them more conceptually appealing.

  • A second problem is that the $LRT$ test cannot technically be used to compare non-nested models. The Corners and Uniform Association model, for example, are not nested so they cannot be compared directly with $LRT$
  • The $BIC$ can solve both of these problems. Because $BIC$ is more parsimonious, it will often prefer models to the saturated model that $LRT$ will not. It can also be compared across non-nested models. Underlying this distinction are some fairly deep differences in the way inference is conceptualized that are beyond our scope.
  • Remember that $BIC$ without the apostrophe is measured relative to the saturated model. This is the most logical form of $BIC$ for log-linear models. The formula is:
    $$G^{2}-(df)\log n$$
    Negative $BIC$ indicates that the current model is preferred to the saturated model. Two models can be compared directly by a comparison of their $BIC$ scores.

    Looking at the $BIC$ scores for our mobility models, it is clear that the only one preferred to the saturated model is the Corners model which seems to clearly be the best model overall.