Test of Significance

The EIC’s third output statistic is the probability that the results would have occurred by chance. This is calculated if the user knows the number of students that have been tested. The measure of statistical significance should always be interpreted cautiously, primarily because calculations of statistical significance are highly influenced by the number of students in a comparison. With large samples relatively small differences will be statistically significant; with small samples relatively large differences will not be significant. In contrast, the effect size and improvement index are not affected by the number of students in a comparison. They remain the same no matter how many students are included. As a result they are easier to compare from one situation to another and are often more useful for educational consumers.

Traditionally probability values of .05 or less have been considered statistically significant. However, when looking at results in real life settings and, especially with very large or very small samples, this value should only be considered a general point of reference. Users who are not familiar with the notion of probability should use this output statistic cautiously.