Measuring Success, Chapters 4 and 5

Chapter 4 was a case study from one school that used merit scholarships to boost enrollment by good students, and included SAT scores in the mix.  They focused more on enrollment than performance after enrollment, so I ignored it.

Chapter 5 is on "discrepant" students.  We've all known This One Person who got bad test scores and good grades, and This One Other Person who got good scores and bad grades.  The single most important thing any academic can consider in any conversation on this topic is whether a new rule would be fair to This One Person and This One Other Person.  But for the bad people (like me) who want to look beyond This One Person, chapter 5 looks at college performance on the large scale, not just the anecdotes.  And it appears that the people with better high school grades than scores have outcomes that are almost as good as those of people with similar grades.

On the surface, this would seem to undermine the case for using test scores.  However, these people have scores that are SUBSTANTIALLY worse than their high school grades might suggest.  If you just look at people with test scores that are within the normal range for people with those grades, differences in performance between people with similar grades but different scores (albeit not outrageously different scores) are indeed correlated with grades.

Interestingly, the people with substantially better grades than scores tend to be women and minorities.  Again, at first glance this might seem to lead to a slam-dunk case against tests, but (1) we're talking about the outliers, not the people in the normal band (i.e. there are still plenty of women and minorities whose test scores are not discrepant, and the test scores continue to have predictive power for them) and (2) while women and minorities are more likely to have test scores that are substantially worse than high school grades than the other way around, the situation seems to reverse if we look at college grades for women and minorities in the "normal" (non-discrepant) band of grades and scores.  So, complicated things are complicated.

On the other hand, people with poor grades but good scores do somewhat worse than people with similar scores (not surprising), because "smart but lazy" is a thing. (On the other hand, sometimes people with bad scores deliberately take easy classes, and sometimes people with good scores deliberately take hard classes.  Complicated things are complicated.) Again, on the surface this would suggest that only grades matter, but we're talking about outliers.  If you look at people who are closer to the normal range, differences in scores still have predictive power.

Interestingly, there's some evidence that people with better scores than grades tend to go into harder majors than people with better grades than scores, and this confounds some of the analysis.  This is not surprising to me; STEM does have some epically smart but lazy people.  (Yes, I'm sure that somewhere out there is a supremely lazy literature major with a perfect SAT score and horrible grades, but those people are more commonly STEM majors.)

My main take-aways are:

1) Test scores matter but they aren't the ONLY things that matter.  (Duh.)  This point has been agreed on by just about everyone who's ever suggested using test scores for decisions.  Yes, I'm sure that somewhere out there is a literal straw man who has suggested eliminating grades from consideration and ONLY looking at scores, but that guy (and you just know it's a guy) is ignored by everyone else.

2) The studies in this chapter are mostly based on small samples and so should be interpreted with caution.  We should probably err on the side of rewarding work, while not completely ignoring tests.