American Idyll, Chapter 1: Meritocracy

 Chapter 1 is written as a summary of history and thinkers. She covers a lot of things I've heard before, including standardized tests and Hofstadter's critiques of anti-intellectualism. I could summarize everything she's said, but that would be a lot of space (she packs a lot in) and frankly I don't take issue with much. The meta issue is that she is critiquing meritocracy as a technocratic hierarchy. I have more than enough issues with that, so I don't disagree. But meritocracy is an idea with two very different usages, subject to two different critiques. Besides an overall societal hierarchy based on technocracy, "meritocracy" also refers to hiring workers or selecting students based on qualifications for the particular position in question. Merit for the job is narrower than merit to run society. Being qualified to study a particular subject at a particular level is different from being qualified to enter the leadership class.

Liu critiques standardized tests alongside efforts to make education more "relevant" that Hofstadter critiqued. I'm not sure that the two things really belong together for critique. Sure, testing can be combined with tracking to give some people a less "academic" and more vocational education, while others get more traditional liberal arts or whatever. Whatever the advantages or disadvantages of such tracking, when you DON'T track you wind up having to dumb things down. (Ask me how I know.) Testing and tracking is the best hope for offering rigorous education at all. Otherwise we have to go to the lowest common denominator.

I'm not sure that I'm seeing how Liu will tie these things into anti-elitism. Testing is elitism.

Now, Liu does note that if we make teachers accountable for test scores then we can wind up stripping the richness from education and make it into desperate attempts to boost scores. That ultimately weakens intellectualism and is a form of anti-elitism. But testing is also one of the few tools we have to actually preserve some truly academic education. Educators who know their students are ready can demand more than educators who fear that their students aren't ready. (Ask me how I know.)

Now, Liu does make the point that one of the early goals of testing was to smash a hereditary elite by opening it to smart kids of every background, e.g. Columbia opening up to children of immigrants instead of traditional WASPs. That's a type of anti-elitism, but also a new type of elitism.

This is NOT a test! I repeat: This is NOT a test!

The University of California, possibly in collaboration with the California State University, as well as anyone else who would like some roasted unicorn with a side order of rainbows and fairies, is going to devise its own admissions tests.  These tests will be totally better than the SAT/ACT, will do a better job of predicting college performance, and will NOT perpetuate inequities, because they will only test what REALLY matters.  So all groups will score equally well and all of our problems will be solved.

These tests will be made from scratch and available by 2025 (really, 2024 if they want to admit freshmen freshpersons people of freshness with it in the fall of 2025). Whereas so many other educational measures reflect disparities in American society, these tests will not show disparities, and so they will be much fairer than the SAT/ACT, and will get buy-in from all ethnic groups.  (Except for Asian Americans, who are doing great on the SAT/ACT and will lose spots from this.  But I'm told that Asians aren't diverse, so I guess that's OK.)

Look, if it is true that a test of what REALLY matters for academic performance is blind to all of the disparities and inequities in our k-12 system (not to mention the wider societal context in which k-12 students are reared and prepared) then those equities don't actually matter.  Years of under-preparation?  Irrelevant! None of it mattered.  The preparation that other kids got wasn't an advantage, and the preparation that some were denied was not a disadvantage.  Everything is the same for all people in all context so who cares about anything? Inequality of opportunity is just a lie. (And, well, yes, it springs from falsehoods, but falsehoods can do real harm.) If we accept these premises, then everyone is off the hook for the first 18 years of life, and all responsibility now rests on the shoulders of college professors.

Sadly, under-preparation is real.  Some of it can be remedied, because people are (to some extent) adaptable.  Some of it can't be, because what happens early in life is (to some extent) of lasting consequence.  But pretending that it's irrelevant, and everyone can just be admitted to the same programs that start from the same place and proceed at the same pace, that's insanity.

Let us not talk falsely now.  Everybody knows what this is about. Everybody knows what they want.  So let's do it.  Let's replace the admissions tests with neural nets that infer race.  It's what they want.  It's illegal under Prop. 209, but it's nonetheless ethically defensible (at least from certain premises), so do it, and let the righteous battle be joined.  Fight for principles and premises that you believe in, rather than demanding that we all tell lies about preparation and tests. My 100% sincere opinion about affirmative action is that it is far more honest than anything else we do in higher ed, and I hate lying more than anything else. So let's do it.

Hofstadter: More of Chapter 6 and into Chapter 7

Before starting with Chapter 7, I want to note something from Chapter 6: On pages 167-168, Hofstadter notes that Jefferson was able to invite talented and accomplished men who were educated but hardly wealthy to run for political office with the support of a strong party system, but this was less feasible once we moved toward popular democracy.

This is relevant to the topic of civil service exams in Chapter 7. Hofstadter chronicles the myriad objections to exams and other civil service reforms, all centering around a fear that only rich kids from Yale could pass them. It didn't matter that the statistics said otherwise, nor did it matter that the content of the exams was hardly college material. Everyone was terrified of being ruled by intellectuals. I find this darkly amusing, because when I think of people who could ace exams, the very last people I think of are the sort who got into Ivy League schools in the 19th century.  Frankly, even today's Ivy Leaguers have a sizable contingent of legacy admits. And the people I know who most vehemently object to standardized tests in university admissions are disproportionately (though not exclusively) people from the sorts of families that don't actually need good grades to get into good schools, if you know what I mean.

It's also worth noting that federal civil service jobs have long been a path to the middle class for African Americans, and this was true long before the feds did anything that could be construed in any way as affirmative action.  Civil service exams may not be perfect, but they seem to be a bit more than just a means for Ivy Leaguers to feather nests for themselves and their children.

Letter in the LA Times

FWIW, I recently had a letter in the LA Times, in which I critiqued a proposal to do away with standardized tests in college admissions.

Please allow me to introduce myself, I'm a man of wealth and taste

Clarissa has a heresy-ridden response to the College Board's decision to offer an "adversity score" to measure the disadvantage experienced by test-takers. I agree with everything she says here, but the critics of the SAT have two responses:

1) The SAT doesn't actually measure your vocabulary, math skills, etc.  It measures your ability to take this test, and privileged people do better on it. This assertion is not supported by any systematic data, but it is supported by the anecdotes we've all observed of people who do far better or far worse than their SAT scores would have predicted.  They don't get that statistical predictions just tell us averages and ranges, and some fraction of people will fall outside those ranges on either side.

2) OK, the SAT does measure vocabulary, math skills, etc., but colleges have a duty to "meet students where they're at."  On this I have a tiny amount of sympathy--I do think we need educational institutions that will help people who could go far but have not come out of high school well-prepared.  But that's a long road, it will take far more than 4 years of college (let alone 1-2 years of remedial coursework), and what they really need is a "high school do-over" BEFORE a 4-year degree.  (Some might say that that's what an Associate's Degree is for, but the AA/AS degree is supposed to roughly correspond to the first 2 years of a Bachelor's degree.)

If a Bachelor's degree is to be "accessible" to people who start the program with woefully inadequate preparation, and if we are to fit this into the confines of 4 years of courses on the typical academic schedule, then institutions whose Bachelor's degrees "meet students where they are at" will be conferring credentials that employers and graduate schools justifiably treat as different from those offered elsewhere.  And this will just amplify rather than mitigate the class divisions in higher ed.

Even worse, this will actually work against efforts to diversify the academy.  Everyone out there wants to diversify PhD and faculty ranks.  The schools that disproportionately teach people from under-represented backgrounds disproportionately get under-prepared students because disadvantage has consequences.  If such schools must "meet students where they are at" AND do so within the usual confines of 4 years and roughly 120 credits (give or take), then their degrees WILL mean less.  That is an unavoidable fact.  And why should PhD programs take students whose credentials mean less?  Unless those PhD programs must also "meet students where they are at."  Which will either mean that students take longer to finish (and PhD programs face pressures on this front, including but not limited to the financial pressures of supporting students for longer times) or that students come out less accomplished.

And then people who hire PhDs will have to decide how to evaluate accomplishments...

Measuring success, chapters 10 and 11

These last two chapters compare test-optional institutions with similar institutions.  They focus on liberal arts colleges rather than the sort of place that I work in, but these comparisons get at the setting in which these debates are loudest.  The short version is that test-optional and test-requiring institutions followed almost identical trends from 1992 to 2010 when looking at measures of diversity, application numbers, and average test scores.  There are small differences, but they are small.  I think it comes down to the fact that test-optional initiatives are parts of much larger contexts.  Institutions that don't abandon the SAT still strive for diversity, and institutions that do abandon the SAT don't just say "OK, no need to do anything else about diversity.  We did what matters."  So the presence or absence of tests in admissions is just one factor among many.

Which is a metaphor for so many things, if you think about it.

Measuring Success, Chapter 9

This chapter, by Rebecca Zwick (researcher at ETS and professor emerita in the education school at UC Santa Barbara, aka U Can Study Buzzed, aka my beloved alma mater), looks at the tangible outcomes from test-optional and top-percentile admissions.  "Top percentile" admissions let in anyone who graduates in the top X% of a public high school class.  This has certain obvious virtues (anyone who makes it to the top must be more driven than most around them), it has the potential to increase diversity (the top students in a poor, minority neighborhood get in on the same footing as the top students in a rich, white school), and it does so in a race-blind manner.  Anyway, Zwick looks at the data, and a few take-aways:

1) These admissions policies don't seem to hurt graduation rates or college grades much, if at all.  This is consistent with the finding in earlier chapters that kids with high grades but low scores (and we can reasonably assume that such kids are common among those who don't report scores or get in because they graduated at the top of a school in a disadvantaged neighborhood) do pretty well.

2) On the other hand, there is still sorting:  Students who don't report test scores tend not to major in STEM.  That isn't a bad thing, IMHO.  If a kid has a great art portfolio then they should go to college and major in art, regardless of what their SAT math score is.  OTOH, that kid probably shouldn't do physics if their SAT math score is abysmal.  The previous sentence is only disparaging if you place physics on a pedestal that towers over the arts, and I don't place it on such a pedestal.

Anyway, these findings are reassuring, because there's something of a cottage industry in newspaper articles about minority kids who do well in a non-challenging high school but then flounder at a flagship.  Such kids surely exist, and definitely deserve some compassionate counseling on alternatives, but they are apparently not a major factor in the big picture, which means they are not a major impediment to diversifying large cohorts.

3) The gains for diversity are nowhere near what people were hoping.  When you go test-optional you have to look at resumes and essays and letters, all of which are at least as susceptible to manipulation and response to class and culture as anything on the SAT.

4) In a humorous aside, regarding the legality of affirmative action and alternatives to affirmative action, the author quotes Justice Ruth Bader Ginsburg as saying that "only an ostrich" would perceive top-percent admissions plans as race-neutral.  Whether or not the effects actually match the intent, they are designed and scrutinized in a discussion about race, with everyone hoping to achieve a diverse outcome without mandating a diverse outcome.  Say what you will for or against such agendas, but I admire Ginsburg's rhetorical flourish.

Measuring Success, Chapter 8

This is an interesting chapter.  Most of it is actually a republished chapter from Crossing the Finish Line, a 2011 book by Bowen, Chingos, and McPherson.  They used large data sets from 21 flagships and 4 state-wide systems to look at a very large cohort of students who started college in 1999, and see what predicted success.  They found that high school grades were less important than test scores, though test scores still have some predictive value.  This makes sense to me:  The ability to succeed at sustained tasks is distinct from (though not wholly unrelated to) the ability to do well on a test.  Also, a person who isn't strong at the things measured on tests might still find areas where they can succeed, while a person who can't devote themselves to regular academic work will have trouble succeeding at anything, even if they have certain mental traits.

After that portion of the chapter, some researchers at the College Board re-run that analysis with more recent data, and find similar trends, though the predictive power of grades has gone down while the predictive power of tests has gone up.  This is consistent with a hypothesis of grade inflation (per an earlier chapter).

Measuring Success, Chapters 4 and 5

Chapter 4 was a case study from one school that used merit scholarships to boost enrollment by good students, and included SAT scores in the mix.  They focused more on enrollment than performance after enrollment, so I ignored it.

Chapter 5 is on "discrepant" students.  We've all known This One Person who got bad test scores and good grades, and This One Other Person who got good scores and bad grades.  The single most important thing any academic can consider in any conversation on this topic is whether a new rule would be fair to This One Person and This One Other Person.  But for the bad people (like me) who want to look beyond This One Person, chapter 5 looks at college performance on the large scale, not just the anecdotes.  And it appears that the people with better high school grades than scores have outcomes that are almost as good as those of people with similar grades.

On the surface, this would seem to undermine the case for using test scores.  However, these people have scores that are SUBSTANTIALLY worse than their high school grades might suggest.  If you just look at people with test scores that are within the normal range for people with those grades, differences in performance between people with similar grades but different scores (albeit not outrageously different scores) are indeed correlated with grades.

Interestingly, the people with substantially better grades than scores tend to be women and minorities.  Again, at first glance this might seem to lead to a slam-dunk case against tests, but (1) we're talking about the outliers, not the people in the normal band (i.e. there are still plenty of women and minorities whose test scores are not discrepant, and the test scores continue to have predictive power for them) and (2) while women and minorities are more likely to have test scores that are substantially worse than high school grades than the other way around, the situation seems to reverse if we look at college grades for women and minorities in the "normal" (non-discrepant) band of grades and scores.  So, complicated things are complicated.

On the other hand, people with poor grades but good scores do somewhat worse than people with similar scores (not surprising), because "smart but lazy" is a thing. (On the other hand, sometimes people with bad scores deliberately take easy classes, and sometimes people with good scores deliberately take hard classes.  Complicated things are complicated.) Again, on the surface this would suggest that only grades matter, but we're talking about outliers.  If you look at people who are closer to the normal range, differences in scores still have predictive power.

Interestingly, there's some evidence that people with better scores than grades tend to go into harder majors than people with better grades than scores, and this confounds some of the analysis.  This is not surprising to me; STEM does have some epically smart but lazy people.  (Yes, I'm sure that somewhere out there is a supremely lazy literature major with a perfect SAT score and horrible grades, but those people are more commonly STEM majors.)

My main take-aways are:

1) Test scores matter but they aren't the ONLY things that matter.  (Duh.)  This point has been agreed on by just about everyone who's ever suggested using test scores for decisions.  Yes, I'm sure that somewhere out there is a literal straw man who has suggested eliminating grades from consideration and ONLY looking at scores, but that guy (and you just know it's a guy) is ignored by everyone else.

2) The studies in this chapter are mostly based on small samples and so should be interpreted with caution.  We should probably err on the side of rewarding work, while not completely ignoring tests.

Measuring success, chapters 2 and 3

Chapter 2 is a very detailed breakdown of data on college grades (freshman and 4-year) and college completion rates versus SAT score and high school GPA.  The message from many analyses is clear:  In every band of high school GPAs the SAT has real predictive power for performance in college, by multiple measures.

Of course, one cannot discuss this without discussing equity, so they make the same point as the previous chapter:  The SAT actually over-predicts college performance for minorities (because performance is also affected by disadvantages that are NOT fully captured in scores).  More importantly, they cite a different study than the previous chapter cited.  This gives me somewhat greater confidence about the point, when multiple investigators can cite a plethora of studies rather than everyone rallying around the same study.  We should always be suspicious of narratives built around This One Study.

Chapter 3 has one big point:  Grade inflation is real, and it's more prominent at schools serving affluent and white kids than schools serving poor kids from disadvantaged minority groups.  Consequently, if you base admissions decisions on grades rather than test scores you won't actually accomplish your equity goals.  Their main evidence for inflation is that they look at how high school grades have trended upward (in many but not all schools) for students in comparable bands of SAT scores.  Since the College Board does a lot of work to try to make SAT scores comparable across years, this strongly suggests that grading is getting more lenient (in some but not all schools).  And the schools at which grading is getting more lenient (based on this line of analysis) are whiter and more affluent than the schools with less grade inflation.

Not having examined the data myself, I am obviously not in a position to weigh in on the validity of this work, but if other authors have found similar things in multiple independent analyses then we should seriously consider the implications.

Next book: Measuring Success

The next book that I'll blog about is Measuring Success, an edited volume with 11 chapters, 3 authors, and 26 contributors.  The book is about the predictive validity of standardized tests in college admissions.  This book poses something of a dilemma for me.  On the one hand, it is rich in data and citations to the peer-reviewed literature. On the other hand, the editors include Lynn Letukas and Ben Wildavsky from the College Board (the organization that produces the SAT) and someone from a research center that includes the College Board on its client list.

I think the proper response is to take this with a big grain of salt.  Nothing should be taken as a priori truth, but can be taken with a suitable dose of "Assuming that similar results are found in multiple, independent investigations..."

The first chapter is probably the least rigorous, simply because it's an introductory chapter titled "Eight Myths about Standardized Admissions Testing", hence several points are touched on briefly (though many of the same themes are visited in more depth in subsequent chapters).  The putative myths are:

1) Standardized tests are very poor predictors of freshman grades:  The authors concede that weak correlations are found if you look only at the set of admitted students, but if you correct for the fact that the proper comparison includes the students who were not admitted but nonetheless went to college elsewhere (or were admitted but chose other schools) the correlations improve.  This is a point that I've made before and with quantitative detail.  Moreover, students with weaker preparation often choose different majors than students with stronger preparation, so their grades might not be comparable.  But the authors include data showing that when you look at a wider pool and control for common college curricula and also high school GPA, the correlation between SAT score and college grades improves considerably, reaching 0.8 (versus 0.35 in poorly-controlled studies).

2) Tests do not predict anything beyond grades: The authors show data indicating that students with higher test scores take more advanced college classes than those with lower scores, and enter majors that reflect their higher test scores (e.g. verbal versus math).

3) Alternatives to testing are superior:  The authors reference work on various alternative measures, and show that often the sample sizes are small and correlations are weak.  Moreover, even if there are superior sources of data for admissions decisions, I'm not sure why one would ONLY use one source of information.  Why not build a multi-variate model?  And since tests are not as vulnerable to the subjective biases of raters (e.g. interviewers, letter writers, essay readers) the claim of superiority over tests seems to be an extraordinary one, requiring extraordinary evidence.  (Or, at a minimum, a very careful articulation of what could count as "superiority"--is lack of bias not one of the desiderata?)

4) Tests do not measure anything relevant to life after college:  Here the authors cite correlations between test scores and quality of graduate work as evaluated by faculty (i.e. evaluations of the quality and quantity of research output) as well as work performance after graduate study.  However, this is a weak point, because it is focused on the Miller Analogies Test as an admissions test for students preparing to work as counselors, rather than more widely-used tests for undergraduate admissions (e.g. SAT, ACT) or graduate admissions (the GRE is the main game in town here).

Still, the authors are psychologists, so counseling programs would seem to be near to their hearts.  I'll give them one fumble here.

5) Beyond a threshold tests have little predictive power:  In other words, this is an argument that above a threshold you can't use tests to distinguish decent from great performance.  And, of course, it is true that tests can't do that with perfect predictive power. However, the authors cite evidence from large studies (6656 students in one study, 15,040 in another, and 150,000 in the third) that tests have non-trivial predictive power even at the upper end of the talent pool.  Intuitively this makes sense:  If test scores only had threshold predictive power then the correlations under point 1 would probably not be as large.

6) Tests only measure socioeconomic status:  This one was an eye-opener. They show mean SAT score varying from 1300 (on a 2400 scale) for the lowest income bracket (<$20,000/year household income) to 1700 for the highest bracket (more than $200,000/year household income).  That variation isn't trivial, but it is also hardly enough to generate the correlations seen earlier, especially when you take into account the wide variation within brackets.  More importantly, even when controlling for family income, the predictive power of SAT scores remains quite strong.

7) Test are biased:  Here the authors are careful to unpack what "bias" means.  If a test is biased against group X and favorable to group Y, then if we take a bunch of students with the same test score and look at their college performance, the X students should do better than the Y students.  In other words, such an outcome would tell us that X students are doing better than their score would predict while Y students are doing worse, so admitting based on the test gives X students a disadvantage (they're being treated the same as weaker Y students).  However, SAT scores slightly over-predict college grades for minority students. The over-prediction makes sense to me, since disadvantage is multi-faceted, and there are aspects of it that cannot be fully captured by family income.  If disadvantage matters and is related to ethnicity then I would expect minority students with a given family income and same academic preparation to fare slightly worse (on average) because they face burdens that otherwise-similar white students do not face.

8) Coaching produces large gains:  The authors show data suggesting that gains from test prep and coaching are over-stated.  This makes sense to me on a few levels.  First, the students who avail themselves of test prep include a substantial pool of students who did poorly on their first try, and regression to the mean is surely a factor here.  Second, this pool includes kids who did not make the minimal effort to familiarize themselves with the test beforehand. A bit of effort to get familiar with the task at hand is a modest task, one that does not require expensive tutors, but expensive tutors will nonetheless be happy to collect a fee for helping one with that modest task.  There are no doubt gains from minimal due diligence, and gains from re-testing after some coaching may in part reflect that modest due diligence.  The open question is whether those gains reflect much beyond that, i.e. reflect things that a kid couldn't do without substantial resources.

Besides, if shilling is a concern, then taking claims about the value of coaching at face value amounts to trusting marketing materials from Kaplan, Princeton Review, etc.  That's a dubious thing.

Now, that said, all of the points made here are worthy of follow-up. The subsequent chapters have more in-depth analysis that we need to examine.  The first chapter is suggestive motivation but hardly conclusive evidence.

Statistics and Standardized Tests

I have posted a paper on the arXiv (a website for physics articles) addressing statistical issues in interpreting standardized tests.  This is a volatile topic, and my argument is long and nuanced, so I don't want to retype the entire thing in a blog post.  The short version is that a couple years ago somebody noticed that there are many successful scientists who did poorly on the GRE.  One possible interpretation is that the GRE has no predictive power for performance in a PhD program.  Another possibility is that performance is predicted by a combination of several variables, and a low score on any one of them might easily be compensated for by a high score on another.  Moreover, admissions processes typically enforce such a condition, so the only people getting in with low GRE scores probably did well by some other measure (e.g. research experience, relevant work experience, lab skills, etc.), while people who did poorly by some other measure must have had something to compensate (e.g. the GRE).  Consequently, when you compare people with high and low GRE scores, you aren't actually holding everything constant.  I use computer simulations to illustrate these points.

I also include a digression on "ought" vs. "is" statements, illustrated with two characters (Helena and Cosima) from Orphan Black, my current favorite science fiction show.  It may be that Helena is less likely to succeed in graduate school than Cosima is, but if Helena had a disadvantaged upbringing (and believe me, she did) then we ought to give her a chance to exceed our expectations.  This is an important point, given the diversity implications of standardized tests.

Why I am what I am

I went to a decent high school.  Not fancy, not lousy.  Science labs were OK but we weren't putting out national science fair competitors.  Most kids weren't poor but most weren't upper-middle-class either.  Not particularly disadvantaged but not a lot of affluent kids either.  Pretty much in the middle.  Which is fine.  We had college-prep classes but we also had a lot of vocational classes.  It was a decent school in the middle of the class hierarchy.  I was more privileged than a lot of the US but less privileged than a lot of the people I later met in college.

One thing that was repeatedly driven home to me by my teachers was that we would be on our own in college, and that there would be no hand-holding.  It was repeatedly said that we would have to figure it out, that they wouldn't patiently explain to us how much we should be reading and reminding us to study and reminding us to do our homework and turn it in on time.  It would be hard and we would be on our own.  This was repeatedly said to me and my middle-class friends.

And, to a large extent, that was true.  College was a bit more structured and supportive than my high school teachers made it sound, but only a bit.  Ultimately, it was on me.  I accepted that from day one, and it felt utterly unsurprising to me that I was doing well because I studied all the time.  It felt utterly unsurprising when friends and dorm-mates who studied less didn't do as well.  It's the natural order of the world.  Likewise, in grad school I felt a bit more burned out, a bit more interested in my own life, and I didn't do as well as I did in college.  I still got through, but I was not on top, and that was no surprise to me because I didn't put as much in.  Then I became a professor and buckled down more and published more and got decent student evaluations because I put in a ton of prep and none of this seemed at all surprising to me.

So you can imagine how strange it seems to me when people keep saying that we need to do more for students who don't know how to study, how to take notes, how to manage their time, etc.  It baffles me that we are supposed to be responsible for their success.  Why would we be?  Why would it be my problem that people who never put much effort into mastering freshman material are now doing poorly with advanced material?  Why is it my duty to fix this?  It's not like I could actually do anything about it (you only get the benefit of years of effort by putting in years of effort--there are no secret tricks), so why is it my responsibility?

Likewise, it seems strange to me when people say that it's so unfair that we attach weight to grades and test scores.  I got into a college with people who were way above me in the class hierarchy, people whose parents had more money and more degrees, people who had attended prep schools and magnet schools.  I was there because I had done at least as well as they had done on the SAT (among other things).  I got better grades than them in college.  I out-studied them, out-performed them in the academic arena, and moved ahead academically.  Isn't that how it's supposed to be?

I've spent more than two years reading and blogging about the cultural currents that underlie the notions that seem so strange to me, but I'll be honest:  To this day there's still a part of me that thinks "Well, duh, I studied my ass off because my teachers all told me that that's what I'd have to do to succeed in college.  What else were you expecting?"

Project idea: Incentives, measurement, and fairness

Among the problems that I find most fascinating and most frustrating is the that of people who want to systematize education and incentives.  We all agree that what we really want is meaningful, intensive, in-depth discussion of important topics, creative assignments, and lots of attention to students as individuals.  We all agree that this is hard to systematize and measure, even if some approaches are clearly less bad than others.  We all agree that the things that are easiest to measure are not quite what we desire. We all agree that the more you incentivize something the more that people will focus on hitting the metric rather than doing what the metric is supposed to be a proxy for (sometimes called Campbell's Law).  And we all agree that without incentives of SOME sort people will get lazy.  So we have a hard problem.

My thought is that if a simple metric isn't TOO distant from what you really want, and if the competition isn't TOO intense, then being measured will keep people from being lazy, but because you still have some flexibility (the competition isn't so intense as to push you into a single-minded focus on that metric) and because you yourself actually value the thing that REALLY matters (and the metric isn't completely decoupled from it) then you'll show up to work and split your effort between hitting the metric and doing the more meaningful thing that everyone REALLY values.

But if the competition is more intense then you have to hit that number no matter what.

As to the people measuring you, on some level they know that the simple metric is flawed, but they face two other pressures;

1) Measuring something closer to what they want would be more expensive.

2) Because those measurements are of limited precision and may be subjective, it would be seen as unfair to focus so much on them.  Indeed, I sometimes think that the current focus on the findings of bias research plays into this:  The output of the bias research community has demonstrated that everything we do can be unfairly biased.  It's thus tempting to seek simpler and more transparent technical evaluations instead of subjective appraisals of nebulous quality.

So here's my idea for a model:

An administrator has a pot of money to allocate and people.  The funder assigns money based on two measures, and easy one and a hard one.

The researchers getting the money have to allocate their efforts among three things: Leisure (bad!), hitting an easy target (which generates some utility, but quickly reaches the point of diminishing marginal returns), and hitting a hard target (which takes more effort but yields more satisfaction).  

The administrator's utility payoff is based on 3 things:

1) How much effort people allocate to doing hard things.  The payoffs here are huge, because the administrator gets to take credit for good things that happen in the system that he/she oversees.

2) How fair and transparent the administrator is.  The more the administrator rewards easy things, the more that political bosses and/or the public will perceive him/her as fair and transparent.

3) How much time the administrator invests in measuring the hard thing.  The payoffs here are negative because it's hard to measure.

The researchers' utility payoff is based on 3 things:

1) How much time they spend on leisure.  There are diminishing marginal returns here.  Zero vacation days will kill a person, but after too much vacation they want to get back to the lab.

2) How much time they spend on hard things.  There are increasing returns on scale here, because you don't get anywhere until you've invested a lot of effort.

3) How much money they get from the administrator.

What can we say about the Nash Equilibria of this problem?  It strikes me as posing issues similar to Holmstrom's Theorem.

On some level I've basically outlined the issue to the point where a key idea can be presented:  The easy measure is useful to the extent that it satisfies the public's need to see that the system is honest, and also keeps researchers from being lazy.  The easy measure is parasitic to the extent that it diverts time away from what is meaningful.  You can say "Just improve the easy measure so it's more aligned with what you value!" but the key points are:

1) Measuring valuable things is hard (by assumption).

2) Aligning the easy measure more closely with what you really value is great, but if the competitive pressures are high enough you'll eventually see the difference between the easy measure and what  you really value.  When competitive effects are weak then people will still allocate a lot of time to hard but satisfying efforts.  When competitive effects are strong then people will devote much more time to easy things.

Of course, it's nice to make these points, but the system that I work in doesn't reward blog posts.  The system that I work in rewards peer-reviewed journal articles, and journals reward mathematical formalism.  I could just sit down and do it but I have lots of other projects on my plate, so I need a collaborator to prod me.  If anyone wants to help please let me know!

Blank slates to the left of me, blank slates to the right, here I am stuck in the middle with truth

There's a nice essay on the concept of a "blank slate" in Quillette.

The basic idea of a blank slate is that all humans are equal in capacity.  It's a foolish notion, but it's a nice one, and it's usually a notion associated with the left:  The left recognizes that the world is not a place of equal opportunity, that people are saddled with all sorts of disadvantages by nurture rather than nature, and that all too often people try to wrongly shift the blame from the cruelty of man (nurture) to the cruelty of unequal genetic endowment (nature).  Some even go farther and (wrongly!) ascribe the disadvantages of entire groups not to the cruelty of man (nurture) but rather genes (nature).  Consequently, people on the left tend to be suspicious of genetic explanations of human behavior.  To the extent that this suspicion is merely skepticism of hypotheses that are difficult to prove and often wielded to evil ends, it is an intellectually healthy suspicion.  To the extent that it is a rejection of neurobiological inquiries into human behavior it is an unhealthy suspicion.

The author in Quillette points out that the right has its own version of a blank slate:  They argue that not only are people born equal in natural endowment, they remain equal in practical capacities even in the face of unequal nurturing, and hence unequal outcomes are primarily the consequence of bad choices, or moral failings.  It is as absurd as rejecting the idea that individuals' brains differ in part because of genetic variability among humans.  If the left is insisting that the slate was blank at birth, the right is insisting that the slate has NOT been vandalized even after years or decades of unequal conditions in a cruel world, or that the marks are at least easily erased.

Of course, the left has its own moral analysis as well:  While recognizing that it is absurd to expect the disadvantaged to simply shrug it all off and succeed equally, some do insist that college professors would obtain equal outcomes if we merely gave enough chances, and were enlightened enough in our evaluations.  If we just followed "best practices" then we would see gaps shrink substantially, and the persistence of gaps thus follows from our choice to abstain from "best practices."  It shifts the moral responsibility from the vandal who made the mark to the professor who was unable to erase it.

Like all seductive lies, it's partly true:  Some people will, if given a chance and some support, defy the odds and rise to the occasion.  To structure educational programs as Darwinian competitions with one chance and no more is both foolish and unethical.  (I will note that my grading formula rewards improvement and provides some forgiveness for initial failures.)  But it is important to recognize that working to remedy inequity in this way is costly and risky.  It won't always work, and it will cost more than simply giving one chance and moving on.  If you nonetheless value opportunity then you will accept costs and risks.  If you don't actually value opportunity then you will deny that it has a cost (because that which is without value is without cost) and insist that the only problem is a failure to identify the "right" criteria, the "right" measures.

I remark on this because, as I have noted before, our theories of success and failure can be double-edged swords.  A theory that success that success is rooted in "grit" or "growth mindset" is the key to success (rather than evil standardized tests) can be shiny and progressive if it is used to undermine evil standardized tests, but it can also be turned around and used to blame the poor and disadvantaged for lacking the moral virtues of grit and growth mindset.  A moral theory that says we are all equal in capacity can be used to blame a cruel world for failure, but it can also be used to blame failure on choices by people of presumably equal capacity.  The fact that the left and right go in such different directions from ostensibly similar assumptions means that there are additional embedded assumptions that people aren't always articulating openly.

For me, I will be disappointingly wishy-washy and say that EVERYTHING matters for success and failure.  Some kids really ARE born smarter than others.  Sorry, but it's true.  On the other hand, unequal conditions in life will ALSO leave their marks.  Some of those marks CAN be erased.  Others are much harder to erase.  Some people will surprise you and outperform expectations.  Some won't.  It's worth it to give people a chance.  It's ill-advised to keep pouring in resources as the returns diminish.  Some people really do fall on bad choices or rise on good choices.  Some people are stuck.  Abandoning personal responsibility in your moral calculus is ill-advised, but so is treating personal responsibility as the only variable.  It's complicated and we have to muddle through.

I wish I had some sort of magical solution, but I don't.

Transparency

In this morning's Los Angeles Times, there's an op-ed calling for the College Board to release detailed question-by-question data on the SAT.  The op-ed claims that this data is necessary to address questions about fairness and achievement gaps.  This is a fine thing to be concerned about, but before you let this op-ed push your buttons take a look at the author info at the end of the article:

Jay Rosner is the executive director of The Princeton Review Foundation.

I wonder why a test prep service would want detailed data on a test...

Mind you, there are plenty of good reasons why the world would benefit from more transparency in such a high-stakes matter as the SAT, and the pecuniary interests of the op-ed writer do not make the concerns any less valid.  However, it is important to be reminded that even righteous causes attract all sorts of people with financial angles.

No, see, the revolution was just never tried

In the previous post I noted that personality surveys are the easiest things in the world to game.  I closed with a link to an article on a college that decided to give applicants a survey on study skills, which is pathetically easy to game.  Well, just google the name of that school and "locus of control" (a psychological term related to what the school believes they will be measuring.)  See what you get:  Admissions coaches!  WHO COULD HAVE PREDICTED THIS??!!11?1!?1!!????111!?

You cannot game your way out of advantaged kids gaming the system. By definition, the advantaged will out-game you.  Especially in an asymmetric game where the first mover loses their advantage by carving in stone the rules that the second mover can respond to and the first mover is bound by.

What is measured improves, but nothing else does

Oh, good!

A recent update to federal education law requires states to include at least one nonacademic measure in judging school performance. So other states are watching these districts as a potential model. But the race to test for so-called social-emotional skills has raised alarms even among the biggest proponents of teaching them, who warn that the definitions are unclear and the tests faulty.

The fault, dear Brutus, lies not in the tests but in the stakes.  Go ahead and create the world's most perfect, reliable, valid, accurate, and all-around good measure of grit, or growth mindset, or whatever.  Administer it in a setting where nobody has been primed to answer disingenuously and nobody has any real reason to deceive.  You'll get results that tell you something meaningful.  Now attach stakes to it.  And watch as the teachers start feeding students slogans about grit and growth mindset, and students start giving answers that sound good rather than answers that accurately tell you something about their actions outside of the testing situation.

Fortunately, a few of the people interviewed in the article agree:

In their paper published in May, Dr. Duckworth and David Yeager argued that even if students do not fake their answers, the tests provide incentive for “superficial parroting” rather than real changes in mind-set. 

“You think test scores are easy to game?” said Martin West, a professor at the Harvard Graduate School of Education, who is working with the districts in California. “They’re relatively hard to game when you compare them to a self-report survey.”

I actually don't fault the entire idea of measuring non-academic things.  The very fact that people will try to game measures is actually an argument for measuring more things rather than fewer.  The more things you measure the harder it is to game all of them at once.  Still, it's better to make more of those measures be about things that are harder to game.  As opposed to this.

This concludes our monthly feature titled "Something actually interesting in the NYT education section."

Classical education, character, competition, and privilege

I have run across the argument that the Imperial Chinese practice of selecting officials based on tests of Confucian classics was a factor that limited China's ability to innovate and keep up with the West.  It's a tempting argument, and the more I think about it the more I sympathize, but first we have to explore one significant objection to the argument:  Reading Hofstadter makes it clear that the educational systems of all Western powers in the 17th, 18th, and 19th centuries emphasized classical studies for their elites.  I don't know that reading Homer and Plato is any better or worse for one's character and judgment than reading Confucius.  I'll grant that the Western canon taught to elites seems to have had greater diversity of sources than the Chinese canon, but I don't think that the choice of texts was the limiting factor.  Rather, I think it may be that (ironically) the Western system may have put less pressure on elites.  Let me explain.

Advanced schooling was not terribly commonplace in Western countries prior to the 20th century.  Anybody who got an advanced education was probably from a pretty privileged background.  The people teaching in the high schools, boarding schools, universities, and whatnot might be stressing mastery of the classical liberal arts, but the kids in those schools knew that they had a good shot at a decent future just based on their backgrounds.  For most of them, academic excellence was not necessary; getting by was enough.  That's not to say that they all drank and partied their way through boarding school--many did, but not all.  There were still sports and clubs and other pursuits befitting a young gentleman of the right background.  A person who did a decent job in class and also earned respect as a leader on the sporting field and in his clubs might be recognized as a young gentleman of the right character and be recruited into some fast track in public service or business.  It might not be like today's career fairs, but if he attracted the right notice it would serve his future well.

The key thing is that he was in an environment where he stood out for going above the minimum, because he and the people around him all felt some baseline of security.  Within the context of their society and upbringing, they already had good reason to feel fairly secure.  Those who excelled did so because of their character, not because it was their only shot.

Contrast that with today's kids seeking admission to elite colleges.  I'd be the first to say that anybody who can seriously even contemplate an elite college will probably be OK if "all" that they get is their "safety school."  However, they don't feel that way!  They feel like they need to push farther and farther.  They aren't on two athletic teams AND in orchestra AND in a community service club AND doing a special science club AND getting A's AND taking AP classes in five different fields because they are well-rounded and simply love all of those things.  OK, a few of them are.  However, the vast majority know that there's a game to play, and so they play it.  Some will become deeply humanitarian as a result of all of that service, and be humbled by all of the privilege that they enjoy.  Some will become arrogant and narcissistic because they feel that their giant resumes prove that they are so great.  Most will wind up somewhere in between and just keep pushing on in life.  But the key thing is that people observing them in that moment can't know who will become a humanitarian and who will become a narcissist!  The competition obscures character.  You never know who is passionate and who is just doing what they need to do.

There's a lot to be said for taking people outside of their comfort zones, but there's also a lot to be said for observing people in their comfort zones.  Anybody who has ever taught pre-meds (or been a college student taking freshman chemistry alongside pre-meds...) knows that pre-meds are all the exact same ultra-competitive little shits (sadly).  Anybody who has been to more than one doctor knows that doctors vary widely in personality, approach, and skill.  Before they get into med school they are all the same because they have to be.  Once they get into med school, they know that their worst case is a low-prestige residency in primary care...which will still yield them a comfortable income!  Some will push hard anyway and qualify for the most competitive residencies in the most competitive specialties.  Some will just keep their nose to the grindstone and do a good job and become good doctors but not shine in the most competitive heights.  And some will be slackers who keep malpractice lawyers employed.  But they all know what their baseline is, they have a certain margin of comfort, so now their internal motivation and character matter.  Really, it's not so different from observing people after tenure.  Some make good use of their security and leverage it to greater accomplishment, some just keep on doing decent work, and a few are as lazy as you'd fear.

The 17th to 19th century Western systems of elite education varied in many ways, but they all took the sons of privilege, put them in an environment that combined classical learning with extracurricular pursuits, and told them that they'll all do OK in life but how far beyond that they go will depend on their character, effort, and accomplishment.  Those sons of privilege then proceeded to distinguish themselves to varying degrees, and could be observed for all manner of traits, to be recruited (or not) for various paths.  It was more like medical school or tenure than the Chinese system of ever-more-competitive exams for ever-higher stakes.

The problem with our elites is not that they got good grades and test scores.  The problem is not even that they enjoy privilege and security. Every society has privileged elites--even (and especially) the ones that undertook an ideological project of being "classless."  Rather, the problem is that the political system is too closely aligned with sectors that give their greatest rewards to people who play the game for top colleges and then play the game for top law and MBA programs and then play the game to be recruited by the biggest law firms, banks, and corporations.  I don't care that elite law firms and banks and big corporations exist and have their games.  I care that we have a political class too focused on those elements of the economy, while the middle is hollowed out.

The real solution has nothing to do with universities and everything to do with economic policy.  But if you absolutely insist on trying to at least partially improve the elite class through college admissions we are stuck with a paradox: The elites show their character when they are most comfortable, not when they are most competitive.  However, there's no morally or socially acceptable to pitch protection or comfort for elites as a solution to inequality.  There are far more people who would like to attend Stanford and Harvard and Yale than there are spots at Stanford and Harvard and Yale. The only morally acceptable response is to make the applicants demonstrate (in some way) that they are worthy (by some measure).  That measure can be grades, character, writing samples, extracurricular accomplishments, service, interviews, whatever.  However, as long as there are way more people who want it than there are spots, and as long as there are huge returns to getting one of those spots, there will be intense competition.  Consequently, you will be observing how people compete on grades, or interviews, or service, or whatever, not what they do when they are relaxed.

I have no easy fixes to offer, but my best guess is that you'll  have to just accept a certain amount of entrenched elitism, let their kids relax and let their guard down instead of doing 10 different extracurriculars alongside sports and AP classes, and then use some class-based and/or race-based affirmative action to add some non-elite kids to the mix.  No competitive formula will fix things.  But I'll take some bright young gentlemen (and ladies) who displayed their character from a position of comfort over the socioeconomic elite version of a pre-med.

Recommended post: Cheating scandals

Regular reader Phil Ebersole has a good post up that says more succinctly what I have often tried to say about tests, measures, and selection criteria.  The only thing I would add is that many of the problems of standardized tests probably also apply to a lot of other high-stakes selection processes or criteria.