Project idea: Incentives, measurement, and fairness

Among the problems that I find most fascinating and most frustrating is the that of people who want to systematize education and incentives.  We all agree that what we really want is meaningful, intensive, in-depth discussion of important topics, creative assignments, and lots of attention to students as individuals.  We all agree that this is hard to systematize and measure, even if some approaches are clearly less bad than others.  We all agree that the things that are easiest to measure are not quite what we desire. We all agree that the more you incentivize something the more that people will focus on hitting the metric rather than doing what the metric is supposed to be a proxy for (sometimes called Campbell's Law).  And we all agree that without incentives of SOME sort people will get lazy.  So we have a hard problem.

My thought is that if a simple metric isn't TOO distant from what you really want, and if the competition isn't TOO intense, then being measured will keep people from being lazy, but because you still have some flexibility (the competition isn't so intense as to push you into a single-minded focus on that metric) and because you yourself actually value the thing that REALLY matters (and the metric isn't completely decoupled from it) then you'll show up to work and split your effort between hitting the metric and doing the more meaningful thing that everyone REALLY values.

But if the competition is more intense then you have to hit that number no matter what.

As to the people measuring you, on some level they know that the simple metric is flawed, but they face two other pressures;

1) Measuring something closer to what they want would be more expensive.

2) Because those measurements are of limited precision and may be subjective, it would be seen as unfair to focus so much on them.  Indeed, I sometimes think that the current focus on the findings of bias research plays into this:  The output of the bias research community has demonstrated that everything we do can be unfairly biased.  It's thus tempting to seek simpler and more transparent technical evaluations instead of subjective appraisals of nebulous quality.

So here's my idea for a model:

An administrator has a pot of money to allocate and people.  The funder assigns money based on two measures, and easy one and a hard one.

The researchers getting the money have to allocate their efforts among three things: Leisure (bad!), hitting an easy target (which generates some utility, but quickly reaches the point of diminishing marginal returns), and hitting a hard target (which takes more effort but yields more satisfaction).  

The administrator's utility payoff is based on 3 things:

1) How much effort people allocate to doing hard things.  The payoffs here are huge, because the administrator gets to take credit for good things that happen in the system that he/she oversees.

2) How fair and transparent the administrator is.  The more the administrator rewards easy things, the more that political bosses and/or the public will perceive him/her as fair and transparent.

3) How much time the administrator invests in measuring the hard thing.  The payoffs here are negative because it's hard to measure.

The researchers' utility payoff is based on 3 things:

1) How much time they spend on leisure.  There are diminishing marginal returns here.  Zero vacation days will kill a person, but after too much vacation they want to get back to the lab.

2) How much time they spend on hard things.  There are increasing returns on scale here, because you don't get anywhere until you've invested a lot of effort.

3) How much money they get from the administrator.

What can we say about the Nash Equilibria of this problem?  It strikes me as posing issues similar to Holmstrom's Theorem.

On some level I've basically outlined the issue to the point where a key idea can be presented:  The easy measure is useful to the extent that it satisfies the public's need to see that the system is honest, and also keeps researchers from being lazy.  The easy measure is parasitic to the extent that it diverts time away from what is meaningful.  You can say "Just improve the easy measure so it's more aligned with what you value!" but the key points are:

1) Measuring valuable things is hard (by assumption).

2) Aligning the easy measure more closely with what you really value is great, but if the competitive pressures are high enough you'll eventually see the difference between the easy measure and what  you really value.  When competitive effects are weak then people will still allocate a lot of time to hard but satisfying efforts.  When competitive effects are strong then people will devote much more time to easy things.

Of course, it's nice to make these points, but the system that I work in doesn't reward blog posts.  The system that I work in rewards peer-reviewed journal articles, and journals reward mathematical formalism.  I could just sit down and do it but I have lots of other projects on my plate, so I need a collaborator to prod me.  If anyone wants to help please let me know!

Bias, Madison, and Human Perfectability

Today's blog post by Dean Dad is a good one:  He responds to a recent article in the Chronicle, by a hiring consultant talking about the importance of raising awareness of our implicit biases.  Dean Dad makes the point that this smacks of trying to re-educate people and cleanse their thoughts.  There's plenty of bad history behind such efforts, and it is ironically at odds with the idea of diversity:  The goal is less to get everyone on the same page and more to bring together a bunch of different perspectives and let people check and balance each other.  (Hence James Madison.)  I like the way of looking at it.  If our implicit biases are deep-seated, persistent, and always acting then the very utility of awareness is questionable.  OTOH, bringing together different people with different viewpoints and giving them equal voting power in a consensus-oriented process might actually mitigate some of the effects of our biases.

It's also worth noting that the author of that piece in the Chronicle would be delighted if you wished to hire her firm to help you run a search process that will minimize the effects of bias...

Anti-Intellectualism in American Life

Well, here I am, blogging late at night/early in the morning after a shocking election.  I have no interest in dissecting partisan politics here, only intellectual politics.

In the 1990's, an interregnum between the Cold War and War on Terror, academics drew knives on each other for the "Science Wars."  We got to hear from a certain type of humanities student that science is just, like, a Western hegemonic patriarchal cultural construct.  That all ended when Bush The Lesser took office, and all academics agreed that global warming is objective scientific fact.

In recent years we have been going at each other in higher ed over a somewhat different set of issues, but I suspect that we'll come together again.  Most academics will be on the same page about Trump, and there will be a degree of internal unity that we haven't previously enjoyed.  Even many academics of rather conservative views on certain issues (e.g. me) feel that Trump is illiberal, uninformed, and reckless.

What are the things that we will declare a truce on?  Well, we have been talking a lot about unconscious bias and invisible advantages, the subtle things.  But we are now confronted with a large number of people supporting a man who spouted undisguised, clearly visible, not-at-all subtle bias.  The problem we face is not good people who are slightly biased in spite of good intentions, but something much more blatant.

That's the part where I ask for people to give less emphasis to something that I disagree with.  Here's where I give less emphasis to something:

I've been going after the anti-intellectualism of progressive education.  I'm still opposed to it, and I will still pursue the path that I believe is most intellectually appropriate.  But anti-intellectualism manifests in many ways and in many aspects of American life.  Progressive educators cannot be my only targets.

Ethical knowledge is not ethical action

Claude Steele has stepped down as provost at Berkeley, and will return to the faculty. The administration at Berkeley has been under intense scrutiny over their handling of high-profile sexual harassment cases in the past year, and while the announcement of his resignation does not explicitly mention those controversies it is quite reasonable to guess that they were a factor.

I'm blogging about this because Steele is a famous psychologist, best known outside the field for his work on stereotype threat. I do not know enough about the details of those cases and the chain of events inside the administration to say for certain whether Steele deserves criticism, but the issue here is less about specific individual culpability and burdens, and more about assumptions that are "in the air" among academics.  Regardless of whether Steele himself made mistakes of sufficient gravity to merit losing his job according to whatever standard you might wish to apply, it seems clear that things could have been handled better than they were.  What matters here is that the handling was, at the least, sub-optimal, sitting somewhere along a spectrum that goes from "grossly negligent" to "typical bureaucratic response" to "probably well-intentioned but still not what we need."

In the modern zeitgeist there is a belief that raising awareness of research on bias--including but not limited to subconscious bias--will lead to better behavior, better decision-making, more sensitive people, etc.  Now, to my knowledge Steele himself does not explicitly study sexual harassment, but he does study stereotypes and their effects.  One can reasonably guess that he is thus more aware than the average academic about the wider literature on bias, sexism, etc.  If a raised consciousness were the key to getting people to respond promptly, appropriately, and effectively to acts of sexism, racism, etc. then putting a bias researcher into a position of power would be the surest path to reform.  Alas, it seems that bias researchers are as human as anyone else.  Having lots of "is" knowledge about how bias works does not lead them to act as they "ought" when faced with a professional challenge.

This should not be a surprise.  Jonathan Haidt noted in The Righteous Mind that academics who study ethics are no more ethical than anyone else.  I'm from a family full of health professionals, and some of them have all sorts of unhealthy habits.  I'm in a profession of educated people and some of them are anti-intellectual.

There's a separation between what we know intellectually and what we carry into practice.  That's certainly an indictment of my profession, including the traditionalists and the progressives alike, because studying something, even in-depth and for years, doesn't always transform people as you'd hope.  At the same time, I'd like to selfishly and shamelessly argue that it reflects even worse on the progressives than the traditionalists, because I think the traditionalists are more willing to treat problems as hard, whereas the progressives hold out hope for secret tricks and correct politics.  A traditionalist social scientist wants to understand people, while a reformer wants to fix them.  It's the difference between science and engineering. Most news articles about "According to a new study..." (including the one I linked to at the beginning of this post) are written to satisfy a curiosity about how to act, not merely to help the reader understand why people act as they do.

So, I don't claim to know anything about how things were handled (or mishandled) behind the scenes at Berkeley, but at the very least I am unsurprised that a bias researcher would not be any better than the rest of the world when it comes to responding to sexism.  All of the "is" statements in the world cannot change a person's "oughts."  Only "ought" statements can do that.  Social scientists cannot save us any more than priests can.  Only you can save your soul.  Priests and social scientists can advise you, but only you can decide your actions.

Sein und Zeitgeist

In Chapter 10, Lee Jussim gets to the question of why inaccuracies and bias in human perception get more attention than accuracy.  On page 153 he addresses an aspect of the issue that I think is most salient to my concerns as an educator: If we believe that a person is unskilled, under-qualified, under-accomplished, or whatever because our perceptions are inaccurate, then "all" that we need to do is change our perceptions.  That isn't always easy to accomplish as a seamless thing, it often requires conscious effort, but, in the end, bringing yourself to say "You know, he/she is actually pretty talented; I'll hire them/admit them to college/etc." is still easier than saying "Uh-oh, this person is way under-prepared, we have work to do to fix that."  Any research that tells us that problems can be fixed by thinking differently is going to attract more support from the intellectual class than research that tells us that we'll have to do a lot more work on problems that stubbornly defy easy solutions.

On page 158, Jussim also notes that social psychology is more fun if you find bias and inaccuracy.  If people generally perceive things pretty accurately then what is there to say?  What need is there for a consultant?  People will figure things out and, um, yeah.  But if bias and misperception are everywhere then there's something to work on.  This is akin to issues of publication bias in other areas of science.  Perhaps bias research is biased in favor of bias research?

If the concerns in the first paragraph are largely social and political, and the concerns in the second paragraph are technical and pecuniary, then I think there's also a third contributor to the intellectual zeitgeist: Negative results from Western intellectuals in the 20th century.  From them, we learned that measurements of distance and time are not absolute, but rather depend on how you are moving relative to someone else.  (Einstein)  We learned that mathematics is incomplete, and contains statements that cannot be proved by pure logic.  (Godel)  We learned that matter is not composed of solid particles with fixed positions and motions, but rather is "fuzzy" and smeared out, with uncertain positions and velocities. (Schrodinger, Heisenberg) Then Turing taught us that computers, the defining technology of the modern era, would have limits to what they could compute.  Subsequently, Arrow taught us that it is impossible to even construct reasonable and egalitarian procedures for picking a winner in an election!*  All of this set the stage for post-modernism and numerous other ideas from French intellectuals, and ultimately led to classmates in the 1990's telling me that scientific truth is just, like, a social construction, man.**

Mind you, I would happily argue that relativity, quantum mechanics, formal logic, computer science, and election theory should not be taken as death knells for ideas of objective knowledge and accurate perception.  I would be happy to argue that all of these ideas are misinterpreted.  However, if we are discussing how these topics are viewed by the educated classes rather than how they ought to be viewed then I think it's clear that the zeitgeist is favorable to the notion that humans can't really, like, know anything.  And, having just graded a stack of final exams, I am also favorable to the notions that most humans are unable to, like, know anything!  :)

*What Arrow's Theorem really says is a bit more subtle than that, but let's leave that aside for now.  Some other time I'll argue that the Gibbard-Satterthwaite Theorem can be interpreted as a theorem about conflicts of interest.
**However, the "science wars" of the 1990's, in which some humanities scholars went way too far in their critiques of science, became irrelevant after the 2000 election.  The academic community found a new common cultural enemy, and recently I read an article (can't find the link now) in which one combatant on the left wing of the "science wars" now finds himself on the side of objective scientific fact when facing right-wingers who argue that global warming is just, like, your opinion, man.