Current Reading

This blog is primarily for me to blog my responses to books that I'm reading. Sometimes I blog about other stuff too, though.

I'm currently reading Measuring Success, an edited volume on standardized tests for college admissions. The editors are Jack Buckley, Lynn Letukas, and Ben Wildavsky. There are numerous contributing authors.

Word cloud

Word cloud

Tuesday, April 17, 2018

The restlessness

In response to a recent Kevin Drum post on how reading scores are up but nobody wants to talk about it, the bloggers at West Coast Stat Views have a theory:
My opinion: because there is a lot of money in  education and it won't be possible to "disrupt" education and redirect this money if the current system is doing well.  Notice how there is always a lot of money in being a disruptive company, at least for the top management (see Uber -- it is clear that it pays better to run Uber than it does to run a traditional Taxi service).  
It also moves the goalposts.  If everything is falling apart then it isn't such a crisis if the disrupted industry has teething issues once they strip cash out of it to pay for the heroes who are reinventing the system.   
But if current educational systems are doing well, and slowly improving through incremental change, then it is a lot harder to argue that there is a crisis in education, isn't it?
I think that the shilling is definitely half of it.  Failures in the status quo can justify more money for something else.  But the other half of it is a restlessness, a refusal to accept that there are limits to what education can do.  We don't just want schools to produce some good results, we want them to fix all of the problems in the world.  And they can't. No educational innovation will fix that, so eventually people ask educational institutions to shed the pretenses and openly become much more. Look at this post by Dean Dad, asking for community colleges to effectively become full-scale social welfare programs in order to plug achievement gaps.  The moral case for providing for the poor is quite straightforward, but nobody asks the local free clinic to teach college composition.

Anyway, if this were only about marketing disruption, the teachers' unions would be saying very loudly that their results are more than good enough.  Some do, but it's always tempered by lamentation over how much more needs to be done.  And some of that is, of course, a call for more resources for themselves--failure can justify more resources for you as easily as it can justify more resources for your replacement.  Some of it is also a heartfelt conviction that schools need to do more.  The right and left both want the impossible from schools--the left wants to provide people with free schooling on how to catch a fish, the right wants to make sure that nobody can say "I was never taught how to catch a fish" as an excuse for not having anything to eat.

Saturday, April 14, 2018

Buzzwords, information problems, and the self-sucking straw of administration

"I know words.  I have the best words."
--Kremlin spokesman

I have come to the conclusion that universities are trying to allocate resources to academic departments based on the professors' scores on the verbal and analytical writing sections of the GRE.

Let me explain.

Let's say that a department wants to hire more faculty.  Well, so does every other department on campus.  If funds are scarce (and they always are), you can't grant every request, so you have to prioritize.  You might start by looking at student/faculty ratios, something nice and objective and obviously tied to something that matters: teaching classes.(See footnote 1) However, everyone is short-staffed, so that criterion won't narrow down the list very much.  You could make a subjective value judgment to prioritize certain fields over others, but (1) that will earn you a lot of enemies, (2) even if you go there, you can't COMPLETELY neglect the other fields (on a STEM-focused campus somebody still has to teach humanities, and on a humanities-focused campus you still need some science classes), and (3) broad focuses still don't narrow things down.  So what if you decide that your College of Business is the most important thing on campus?  Within that college, will you give the next hire to the finance people or the marketing people?

So eventually you reach a stage where you need to make decisions based on specifics.  Now, if you're close enough to the field, you have detailed knowledge of what the department needs and what they would do if they got people with different types of expertise.  Your decisions still have subjective elements, but they are also informed by real knowledge of specifics.  The decisions are not superficial or silly, even if the people who make them are still fallible.

But what if you aren't close to the field?  Or maybe even somewhat close to the field, but not close enough to know for sure who will make the best use of resources?  I mean, I'm knowledgeable about math, and I have friends in the math department, but I can't sit down and say for sure whether our math department has greater need for an applied mathematician or a pure mathematician.  And I certainly can't say for sure whether the next pure mathematician should be a topologist or logician or algebraist or whatever else.

So what administrators do is they ask departments to explain how their new hire will align with campus priorities.  This might sound like a good idea on the surface, but any half-decent department can say (with at least some degree of truthfulness) that the person they're proposing to hire will be useful to the campus.  If (hypothetically) I wanted to hire an experimental particle physicist, I could align them with just about any plausible priority of the campus.  Critical Thinking?  "Experimental particle physicists will involve students in projects that require detailed data analysis."  Student Success?  "This faculty member will teach core classes required for success in the field."  Career Relevance?  "This faculty member will involve students in projects that teach them instrumentation and data analysis skills relevant to industrial careers in STEM fields."  Global Engagement? Diversity? "This faculty member will involve students in research projects as members of international collaborations with people from around the world."  Community Outreach?  "Particle physics is of high interest to the public, with books and public lectures on the subject being quite popular."

So I can fit this sort of physics professor into almost any buzzword that the people above me might decide to emphasize, and with a little thought I could fit just about any other plausible hire in my discipline into just about any other plausible buzzword.

At this point one might say "OK, so what's the problem?  Buzzwords don't have to get in your way! You can work around them!"  Well, first of all, it's a stupid way to do things.  If resources go to whoever writes the best essay on "How my desires fit with your buzzwords" then it amounts to awarding resources based on verbal ability.  Maybe I should be fine with that--more than fine!--because I like to read and write.  I could be King of STEM without ever doing another calculation or experiment, because I can use words.  But I also have some honesty in me, and I know it's a dumb way to do things.  It disconnects resources from facts on the ground, and instead aligns them with sophistry.  It comes back to what Timothy Burke said about the managerial classes and popular resentment thereof:
We need to identify the necessary heart of our established systems and practices, whether it’s in a small non-profit, a government office, a university, or a corporate department, and be ready to mercilessly abandon the unnecessary procedures, processes and rules that have encrusted all of our lives like so many barnacles. Those of us who are in some sense part of the larger networks of the Establishment world, even at its edges, can endure the irrelevance of pointless training sessions, can patiently work through needless processes of measurement and assessment, can parse boring or generic forms of managerial prose to find the real message inside. We’ve let this kind of baroque apparatus grow up around the genuinely meaningful institutional systems and structures that we value because it seems like too much effort in most cases to object against it, and because much of this excess is a kind of stealthy job creation program that also magnifies the patronage opportunities for some individuals. But this spreading crud extends into the lives of people who are not primed to endure it, and who often end up victimized by it, and even for those of us who know our way around the system, there are serious costs to the core missions of our institutions, to clarity and transparency, and to goodwill. It’s time to make this simpler, more streamlined, more focused, without using austerity regimes or “disruption” as the primary way we accomplish that streamlining. We don’t need to get rid of people, we just need to get rid of the myriad ways we acquiese to the collection of more and more tolls on the roads we traverse in our lives and work.

It would be more sensible to accept that if you've already decided to have programs in Business, Humanities, Social Science, Natural Science, Engineering, and so forth, then you have to trust that they have a good reason to be there, and instead of constantly asking people to argue their subjective intellectual merits in terms of buzzwords you should spread the resources around while demanding results measured by a few tangibles.  Thus we come to the problem identified by Hayek, namely that you can't really solve information problems without tangibles, and in the absence of profit numbers you'll have to look at something else.  Most measures can be gamed, and hard standardized tests produce unequal results.  So you'll have a hard time measuring educational tangibles in a way that doesn't corrupt the process but does satisfy political and societal needs.  While simultaneously satisfying a public that distrusts educational systems and wants hard accountability.

So what happens instead is that power accumulates in the hands of the people who write Strategic Plans and Assessment Reports and whatnot.  They create processes, impose those processes on us, and then suck up resources to further refine and expand their processes.  It's a self-sucking straw.

Of course, none of this is new, and higher ed is actually late to the game.  My mother was a nurse.  When she entered the profession nursing was a 3-year degree rather than a Bachelor's degree.  By the time I was in middle school a BSN (Bachelor of Science in Nursing) was needed to keep getting good jobs, so she went back to college to get her BSN. She found that programs had shifted from hands-on clinical skills to writing Care Plans that align with whatever jargon was being pushed by healthcare administrators.  Healthcare is far from a perfectly objective field (e.g. What counts as a "good enough" outcome when full recovery is not in the cards?  How much responsibility does the care provider bear when the patient's outcome depends in part on the patient's compliance with treatment and lifestyle changes?) but "Is the patient able to walk on their own again?" is still more objective than "What does it really mean to say that a student understands this philosophical tract?"

Meanwhile, I've heard from people in k-12 that you can do anything you want as long as you have lesson plans that summarize each day's activities in whatever jargon is in style.  This bullshit is actually hitting higher ed late, not early, cutting against the idea that we originated it in our Colleges of Education or Business or whatever.  It seems to be a thing cutting across many segments of society.  It's a disease of the bureaucracy, and it's metastasizing.

Also, some might wonder why I'm singling out historically feminine professions like teaching and nursing, but the military reportedly has similar amounts of buzzwords and kool-aid, and I don't just mean the indoctrination (in the best possible sense of the word) needed to get a man to charge into a hail of bullets for his country.  I mean managerial fads that promise quick fixes to intractable problems in the endless series of unwinnable conflicts that America has been involved in over the past several decades.  If the unsolvable problems of human nature and inequality drive educational leaders to embrace buzzwords and kool-aid, imagine how much more kool-aid you'd need in order to persuade yourself that you've accomplished something in Vietnam, Iraq, or Afghanistan.

Finally, I wonder if there's really anything new about awarding resources on the basis of verbal prowess.  Besides everything we could say about charming salesmen and whatnot, writing began as a tool of administration.  The oldest writings that we have are, for the most part, not religious texts and epic poems but rather government and business records:  Harvests, taxes, land allocation, contracts, treaties, etc. Mesopotamian warrior-kings left fewer written traces than municipal administrators.  I suppose that this is how advanced civilizations have always done things.  All of this has happened before and will happen again.

In that sense, Black Panther's Wakanda is incredibly unrealistic.  I don't have a problem with the premise of a secret and super-advanced society with amazing technology.  That's the premise, and if vibranium seems unrealistic just remember that it's a metal in the plotonite family of compounds, with properties determined by plotobolic mechanisms at the molecular plotting scale.  No, what bothers me about Wakanda is that such an advanced society would choose a leader via ritual combat.  Any such society would choose its leader from among elites who were educated in engineering and then promptly pressed into administrative roles.  Shuri, the smartest engineer in Wakanda, would spend most of her time complaining that she's been pushed into administration and is busy writing Strategic Plans instead of making stuff in the lab.  In fact, she'd probably be elevated over her meat-head brother (who just likes to work out and fight) and would be queen while he is a field agent.  (Until he gets promoted to the rank of General and fights from behind a desk rather than inside a suit of armor.)

(1) Note that teaching classes is only the fifth most important thing in academia:  Alumni donations, research, athletics, and parking ticket revenue all rate higher priority.

Thursday, April 12, 2018

In other news, the Pope is Catholic

I came across this meta-analysis of studies of the effect of class attendance on college grades.  A meta-analysis is not a single study, but rather a statistical analysis of a large body of studies.  Any one study might find a given result for any number of reasons.  However, by comparing many studies one can get an idea of whether there are any consistent findings, and put outliers in context.  That's what a meta-analysis is.  In short, these authors find that class attendance is an excellent (not to be confused with "perfect") predictor of college grades, even better than test scores and high school GPA.  Also of interest, attendance is a better predictor than personality traits (aka "non-cognitive traits"), throwing cold water on occasional claims that track record matters less than potential as evidenced by non-cognitive traits.

Sunday, April 8, 2018

Measuring success, chapters 10 and 11

These last two chapters compare test-optional institutions with similar institutions.  They focus on liberal arts colleges rather than the sort of place that I work in, but these comparisons get at the setting in which these debates are loudest.  The short version is that test-optional and test-requiring institutions followed almost identical trends from 1992 to 2010 when looking at measures of diversity, application numbers, and average test scores.  There are small differences, but they are small.  I think it comes down to the fact that test-optional initiatives are parts of much larger contexts.  Institutions that don't abandon the SAT still strive for diversity, and institutions that do abandon the SAT don't just say "OK, no need to do anything else about diversity.  We did what matters."  So the presence or absence of tests in admissions is just one factor among many.

Which is a metaphor for so many things, if you think about it.

Saturday, April 7, 2018

Judas Iscariot, Administrator

Last weekend I saw Jesus Christ Superstar for the first time.  I really, really liked it.  But then, as I was singing Judas' big musical number "Superstar" to myself, I realized that the lyrics can be interpreted as the words of a university administrator.  To wit:
Every time I look at you I don't understand,
Why you let the things you did get so out of hand,
You'd have managed better if you'd had it planned,
Translation:  "You need to engage in strategic planning."
Why'd you choose such a backwards time and such a strange land?
Academics love to talk about how horrifying it would be to live in a small town or non-coastal city.
If you'd come today you could have reached a whole nation,
Israel in 4 B.C. had no mass communication.
Translation:  "We could solve all of our problems if we jumped onto the latest digital fads and offered more online classes."
Don't you get me wrong!  Don't you get me wrong!
Don't you get me wrong!  Don't you get me wrong!
I only wanna know!  I only wanna know!
I only wanna know!  I only wanna know!
Translation: "Please submit reports.  We need to know that you are spending your time on teaching, not wasting your time on irrelevant busywork."
Jesus Christ, Jesus Christ,
Who are you? What have you sacrificed? 
Translation: "In this time of tightened budgets (for everything except ballooning administrative ranks), we need to see evidence of shared sacrifice.  Are you doing more with less?  Can you submit a report on that?"
Jesus Christ, Superstar,
Do you think you're what they say you are?
Translation:  "We've reviewed your student evaluations.  Please submit a self-reflection in response, so that we can put it in your file."
Tell me what you think about your friends at the top,
Who'd you think (besides yourself) was the pick of the crop?
Translation:  "These performance reviews needed to be done yesterday.  We'll need you to work on them.  Please fill out the rubrics I'm attaching."
Buddha:  Was he where it's at?  Is he where you are?
Could Mohammed move a mountain?  Or was that just PR?
Translation: "We are pleased to announce our new diversity initiative."
Did you mean to die like that?  Was that a mistake?
Translation:  "Could it be that your problems are a result of lack of proper strategic planning on your part?"
Or did you know your messy death would be a record breaker?
Translation: "Or could it be that your budgetary problems were a deliberate sabotage that you'd use to demonstrate need when requesting more resources?"

On the other hand, it seems that Judas' thirty pieces of silver were not worth a whole lot, whereas university administrators make far more money than the people who engage in such trivial tasks as teaching and research.

Measuring Success, Chapter 9

This chapter, by Rebecca Zwick (researcher at ETS and professor emerita in the education school at UC Santa Barbara, aka U Can Study Buzzed, aka my beloved alma mater), looks at the tangible outcomes from test-optional and top-percentile admissions.  "Top percentile" admissions let in anyone who graduates in the top X% of a public high school class.  This has certain obvious virtues (anyone who makes it to the top must be more driven than most around them), it has the potential to increase diversity (the top students in a poor, minority neighborhood get in on the same footing as the top students in a rich, white school), and it does so in a race-blind manner.  Anyway, Zwick looks at the data, and a few take-aways:

1) These admissions policies don't seem to hurt graduation rates or college grades much, if at all.  This is consistent with the finding in earlier chapters that kids with high grades but low scores (and we can reasonably assume that such kids are common among those who don't report scores or get in because they graduated at the top of a school in a disadvantaged neighborhood) do pretty well.

2) On the other hand, there is still sorting:  Students who don't report test scores tend not to major in STEM.  That isn't a bad thing, IMHO.  If a kid has a great art portfolio then they should go to college and major in art, regardless of what their SAT math score is.  OTOH, that kid probably shouldn't do physics if their SAT math score is abysmal.  The previous sentence is only disparaging if you place physics on a pedestal that towers over the arts, and I don't place it on such a pedestal.

Anyway, these findings are reassuring, because there's something of a cottage industry in newspaper articles about minority kids who do well in a non-challenging high school but then flounder at a flagship.  Such kids surely exist, and definitely deserve some compassionate counseling on alternatives, but they are apparently not a major factor in the big picture, which means they are not a major impediment to diversifying large cohorts.

3) The gains for diversity are nowhere near what people were hoping.  When you go test-optional you have to look at resumes and essays and letters, all of which are at least as susceptible to manipulation and response to class and culture as anything on the SAT.

4) In a humorous aside, regarding the legality of affirmative action and alternatives to affirmative action, the author quotes Justice Ruth Bader Ginsburg as saying that "only an ostrich" would perceive top-percent admissions plans as race-neutral.  Whether or not the effects actually match the intent, they are designed and scrutinized in a discussion about race, with everyone hoping to achieve a diverse outcome without mandating a diverse outcome.  Say what you will for or against such agendas, but I admire Ginsburg's rhetorical flourish.

Thursday, April 5, 2018

Measuring Success, Chapter 8

This is an interesting chapter.  Most of it is actually a republished chapter from Crossing the Finish Line, a 2011 book by Bowen, Chingos, and McPherson.  They used large data sets from 21 flagships and 4 state-wide systems to look at a very large cohort of students who started college in 1999, and see what predicted success.  They found that high school grades were less important than test scores, though test scores still have some predictive value.  This makes sense to me:  The ability to succeed at sustained tasks is distinct from (though not wholly unrelated to) the ability to do well on a test.  Also, a person who isn't strong at the things measured on tests might still find areas where they can succeed, while a person who can't devote themselves to regular academic work will have trouble succeeding at anything, even if they have certain mental traits.

After that portion of the chapter, some researchers at the College Board re-run that analysis with more recent data, and find similar trends, though the predictive power of grades has gone down while the predictive power of tests has gone up.  This is consistent with a hypothesis of grade inflation (per an earlier chapter).