The Journals of Irreproducibly Irreproducible Results

A while back I blogged about a study of reproducibility in experimental psychology.  A critique of the methodology has just been published.  I haven't had a chance to chew on it and form a judgment, but since I blogged the original study I should blog the critique.

My first try at SF writing

I tried writing a short story.  It got rejected.  I probably won't resubmit it elsewhere, because it was written for a rather unique audience and a rather unique venue with a very low word limit.  It wouldn't be suitable elsewhere.  So I'll just put this out there to share.  But don't worry, I am working on other stories, and I have submitted one of them.  And the next one is a story that I could see myself resubmitting to other venues if it gets rejected from the first venue.


There's No Good Coffee in the Martian Lowlands

I should be working, not blogging, but my boss is the biggest idiot in the solar system, so I must vent.  I’m busting my ass on a manuscript because he wants to tell the Project Interim Review Panel that it’s under review at Nature Arescience.  Please.  We’re a year behind, we should be surveying SEC sites on the western foothills of Tharsis Montes by now, but Roger’s hung up on a mudslide on the south face. People have been finding ancient mudslides for decades, but Roger thinks that this particular find will justify our delays. Sure, the first 5,000 year-old mudslide discovery made the cover of Nature, but since then all of the mudslide finds have been in middling-impact places like Journal of Martian Morphology (on a good day).  And, yeah, the paper showing that correlated isotope signatures at different mudslide point to a global wave of hydrological activity in 2,500 BCE made Proceedings of the Interplanetary Academy of Sciences, but we don’t have that.  All we have is really unusual silt composition.  So?  That won’t make a Glamour journal.

I should have taken that postdoc offer at Armstrong Colony University, instead of joining a SEAR contractor for the Superconducting Equatorial Cable.  I’m getting all the “my boss is crazy and the money is soft” downsides of a postdoc without the creature comforts of a college habidome.  Have you tried getting Szechuan tacos or good coffee in the Martian lowlands?  I’m lucky if I can find a decent pizza.

How did I get sucked into this?  I thought I wanted to “see the world” after all those years in school, and here was my chance to literally circle the globe! Roger and his buddy Worf got the contract for a chunk of Western Lowlands and hired me, fresh out of my PhD, to join the racket that is government contracting.  Get a piece of that sweet “Scientific Exploration and Areological Record” money.  It’s silly, really.  The melting ice caps and daily microcomet bombardments are covering half the northern hemisphere with a new ocean, erasing a few billion years of areological history, and the Habitability Engineering Consortium worries about documenting the areological history of a dinky strip of land around the equator before they bulldoze it? This is what happens when you elect Blue Future Coalition to parliament. Well, a job’s a job, and at least I can tell my family I’m helping us get a magnetic field (finally!).

I should move to Earth.  Agriculture’s recovered, the gravdaptation treatments are painless these days, and we Martians are tall enough to rule the basketball court.  My grad school classmate Fiona says that aquifer engineering in Nevada is an easy transition from Areology.

Anyway, this is all Roger’s fault for relying on Worf.  Worf is from Miner stock, so he’s too frail to set foot on the surface.  Still, he can run drone support remotely, and the rules favor subcontracts with Miners because Blue Future gets votes from the orbiting districts. You’d think that it would be enough that all of the superconducting wire suppliers are Miner-owned companies, but no. The result is that ground-based projects subcontract with people whose ancestors underwent genmods for zero-g work in the asteroid belt. Total spoils system, but that’s government contracting for you. Most people just subcontract back-office support and exotic materials to orbiting companies, but Worf is Roger’s buddy, so he gets mission-critical operations.

So Worf is running drone support, for some loose definition of “support.” I’m in the hopper pod, with the full complement of sample grabs and drills, but site selection and real-time evaluation requires integration of data on nearby sites in about 10 different imaging channels.  That’s what Worf’s drones are for.  There’s no other way for a measly twenty contract teams to survey all the “interesting” stuff on the entire equator in eight years.  Except Worf’s inefficient, because he has no understanding of rock formations.  The only rock formations that he’s touched with his own hands are on Phobos.  Not the same.  Not even close. So I’m training Worf rather than collaborating with him; he doesn’t really “get” that when I’m sampling cliff walls I need data on what’s downstream, even if it isn’t on an SEC site, more than I need data on an adjacent branch of the delta.  He’s never walked a lava field and doesn’t appreciate the cues for picking features out of those fractal landscapes. I kept begging Roger to hire a second drone coordinator and Roger kept giving shifty answers about budgets.

A month ago, though, Roger went into full panic mode.  HEC announced new rules for Project Interim Reviews, and Roger is suddenly obsessed with showing “advancement of public knowledge via high-impact dissemination.”  So not only are we behind on surveying, now we have to show that what we do have is attention-getting.  That did, however, give Roger an “out.”  He just has to show that we’ve been emphasizing quality over quantity.  His solution?  Send a paper to Nature Arescience.  Getting published is Not Happening, but if we avoid a desk rejection then Roger can make up some story about a “highly interested” editor to keep from getting sunk in interim reviews.  As long as it’s sent before the reviews and gets rejected after the reviews, we buy two years.  Two years is a long time on Mars.

So, here I am, procrastinating from this manuscript, because knowing that Roger will bug me in the morning just isn’t motivation to write.  On the bright side, in the time that it took me to blog this, Fiona got my message and replied.  After I tease her about Vegas debauchery I absolutely MUST get back to work.

Copyright Alex Small, 2015

No, see, the revolution was just never tried

In the previous post I noted that personality surveys are the easiest things in the world to game.  I closed with a link to an article on a college that decided to give applicants a survey on study skills, which is pathetically easy to game.  Well, just google the name of that school and "locus of control" (a psychological term related to what the school believes they will be measuring.)  See what you get:  Admissions coaches!  WHO COULD HAVE PREDICTED THIS??!!11?1!?1!!????111!?

You cannot game your way out of advantaged kids gaming the system. By definition, the advantaged will out-game you.  Especially in an asymmetric game where the first mover loses their advantage by carving in stone the rules that the second mover can respond to and the first mover is bound by.

What is measured improves, but nothing else does

Oh, good!

A recent update to federal education law requires states to include at least one nonacademic measure in judging school performance. So other states are watching these districts as a potential model. But the race to test for so-called social-emotional skills has raised alarms even among the biggest proponents of teaching them, who warn that the definitions are unclear and the tests faulty.

The fault, dear Brutus, lies not in the tests but in the stakes.  Go ahead and create the world's most perfect, reliable, valid, accurate, and all-around good measure of grit, or growth mindset, or whatever.  Administer it in a setting where nobody has been primed to answer disingenuously and nobody has any real reason to deceive.  You'll get results that tell you something meaningful.  Now attach stakes to it.  And watch as the teachers start feeding students slogans about grit and growth mindset, and students start giving answers that sound good rather than answers that accurately tell you something about their actions outside of the testing situation.

Fortunately, a few of the people interviewed in the article agree:

In their paper published in May, Dr. Duckworth and David Yeager argued that even if students do not fake their answers, the tests provide incentive for “superficial parroting” rather than real changes in mind-set. 

“You think test scores are easy to game?” said Martin West, a professor at the Harvard Graduate School of Education, who is working with the districts in California. “They’re relatively hard to game when you compare them to a self-report survey.”

I actually don't fault the entire idea of measuring non-academic things.  The very fact that people will try to game measures is actually an argument for measuring more things rather than fewer.  The more things you measure the harder it is to game all of them at once.  Still, it's better to make more of those measures be about things that are harder to game.  As opposed to this.

This concludes our monthly feature titled "Something actually interesting in the NYT education section."

More about controversial speakers

At the moment CSULA is saying that Ben Shapiro can speak at some future event that includes an opposing viewpoint.  This sounds nice and open and pluralistic on the surface.  However, there are two problems.  The first is that it is being applied selectively.  Most speeches are allowed to go forward without a rebuttal.  There's something to be said for the educational value of structuring all formal activities as debates rather than monologues, but if that educational notion is going to be practiced then in order to avoid becoming censorship it has to be practiced consistently, not selectively.  Otherwise, you're saying that certain viewpoints need no rebuttal or moderation, but others are dangerous and need official opposition.

Second, who's to say what constitutes a valid or sufficient degree of opposition?  If a biologist is talking about research that might lead to a vaccine against a particular disease, is it sufficient to bring in another scientist who believes that the evidence more strongly favors targeting a different molecule?  Or do they have to bring in somebody who believes that vaccines cause autism?  Or somebody who believes that the researcher is pursuing a scientifically sound line of inquiry but also believes that vaccines should be a matter of personal choice rather than a public policy mandate?

If somebody wishes to speak in favor of tighter regulation of stock trading, should the opposing speaker be somebody who believes that trading by humans doesn't need further regulation but agrees that electronic high-frequency trading is dangerous?  Somebody who believes that we should have less regulation of trading?  Somebody who believes that the debate over regulation of financial markets distracts from the bigger issue of problems with capitalism itself?  Which of those viewpoints is sufficient rebuttal?

The most painful compromise of my academic career thus far

The Cal State system has done many things that deeply offend me, and the CSU has forced me to make many compromises, but the very worst compromise that the CSU has ever pushed me into is defending a smarmy little shit like Ben Shapiro. He had been invited to give a speech to a student organization at Cal State LA (our sister campus), but the campus president canceled his speech because he's an obnoxious firebrand who says offensive things. If the CSU were properly upholding the value of free expression I wouldn't have to say one damn word in defense of Ben Shapiro. I could simply do what I was planning to do, which is to ignore him. Instead, I now feel a responsibility to say that Ben Shapiro is in the right and academics are in the wrong. WHY ARE YOU MAKING ME DO THIS?????

You think I like defending Ben Shapiro? Believe me, I don't, but we live in a world with academic values, and those values need to be guarded by people with convictions. Who's gonna do it? You? I have a greater responsibility than you can possibly fathom. You weep for being offended, and you curse academic traditionalists. You have that luxury. You have the luxury of not knowing what I know. That Shapiro's speech, while repugnant, is necessary for free expression to thrive. And my stance, while grotesque and incomprehensible to you, makes spaces safer than you'll ever make them. You don't want the truth because deep down in places you don't talk about in safe spaces, you want me in that committee, you need me in that committee. We use words like academic freedom, debate, and free expression. We use these words as the backbone of a life spent defending something. You use them as a slur. I have neither the time nor the inclination to explain myself to a scholar who reads and writes under the blanket of the very academic freedom that I am trying to defend, and then questions in the manner in which I defend it. I would rather you just said thank you, and went on your way. Otherwise, I suggest you donate to FIRE. Either way, I don't give a damn what sort of space you think you're entitled to.

You want a free European college? No, you don't. Trust me, you don't. I do, but YOU don't.

This article outlines some of the substantial differences between US and European universities, and they go much deeper than the much-lampooned "climbing walls." (I'm pretty sure that European universities could add a gym for, well, the price of a gym membership.) There are differences in the educational model and the socialization provided. These differences are not just recent things; in the 19th century (and earlier) US academics were noting that our universities and colleges offer undergraduate education at a lower level than their European counterparts.  All of this has happened before and will happen again.

If you want giant lecture classes and sink-or-swim academic models, with physics taught at a far more rigorous level than we currently do it, believe me, that can be provided, and it can be provided more cheaply than the current model. I am positively salivating at the thought of offering it. Shit, I'll take a pay cut to offer it. I have a list of students to fail, and they include YOUR favorite students. Yes, I'm talking to YOU.  Your precious little offspring who gets B+ and A- grades in a California k-12 school?  Your poster child for your grant-funded program?  This One Student who is getting mostly B's but is So Nice And Friendly?  I'll give all of them F's.  ALL OF THEM!!!

Anyway, whether you prefer the US model or the European model depends on what you want from a system, but realize that the differences in educational model are directly related to the differences in funding model.