The Journals of Irreproducible Results

A group of psychologists recently attempted to replicate 100 studies published in psychology journals in 2008.  The results are in, and they aren't great.  Roughly a third of results were replicated at a statistically significant level.  Before I unpack some of the implications, I should put in an important qualifier, namely that this does not mean that those studies should not have been published or that the investigators did something wrong!  A study isn't conducted and published in order to provide the final word, but rather to invite the rest of the community to scrutinize the work and attempt to replicate it.  If a conscientiously-conducted and well-designed experiment gives a statistically significant result (positive or negative), that result is fair game for dissemination, discussion, and further scrutiny.  This replication effort is not evidence that science is not being done, but rather evidence that science is being done right.  We need more replication efforts, not fewer studies.

That said, it should give pause to people who point to a study and make prescriptions.  I've written before about the cult of This One Study, and the challenges of actually reproducing so many psychology studies point up the follies of that cult.  Moreover, as challenging as it is to get reproducible results in psychology, many educational research studies that I read do an even worse job on blinding participants and maintaining proper controls,  Some of that is inevitable, given the nature of classroom work, but some of it is a result of people uncaringly playing with too many variables at once.  If the person teaching the class is enthused about trying a new educational method and transmits that enthusiasm, and also varies the topics and assignments, while the person in the "control" group is not doing anything to get fired up and refreshed, you can't treat this as a single-variable experiment.  Moreover, often educational experiments are tapping into deep wells of cultural baggage; will the experiment get the same effect if it is conducted with people from the opposite side of that cultural divide?  Finally, it has been pointed out to me by at least one person trained in statistics and educational psychology that when you have one class section taught by New Method and another class section taught by Old Method, even if you have hundreds of students in each section you still have n=1 in the experimental and control groups, not n=hundreds, if your goal is to study methods rather than students.  "n=1" is a fancy statistical term for "anecdote."  Despite all of this, educational research tends to lead to a lot of teaching and prescription.

Strangely enough (or maybe not strangely at all, depending on your degree of cynicism), I suspect that Right-Thinking People will delightedly cite this finding that psychology experiments are not always reproducible as a way of reminding people that they don't necessarily know what they think they know, but will also continue to cite their favorite studies as rhetorical cudgels.  Consistency conshmistency!

Surprising data on student loans

Like just about everyone else out there, I took narratives of spiraling student loan debt at face value.  Why?  I don't know.  I'm not normally one to take popular narratives at face value.  But I did.

Big mistake.

Via Matt Reed, I learned of an analysis by higher education policy analyst Robert Kelchen, who used data from the Department of Education to show that the annual rate of borrowing for undergraduate education (whether subsidized  loans, unsubsidized loans, or parental loans) has been decreasing, even as the  amount of Pell grants given out has gone down and state legislatures have proven reluctant to hike higher ed spending.

If this is the case, why does the media constantly tell us that ever-spiraling college education costs are pushing more students into debt?  If I had to guess, I'd guess that it has something to do with private colleges and universities, who notoriously jack up their "sticker price" and then give almost everyone some sort of scholarship, financial aid, or other discount.  Never mind that net price hasn't budged much in comparison with sticker price.

The next question is why the media would nonetheless push this narrative?  Well, first of all, the media loves a good story, and certainly nobody has ever reacted positively to even a small price increase.  So it gets attention.  Also, the sorts of people who write for certain media outlets tend to be in a certain socioeconomic class, a class that (1) will definitely not send their kids to public universities and (2) might actually be expected to pay something close to sticker price.  If a NYT columnist gets sticker shock from the prices at Princeton these days, that's a national crisis!

Mach-ing sense of buckets

I've thought more about why Mach worried over Newton's bucket, and I understand it better now:  Mach took Galileo seriously.

Galileo really laid down most of the intellectual foundations for Special Relativity.  I'm not trying to take anything away from Einstein there; the fact that he actually found a way to add something to Galileo's work is pretty damn impressive.  Anyway, if you take those foundations seriously, then the idea of absolute acceleration should bother you.  For linear motion, we can't really know for sure if we are falling toward the earth or the earth is falling toward us.  Or, more accurately, we cannot use purely local measurements to determine who is at rest and who is accelerating.  We are not stopped from attributing causes to motions (e.g. we could notice that we are feeling the attraction of a giant rock below us, while the earth is only feeling the attraction of a miniscule flesh bag filled with water, or we could notice that the train has an engine in it while the earth has no engine attached to it), but even if we conceptually think of things "really" moving and "really" standing still, we need laws of physics that make no such distinction.  The fundamental equations of physics have to work equally well if I assume that the train is moving or that the ground is moving, even if there is no law of physics to prohibit an intelligent physicist from asking "Well, which object has a fuel source?"

So, for linear motion the idea of absolute acceleration is clearly problematic.  Galileo surely knew that, even if he dwelt more on relative velocity than relative acceleration.  If we take that notion seriously, we should be a bit bothered when Isaac Newton comes along with a spinning bucket and declares "I know for certain that this is really accelerating."  I mean, it's obviously true--Newton put his hands on the bucket and set it in motion; he didn't apply a large enough force to change the earth's spin--but a person who has read enough Galileo should worry about whether it's a statement that we can fit into the framework of the laws of physics.

There are a few reasons why the modern observer might not be as bothered as Mach:
1) Inertial and non-inertial frames are different.  Hence special relativity is studied by sophomores while general relativity requires a whole lot of differential geometry background.

This is also true, but it was not yet clear just how different inertial and non-inertial frames are.  In fact, answering Mach's challenge on that point was one of Einstein's key motivations.

2) In a uniformly accelerated reference frame there is no special point in space.  In a rotating reference frame, there is an axis of rotation, relative to which no observer changes his distance.  That's clearly a broken symmetry.

Again, I agree, but Noether's Theorem was not yet in the toolkit of practicing physicists.  Lagrangian mechanics certainly made it apparent that symmetry was important, but the full significance of symmetry was not yet manifest.  If you  just look at the Euler-Lagrange equations you can immediately see that some coordinates are ignorable and thus lead to conserved quantities, but Noether proved that you don't just have to limit your analysis to obvious coordinates--any symmetry transformation that leaves the Lagrangian invariant (a notion that's a bit more subtle than just picking a coordinate system and doing the algebraic gymnastics to rewrite variables) also gives us a conserved quantity.

My conclusion is that Mach was asking the right questions.  I do wish that he had left out the part about "fixed stars" (Halley had dispelled that notion) but his basic program was a sound one.  He wanted to know how it was that the bucket constitutes a valid example of absolute acceleration when Galileo had taught us to be skeptical to our core when somebody says that they can measure absolutes related to motion.  And, sure enough, Einstein showed that there is no fundamental way of distinguishing between acceleration and gravity, just as there is no way to tell if my car is moving and the ground is standing still or vice-versa.  Mach pushed hard on the right question, and Einstein was right to revere him.