Amazingly, doing homework problems is a good way to learn physics material

I have graded my final exam and looked at correlations between homework scores and test scores.  Besides looking at overall homework average and final exam score, I also look at the correlation between each midterm and the homework assignments with the material from that midterm.  Some points to note:

1. I give two types of homework.  The first is a long problem set full of calculations every week.  There are 9 of these in the quarter.
2. I give 1-2 shorter assignments per week that are either warm-ups for the day’s lecture topics, reviews of the previous class session’s lecture topics, or reviews of the reading.  There are 14 of these in the quarter.
3. I drop the worst long homework and 2 worst short homeworks, so that there’s no penalty for having one particularly bad week during the quarter.
4. Each midterm covers 3 problem sets and 4 short assignments.  Half of the final exam was on the last 3 problem sets and the last 6 short assignments, and the other half was a review of topics from the first two midterms.
5. I was able to break out the final exam problems related to problem sets 7-9, and correlate those final exam problems with the relevant problem sets and relevant short homeworks.
6. I also computed correlations between the total final exam score and the homework averages for the quarter (with the lowest dropped to remove penalties for a bad week).
7. There are 16 students who took all 3 tests.

Without further ado, the correlations:

	Corresponding exam or section	Final exam score
Short HW 1-4	0.46
Problem sets 1-3	0.46
Short HW 5-8	0.45
Problem sets 4-6	0.53
Short HW 9-14	0.46
Problem sets 7-9	0.63
All Short HW		0.78
All problem sets		0.81

Not surprisingly, the correlation between one test and a small number of assignments is weaker than the correlation of the final with a larger number of assignments, because when I average 9 assignments instead of 3 some noise averages out.

The fact that the homework (which is not timed) is so strongly correlated with the tests (which are timed) suggests that the homework and tests are measuring the same thing, in spite of possible issues of test anxiety, etc.  Also, while the correlations between the subsets of homework and relevant exam problems are weaker than the correlation of the average homework score with the final exam, the correlations are quite close to each other, suggesting that all of the tests are similarly representative of the homework.

Most importantly, it appears that people who work hard at homework problems do well in the class overall.  Who knew that doing homework is the best way to learn the material?

The voice of the opposition

I was interviewed to provide a critical perspective for an article about Eric Mazur in the Chronicle of Higher Education (paywall). There are a few quotes from a short interview, a link to an article I published in the Chronicle a few years ago, and a quote from a blog post.  I'm basically quoted as saying that if people take notes and organize their thoughts they can learn, that expert perspectives should be important and held up alongside group activities and whatnot, and that what may have revolutionary 25 years ago is now mainstream to the point of being nearly obligatory.

Resurfacing

I have been reading fiction lately, and the themes of this blog are much more suited to discussing non-fiction than fiction (unless we count the promises of education reformers as fiction).  But the LA Times has an article on how the Gates Foundation is beginning to acknowledge that silver bullets are in short supply, and that is worth sharing.  I used to be a big fan of the TV show Once Upon A Time, and in Once Upon A Time there are two lines particularly worth quoting in discussions of education reform:

1) "All magic comes at a price."  Said repeatedly by Rumpelstiltskin, the point is that whenever you use magic to solve a problem you might get a a quick and easy way to do something, but you'll pay for it down the road.

2) "That's the problem with this world.  Everybody wants a magical solution to their problems but nobody wants to believe in magic."  Said by the Mad Hatter, one of the few characters in the first season who knows that magic and fairytales are real, he's complaining about people who want something that's as quick and easy as magic, but somehow fits into their (supposedly) rational world-view.  They want things neat and tidy.  He was talking to a character who wanted to believe that there was a way to fix her situation without accepting the reality of other worlds and much bigger things than she had previously contemplated, things that could shatter her world and thrust her into a much bigger drama.

But it could just as easily be applied to technocrats.  They want to believe that with properly-designed studies they will find the tricks that nobody else found before, and use those tricks to solve the problems of education.  They are forgetting a few inconvenient facts.  For starters, scientific discovery is never as clean and linear as the technocrats want to believe.  You don't simply follow a best practice for research and thereby identify a best practice for teaching or social work or whatever.  True insights, let along truly new insights, are dearly bought.  They come at unexpected moments and from unexpected directions.  New knowledge is the most expensive thing imaginable. There's a reason why thousands of years ago the Egyptian scribe Khekheperre-Sonbu, living in a time when so many of today's great innovations were unknown, lamented the difficulty of developing new and meaningful ideas.

Also, they want to think that success is just a matter of telling people to do the right things and then watching it happen.  Just today I was talking to a couple people who found in their research that teaching assistants who were sent to a seminar in which they were told about the (supposedly) best practices for teaching would start out following those practices but would then deviate.  Their belief was that they need to find some way to persuade or train the teaching assistants to stick with the best practices.  Well, first, let me note that maybe some of these practices are only best for particular people in particular settings, and the teaching assistants might not be the right audience for these practices.

But second, getting people to do the right thing is more complicated than telling them to do the right thing.  Perhaps the best way to instill a practice is not with a seminar that meets for a couple hours once per week over a period of 10 weeks, but rather to provide a role model.  Perhaps they need to take a class from an inspiring teacher who uses the (supposedly) newer and better practices.  Perhaps that is a better way to instill something into the brain.  If you want people to believe down to their very bones that a particular method of teaching physics (or whatever subject) is better than any other method, perhaps they need to actually experience the method itself, and actually learn and appreciate physics via that method, and experience enough growth and insight into physics that they will emulate that method as much subconsciously as consciously.  Perhaps role models and meaningful experiences matter more than simply hearing and believing best practices.

Think about how much we emulate our parents, and ask yourselves this:  Do you do what your parents told you to do, or do you do what your parents did?  You probably got some good habits from them, but did you get any bad habits from them?  And did they tell you to adopt those habits, or did your mothers tell their children not to do what they had done? (To quote a great song.)  I'm pretty sure that your parents discouraged you from emulating their worst habits, but that probably didn't stop you from emulating them (probably without consciously intending to do so).  We know from more than a century of research on k-12 education that parents and family background predict educational outcomes better than any practice on the part of teachers.  Why?  Because people spend far more time around parents than around teachers, and (usually) get more attention from parents than teachers.  (If they get more attention from teachers than parents then there's an entirely different set of problems here.)  Likewise, people spend far more time with teachers trying to teach them subject matter than teachers trying to explicitly teach them about teaching.  Perhaps the best way to shape the next generation of teachers is to teach subject matter as well as possible and inspire people to do what you did, not to spend 2 hours per week over 10 weeks telling people about best practices.

Of course, teaching the subject matter as well as possible is slower-impact than a special seminar, even if it is higher-impact.  And we want the fast solution in our society.  We want to believe in the magic without accepting that when the magic actually does happen it happens in a much bigger context.  So I'm quite confident that when Bill Gates has left the stage Mark Zuckerberg or some similar figure will attempt to implement an education reform agenda, convinced that This Time It's Different.  All of this has happened before and will happen again.

All of this has been shilled before and will be shilled again

I'm looking at a bunch of things I read a few years ago.  Here's a fascinating infographic from the Brookings Institution:

I don't have any problem with the basic concept of parsing out different segments of the STEM job market,m and I certainly do NOT object to  people who note that not all STEM jobs need a 4-year degree.  At the same time, I have this nasty feeling that definition creep can easily lead to even more inflated estimates of our country's alleged "need" for STEM graduates.  You start with a broad definition, one that isn't just about 4-year grads, and get a large number of jobs.  Then you point to the large number while arguing for the importance of a particular segment, without noting any of the crucial context or caveats.

All of this has been hyped before and will be hyped again

This is a fascinating survey of hype over educational technology.  Naturally, though, I must remind the reader that this time it's different!

Galileo had balls

Prompted by a post by Gene Callahan's recent post, I have thought more about Galileo's thought experiment concerning the two balls (of different mass) tied to each other.  Galileo attempted to prove the inconsistency of Aristotelian physics by arguing that two balls tied to each other are a composite object of greater mass than either, so they must fall faster than either constituent ball, but at the same time the lighter ball has a natural tendency to fall slowly and so it will try to slow down the heavier ball.  Based on this, Galileo concludes that the model is inconsistent and so the balls must accelerate at a mass-independent rate.

This is, of course, wrong in one very important sense:  In a world with air resistance things do fall at mass-dependent rates!  Yet, at the same time, we know that Galileo is right about purely gravitatinal effects.  So, how did he get that right, and was it a coincidence?

Well, first, he got it right about purely gravitational effects because of an embedded assumption of inertia.  He is assuming that the lighter ball is trying to retard the motion of the heavier ball, which means that he's assuming that it's harder to move if you're pulling something.  While that's not a fully Newtonian model, it has a general idea of inertia in there.  If you assume a mass-dependent resistance to changes in motion then adding mass to an object should slow its acceleration, which can (if the dependencies are the same) cancel the mass-dependent pull of the earth.

Galileo got this right, but it was partly a coincidence because these cancellations require that mass enter into inertial and gravitational effects in the same way, so that you can divide it out.  Still, he got the qualitative idea.  I wouldn't recommend acceptance of his theoretical arguments at a journal today, but I give him full marks for groping toward the right physical idea and making a major leap in 17th century physics.

The other reason Galileo got this right is that he made the Aristotelian assumptions explicit while keeping inertia implicit.  When your suppositions lead to a contradiction it means that at least one supposition is wrong.  If you only make one assumption explicit then you'll reject that assumption, rather than concluding that there's a tension between two assumptions and we need more investigation to figure out which is wrong.  Galileo may not have realized to what extent he was invoking an assumption of inertia.  I'd probably need to read Two New Sciences to see how well he understood inertia.

Advantage: Advantaged.

A few years ago the Obama administration announced that it was going to put out a "college scorecard" showing data that would help students evaluate colleges.  Supposedly this was Going To Change Everything.

Well, the Chronicle of Higher Education has looked at early results, and it appears that the students who made the most use of this information are students from advantaged backgrounds.

The earnings data, however, did make a difference. (While there have been other efforts to measure postcollege income, this particular metric was new.) Colleges with higher-than-median earnings saw higher-than-expected growth in scores sent in the months after the September release, while those with lower-than-median earnings had lower-than-expected growth. The researchers estimate that for every 10-percent increase in earnings, the number of scores sent rose 2.4 percent.

But when the researchers dug deeper, they found that only some students had been swayed by the new earnings information. "The impact," they wrote, "is driven almost entirely by well-resourced high schools and students." When the researchers looked at parental education, the Scorecard’s impact was concentrated among students whose parents had at least some college. When they looked at race, it was concentrated among students who were white and Asian. And when they looked at school type, it was concentrated among those attending well-resourced public — and even more so, private — schools. 

"The subgroups of students expected to enter the college-search process with the most information and most cultural capital," the researchers wrote, "are exactly the students who responded most strongly to the Scorecard."

If you think about it, it makes sense:  The people who will make the greatest use of information will be those who are the most sophisticated at using information or have people to help them make sophisticated use of information, and those will mostly be students who are already pretty advantaged.  It's not so different from Kentaro Toyama's idea of a "Law of Amplification" in his book Geek Heresy.

But, rest assured dear reader, that the next set of metrics and information resources will Change Everything and totally level the playing field.  Mark my words.  It's coming.  None of this has happened before...or something.