OK, kids, gather around and learn some statistics. Suppose that you're well-meaning and conscientious and want to make sure that colleges are doing the best possible job in admitting students and selecting them only on the basis of meaningful variables, not meaningless garbage input. So you recommend that every college--EVERY!!!--do a study of whether its admissions criteria actually predict success, and particularly whether standardized tests predict success on their campus.
There are two problems with this. One is actually an ethical problem that gets to the heart of equity concerns: Why should predictors of success be the only factors in determining who gets an opportunity to study at a college? You might argue that predictors of success are pretty important in justifying that investment of time, money, and other resources, or you might argue that giving a chance to people who are less likely to succeed is a justifiable endeavor. We can't decide between those propositions empirically. We can use empirical studies to determine if a test, or a GPA, or an essay, or an interview, or whatever "is" a statistically sound predictor of success, but we can't use empirical studies to decide if opportunities "ought" to be extended only on the basis of statistically likely success. That "is"/"ought" distinction is something that philosophers have long discussed. One could determine that a given test or whatever "is" a sound predictor but a student "ought" to be admitted in spite of a low score (or whatever) because of a commitment to opportunity. Whether you agree or disagree with that action probably depends on just how weak the student is, but resolving that conundrum is ultimately a value judgment.
But there's a second problem: Let's say that you've decided that you care about the statistical power of different predictors of success. Your "ought" questions are resolved, but answering "is" questions is tricky. If a student gets into your college with low grades, or low test scores, or weak extracurricular accomplishments, or weak letters of recommendation, or whatever it may be, then they're probably either strong by some other measure or else they're from a rich family and your development office has a lot of internal political clout. The second issue is beyond our scope here, but the first issue is a big one. If many of the students with low test scores are strong by some other measure, and many of the students with high test scores may not be as strong by some other measure, then it's hard to do "apples to apples" comparisons. If one group does better than another you won't know if it's because of how they differ on one measure or how they differ on another measure. If both groups perform similarly you won't know if the differences between them are actually meaningless, or if the differences simply cancel out. If the differences are meaningless then you shouldn't look at test scores (or whatever) at all. If the differences cancel out then you should absolutely look at test scores, but also look at other variables.
Now, some of you may be thinking "Wait, why is it that you're assuming people will only be strong by one measure but not another?" I'm not assuming that everyone fits that dichotomy, I'm just assuming that the people who are weakest by one measure could only get in by compensating in some way, while people who are average or stronger by that measure have more latitude on the other criteria. Of course there will be people who are strong by multiple measures, and looking at them will only tell you that being talented and accomplished and well-prepared by multiple measures is a good thing (but we already knew that). But there's a good chance that the people who are strongest by multiple measures will be poorly-represented at your school because they got into a more prestigious place. So the range of students that you can observe at most schools will be limited to a narrow band composed mostly (but not exclusively) of people who are decent by multiple measures but not amazing and people who are strong on one measure and weak on another. The result is a phenomenon called "range restriction" in the statistical literature.
There are ways to attempt to correct for range restriction, but those techniques work best if you have a decent understanding of the relationship between your sample and a wider pool. The best thing is to look at a wider pool that includes not only your students but also the people who were worse (or better) than most of your students and went elsewhere. This is why it's best if each school not attempt its own DIY social science, but instead people look at the wider literature and larger studies.
Thursday, June 16, 2016
Tuesday, June 7, 2016
Defeated by a Russian...and it's summer!
I can't read anymore of Anna Karenina. I can't. I tried, and made it 100 pages, but it's so slow, and the day-to-day activities of these pampered rich people are so mundane, so boring, that I keep hoping some Communist revolutionaries will come along and send them to Siberia. I actually like the inner lives of these characters, but I hate their mundane activities, and I wish the story would move. I toughed it out for 120 pages and that was enough.
Sorry, Tolstoy. Maybe I'll try War and Peace one of these decades.
My next book is Academaze by friend-of-the-blog Sydney Phlox, aka Xykademiqz.
Sorry, Tolstoy. Maybe I'll try War and Peace one of these decades.
My next book is Academaze by friend-of-the-blog Sydney Phlox, aka Xykademiqz.
Monday, June 6, 2016
Amazingly, doing homework problems is a good way to learn physics material
I have graded my final exam and looked at correlations between homework scores and test scores. Besides looking at overall homework average and final exam score, I also look at the correlation between each midterm and the homework assignments with the material from that midterm. Some points to note:
Without further ado, the correlations:
- I give two types of homework. The first is a long problem set full of calculations every week. There are 9 of these in the quarter.
- I give 1-2 shorter assignments per week that are either warm-ups for the day’s lecture topics, reviews of the previous class session’s lecture topics, or reviews of the reading. There are 14 of these in the quarter.
- I drop the worst long homework and 2 worst short homeworks, so that there’s no penalty for having one particularly bad week during the quarter.
- Each midterm covers 3 problem sets and 4 short assignments. Half of the final exam was on the last 3 problem sets and the last 6 short assignments, and the other half was a review of topics from the first two midterms.
- I was able to break out the final exam problems related to problem sets 7-9, and correlate those final exam problems with the relevant problem sets and relevant short homeworks.
- I also computed correlations between the total final exam score and the homework averages for the quarter (with the lowest dropped to remove penalties for a bad week).
- There are 16 students who took all 3 tests.
Without further ado, the correlations:
| Corresponding exam or section | Final exam score | |
| Short HW 1-4 | 0.46 | |
| Problem sets 1-3 | 0.46 | |
| Short HW 5-8 | 0.45 | |
| Problem sets 4-6 | 0.53 | |
| Short HW 9-14 | 0.46 | |
| Problem sets 7-9 | 0.63 | |
| All Short HW | 0.78 | |
| All problem sets | 0.81 |
Not surprisingly, the correlation between one test and a small number of assignments is weaker than the correlation of the final with a larger number of assignments, because when I average 9 assignments instead of 3 some noise averages out.
The fact that the homework (which is not timed) is so strongly correlated with the tests (which are timed) suggests that the homework and tests are measuring the same thing, in spite of possible issues of test anxiety, etc. Also, while the correlations between the subsets of homework and relevant exam problems are weaker than the correlation of the average homework score with the final exam, the correlations are quite close to each other, suggesting that all of the tests are similarly representative of the homework.
Most importantly, it appears that people who work hard at homework problems do well in the class overall. Who knew that doing homework is the best way to learn the material?
The voice of the opposition
I was interviewed to provide a critical perspective for an article about Eric Mazur in the Chronicle of Higher Education (paywall). There are a few quotes from a short interview, a link to an article I published in the Chronicle a few years ago, and a quote from a blog post. I'm basically quoted as saying that if people take notes and organize their thoughts they can learn, that expert perspectives should be important and held up alongside group activities and whatnot, and that what may have revolutionary 25 years ago is now mainstream to the point of being nearly obligatory.
Friday, June 3, 2016
Resurfacing
I have been reading fiction lately, and the themes of this blog are much more suited to discussing non-fiction than fiction (unless we count the promises of education reformers as fiction). But the LA Times has an article on how the Gates Foundation is beginning to acknowledge that silver bullets are in short supply, and that is worth sharing. I used to be a big fan of the TV show Once Upon A Time, and in Once Upon A Time there are two lines particularly worth quoting in discussions of education reform:
1) "All magic comes at a price." Said repeatedly by Rumpelstiltskin, the point is that whenever you use magic to solve a problem you might get a a quick and easy way to do something, but you'll pay for it down the road.
2) "That's the problem with this world. Everybody wants a magical solution to their problems but nobody wants to believe in magic." Said by the Mad Hatter, one of the few characters in the first season who knows that magic and fairytales are real, he's complaining about people who want something that's as quick and easy as magic, but somehow fits into their (supposedly) rational world-view. They want things neat and tidy. He was talking to a character who wanted to believe that there was a way to fix her situation without accepting the reality of other worlds and much bigger things than she had previously contemplated, things that could shatter her world and thrust her into a much bigger drama.
But it could just as easily be applied to technocrats. They want to believe that with properly-designed studies they will find the tricks that nobody else found before, and use those tricks to solve the problems of education. They are forgetting a few inconvenient facts. For starters, scientific discovery is never as clean and linear as the technocrats want to believe. You don't simply follow a best practice for research and thereby identify a best practice for teaching or social work or whatever. True insights, let along truly new insights, are dearly bought. They come at unexpected moments and from unexpected directions. New knowledge is the most expensive thing imaginable. There's a reason why thousands of years ago the Egyptian scribe Khekheperre-Sonbu, living in a time when so many of today's great innovations were unknown, lamented the difficulty of developing new and meaningful ideas.
Also, they want to think that success is just a matter of telling people to do the right things and then watching it happen. Just today I was talking to a couple people who found in their research that teaching assistants who were sent to a seminar in which they were told about the (supposedly) best practices for teaching would start out following those practices but would then deviate. Their belief was that they need to find some way to persuade or train the teaching assistants to stick with the best practices. Well, first, let me note that maybe some of these practices are only best for particular people in particular settings, and the teaching assistants might not be the right audience for these practices.
But second, getting people to do the right thing is more complicated than telling them to do the right thing. Perhaps the best way to instill a practice is not with a seminar that meets for a couple hours once per week over a period of 10 weeks, but rather to provide a role model. Perhaps they need to take a class from an inspiring teacher who uses the (supposedly) newer and better practices. Perhaps that is a better way to instill something into the brain. If you want people to believe down to their very bones that a particular method of teaching physics (or whatever subject) is better than any other method, perhaps they need to actually experience the method itself, and actually learn and appreciate physics via that method, and experience enough growth and insight into physics that they will emulate that method as much subconsciously as consciously. Perhaps role models and meaningful experiences matter more than simply hearing and believing best practices.
Think about how much we emulate our parents, and ask yourselves this: Do you do what your parents told you to do, or do you do what your parents did? You probably got some good habits from them, but did you get any bad habits from them? And did they tell you to adopt those habits, or did your mothers tell their children not to do what they had done? (To quote a great song.) I'm pretty sure that your parents discouraged you from emulating their worst habits, but that probably didn't stop you from emulating them (probably without consciously intending to do so). We know from more than a century of research on k-12 education that parents and family background predict educational outcomes better than any practice on the part of teachers. Why? Because people spend far more time around parents than around teachers, and (usually) get more attention from parents than teachers. (If they get more attention from teachers than parents then there's an entirely different set of problems here.) Likewise, people spend far more time with teachers trying to teach them subject matter than teachers trying to explicitly teach them about teaching. Perhaps the best way to shape the next generation of teachers is to teach subject matter as well as possible and inspire people to do what you did, not to spend 2 hours per week over 10 weeks telling people about best practices.
Of course, teaching the subject matter as well as possible is slower-impact than a special seminar, even if it is higher-impact. And we want the fast solution in our society. We want to believe in the magic without accepting that when the magic actually does happen it happens in a much bigger context. So I'm quite confident that when Bill Gates has left the stage Mark Zuckerberg or some similar figure will attempt to implement an education reform agenda, convinced that This Time It's Different. All of this has happened before and will happen again.
1) "All magic comes at a price." Said repeatedly by Rumpelstiltskin, the point is that whenever you use magic to solve a problem you might get a a quick and easy way to do something, but you'll pay for it down the road.
2) "That's the problem with this world. Everybody wants a magical solution to their problems but nobody wants to believe in magic." Said by the Mad Hatter, one of the few characters in the first season who knows that magic and fairytales are real, he's complaining about people who want something that's as quick and easy as magic, but somehow fits into their (supposedly) rational world-view. They want things neat and tidy. He was talking to a character who wanted to believe that there was a way to fix her situation without accepting the reality of other worlds and much bigger things than she had previously contemplated, things that could shatter her world and thrust her into a much bigger drama.
But it could just as easily be applied to technocrats. They want to believe that with properly-designed studies they will find the tricks that nobody else found before, and use those tricks to solve the problems of education. They are forgetting a few inconvenient facts. For starters, scientific discovery is never as clean and linear as the technocrats want to believe. You don't simply follow a best practice for research and thereby identify a best practice for teaching or social work or whatever. True insights, let along truly new insights, are dearly bought. They come at unexpected moments and from unexpected directions. New knowledge is the most expensive thing imaginable. There's a reason why thousands of years ago the Egyptian scribe Khekheperre-Sonbu, living in a time when so many of today's great innovations were unknown, lamented the difficulty of developing new and meaningful ideas.
Also, they want to think that success is just a matter of telling people to do the right things and then watching it happen. Just today I was talking to a couple people who found in their research that teaching assistants who were sent to a seminar in which they were told about the (supposedly) best practices for teaching would start out following those practices but would then deviate. Their belief was that they need to find some way to persuade or train the teaching assistants to stick with the best practices. Well, first, let me note that maybe some of these practices are only best for particular people in particular settings, and the teaching assistants might not be the right audience for these practices.
But second, getting people to do the right thing is more complicated than telling them to do the right thing. Perhaps the best way to instill a practice is not with a seminar that meets for a couple hours once per week over a period of 10 weeks, but rather to provide a role model. Perhaps they need to take a class from an inspiring teacher who uses the (supposedly) newer and better practices. Perhaps that is a better way to instill something into the brain. If you want people to believe down to their very bones that a particular method of teaching physics (or whatever subject) is better than any other method, perhaps they need to actually experience the method itself, and actually learn and appreciate physics via that method, and experience enough growth and insight into physics that they will emulate that method as much subconsciously as consciously. Perhaps role models and meaningful experiences matter more than simply hearing and believing best practices.
Think about how much we emulate our parents, and ask yourselves this: Do you do what your parents told you to do, or do you do what your parents did? You probably got some good habits from them, but did you get any bad habits from them? And did they tell you to adopt those habits, or did your mothers tell their children not to do what they had done? (To quote a great song.) I'm pretty sure that your parents discouraged you from emulating their worst habits, but that probably didn't stop you from emulating them (probably without consciously intending to do so). We know from more than a century of research on k-12 education that parents and family background predict educational outcomes better than any practice on the part of teachers. Why? Because people spend far more time around parents than around teachers, and (usually) get more attention from parents than teachers. (If they get more attention from teachers than parents then there's an entirely different set of problems here.) Likewise, people spend far more time with teachers trying to teach them subject matter than teachers trying to explicitly teach them about teaching. Perhaps the best way to shape the next generation of teachers is to teach subject matter as well as possible and inspire people to do what you did, not to spend 2 hours per week over 10 weeks telling people about best practices.
Of course, teaching the subject matter as well as possible is slower-impact than a special seminar, even if it is higher-impact. And we want the fast solution in our society. We want to believe in the magic without accepting that when the magic actually does happen it happens in a much bigger context. So I'm quite confident that when Bill Gates has left the stage Mark Zuckerberg or some similar figure will attempt to implement an education reform agenda, convinced that This Time It's Different. All of this has happened before and will happen again.
Sunday, May 22, 2016
All of this has been shilled before and will be shilled again
I'm looking at a bunch of things I read a few years ago. Here's a fascinating infographic from the Brookings Institution:
I don't have any problem with the basic concept of parsing out different segments of the STEM job market,m and I certainly do NOT object to people who note that not all STEM jobs need a 4-year degree. At the same time, I have this nasty feeling that definition creep can easily lead to even more inflated estimates of our country's alleged "need" for STEM graduates. You start with a broad definition, one that isn't just about 4-year grads, and get a large number of jobs. Then you point to the large number while arguing for the importance of a particular segment, without noting any of the crucial context or caveats.
I don't have any problem with the basic concept of parsing out different segments of the STEM job market,m and I certainly do NOT object to people who note that not all STEM jobs need a 4-year degree. At the same time, I have this nasty feeling that definition creep can easily lead to even more inflated estimates of our country's alleged "need" for STEM graduates. You start with a broad definition, one that isn't just about 4-year grads, and get a large number of jobs. Then you point to the large number while arguing for the importance of a particular segment, without noting any of the crucial context or caveats.
All of this has been hyped before and will be hyped again
This is a fascinating survey of hype over educational technology. Naturally, though, I must remind the reader that this time it's different!
Subscribe to:
Posts (Atom)
