Latest Article: "Studies Have Shown" Is Not Enough

 My latest opinion piece has been published in Discourse Magazine. It's titled "'Studies Have Shown' Is Not Enough." It's on the problem of people wanting to jump on edufad bandwagons as soon as a few studies supposedly show something.

Blogging the countervailing currents

I'm going to blog two social science articles from a high-profile journal.  I should state up-front that such articles often contain results that are hard to replicate, so take them with a grain of salt.  However, these two articles deviate from the current zeitgeist, and hence might have faced slightly more scrutiny in peer review than other articles.  That doesn't guarantee that the results will stand up to replication efforts, but it does make them worth examining.

First, this is a nice study showing that placebo effects can be substantial in "cognitive training" activities.  It's not my area of expertise, but they clearly put a lot of effort into constructing good control and placebo groups, more care than I see in a lot of educational studies.

Second, here's a large-scale study from Denmark, in which the number of instructional hours devoted to reading was increased in thousands of elementary school classrooms, but in half of the classrooms the teachers were told to do whatever they wanted while in the other half they were given a very strict reading curriculum to adhere to in those extra hours.  The teachers with flexibility had statistically significant improvements in their classrooms.  The teachers with the prescribed curriculum had gains that were larger than zero but not as large as those of the teachers given flexibility.  The interpretation is slightly tricky because the error bars on the second gain overlap both zero and the teachers with flexibility.  Still, what we can say confidently is that there's no evidence that giving teachers some amount of flexibility is bad in this context.  The authors interpret it as evidence that teachers respond with instruction tailored to the needs of their particular class.  Not a shining moment for technocracy.

Musing on the Hawthorne Effect

The Hawthorne Effect is simultaneously the greatest indictment of efforts toward systematic and sweeping change and the greatest tribute to human nature.  It is an indictment of systematic efforts because it tells us that most things will not work as well as the pilot projects that had passionate people involved in something that they truly believed would work.  It is a tribute to human nature because it tells us that if people think they're doing something that matters they'll do better.

The progressive teaching personality

I've said a lot about the cultural roots of progressive pedagogy, and the way it intertwines with views on expert voices.  However, this article makes the interesting point that there are also personality issues that go to extroversion vs. introversion.  There's a certain personality type that just plain likes the idea of group work.  I have seen people who are only truly happy when their students are sitting in lots of small circles talking.  They throw themselves into the midst of this chatter like a pop star crowd-surfing.  You can't tell me that this preference concerning teaching styles is solely about statistical studies showing a significant difference in conceptual learning gains.  Statistically significant learning gains get you more discussion of concepts, but culture and personality get you academics eating this shit up like it's humus and pita chips from Trader Joe's.  And it's definitely true that the warmer, fuzzier stuff has been ascendant in the academy for some time, so I suppose it was inevitable that eventually the science faculty would move towards sitting in groups and smiling and saying "Thank you for sharing that."

(Academic eating tip: Eat your humus with organic baby carrots, not pita chips.  Then you get more vitamins and fewer carbs.  This food tip is subject to change when the fads change.)

However, I will surprise my readers by saying that I don't think this is solely about personality types and massive placebo/Hawthorne effects from doing things that get certain personality types fired up. It is almost certainly true that some people do better than others sitting in groups and doing worksheets. (One needn't subscribe to discredited theories of learning styles to submit that some people might be a bit better at one thing than another.) If there was little/no group work and hippy-dippy conceptual stuff in traditional teaching then introducing some of that will OF COURSE improve results.  Those who were desperately craving some sitting in circles and sharing will finally get some of it, so of course their results will improve.  Meanwhile, those who really do prefer a bit of tradition (you know, heirs to Puritan views of experts and all that) will still get plenty of that from the abundant traditionalists.  In fact, the ones who generally fare better with tradition might benefit from stepping outside their comfort zone a bit.  (There's plenty of research to suggest that stepping outside the comfort zone is probably more beneficial than catering to "learning styles" based on dubious evidence.)  However, once you reach a roughly 50-50 mix, if you continue to move away from tradition (or, perhaps more accurately, get to the point where you are now defining tradition rather than defying tradition) you will reach a point of negative returns:  The ones who would honestly rather just pay attention to an expert will get fed up with this group work nonsense and experience negative marginal returns, while the ones who love to sit in a circle and share will have already gotten enough of it.

At this point it would be easy to say that I'm being silly.  OF COURSE group work is better!  The data shows it!  Well, up to a point.  The Hake study, for instance, covered early adopters and showed gains that have been harder to replicate once "active learning" went beyond early adopters in special, early-adopter-friendly environments and became de rigeur.  I think everyone at this point agrees that a certain amount of active learning is a good component of a class, but the jury is out on mass adoption of more and more and MORE AND MORE group work.

Besides, the article that prompted this post was shared with me by a friend who made an important observation:  In the real world there are indeed meetings, but most meetings are pointless, and most people yearn for nothing more than the meeting ending early so they can get back to the cubicle, put their head down, and actually get some damn work done.  Perhaps there's a lesson in there.  Yes, a bad lecture is bad, but a very carefully-led, well-focused Q&A can be more useful than going through a worksheet with the idiot next to you.

The Journals of Irreproducible Results

A group of psychologists recently attempted to replicate 100 studies published in psychology journals in 2008.  The results are in, and they aren't great.  Roughly a third of results were replicated at a statistically significant level.  Before I unpack some of the implications, I should put in an important qualifier, namely that this does not mean that those studies should not have been published or that the investigators did something wrong!  A study isn't conducted and published in order to provide the final word, but rather to invite the rest of the community to scrutinize the work and attempt to replicate it.  If a conscientiously-conducted and well-designed experiment gives a statistically significant result (positive or negative), that result is fair game for dissemination, discussion, and further scrutiny.  This replication effort is not evidence that science is not being done, but rather evidence that science is being done right.  We need more replication efforts, not fewer studies.

That said, it should give pause to people who point to a study and make prescriptions.  I've written before about the cult of This One Study, and the challenges of actually reproducing so many psychology studies point up the follies of that cult.  Moreover, as challenging as it is to get reproducible results in psychology, many educational research studies that I read do an even worse job on blinding participants and maintaining proper controls,  Some of that is inevitable, given the nature of classroom work, but some of it is a result of people uncaringly playing with too many variables at once.  If the person teaching the class is enthused about trying a new educational method and transmits that enthusiasm, and also varies the topics and assignments, while the person in the "control" group is not doing anything to get fired up and refreshed, you can't treat this as a single-variable experiment.  Moreover, often educational experiments are tapping into deep wells of cultural baggage; will the experiment get the same effect if it is conducted with people from the opposite side of that cultural divide?  Finally, it has been pointed out to me by at least one person trained in statistics and educational psychology that when you have one class section taught by New Method and another class section taught by Old Method, even if you have hundreds of students in each section you still have n=1 in the experimental and control groups, not n=hundreds, if your goal is to study methods rather than students.  "n=1" is a fancy statistical term for "anecdote."  Despite all of this, educational research tends to lead to a lot of teaching and prescription.

Strangely enough (or maybe not strangely at all, depending on your degree of cynicism), I suspect that Right-Thinking People will delightedly cite this finding that psychology experiments are not always reproducible as a way of reminding people that they don't necessarily know what they think they know, but will also continue to cite their favorite studies as rhetorical cudgels.  Consistency conshmistency!

Science vs. Engineering, "This one study" vs. Practice

Any reader of this blog (assuming that I actually have readers...) can tell that I'm a curmudgeon who is skeptical of reports of Great New Ways Of Teaching.  My default is that teaching is largely a work of timeless issues and problems, and there are no easy substitutes for lots of time and attention and study and patient attempts to solve a problem many different ways, to understand something from many different perspectives.  I believe that the acquisition of skills and knowledge and insights is difficult, time-consuming, and painful, while also having sweet rewards and attractions that keep us coming back for more.  The difficulty of finding genuinely new ideas, genuine innovations, is not a new problem.

This tends to get me into a lot of arguments at lunch with colleagues.  One genuinely good thing about the people whom I tend to argue with is that they are optimistic and open-minded, and since I believe in balance I believe that it is important to have their perspective around.  I'm right and they're wrong and we thereby get balance :)  More seriously, my reason for dissenting from their optimism and open-mindedness, even while understanding its virtues, is that I think many of our disagreements ultimately come down to the difference between science and engineering.  (Bear with me for a few paragraphs before I get back to the science vs. engineering point.)

A typical lunch argument will involve somebody enthusiastically noting some recent study whose results they hope to incorporate into their own practice of teaching or mentoring in the near future (though we have also argued over studies of things like diets and lifestyle, perennial topics of news reports that begin with "According to a new study...").  I am generally skeptical that going from a small study to large adoption will yield the promised improvements.  Part Most of my skepticism is possibly obviously rooted in my own personality and prejudices, but there's more to it.  Studies of new techniques for solving some problem in human interaction are almost always either small-scale experiments undertaken by teams putting a lot of effort attention into it (notoriously vulnerable to the Hawthorne Effect), or else studies of cohorts that are larger but composed primarily of early adopters, who will tend to adopt things differently than the wider population.  When you are talking about teaching, where the human interaction is everything, the difference between an early adopter and a more cautious person is crucial to implementation.

One famous example from physics is the 1999 Hake Study (journal link) (free .pdf) which compared 62 different introductory physics courses and found that the more interactive courses tended to show better improvements on the Force Concepts Inventory (FCI).  That basic finding has stood the test of time, and I will note that I incorporate many interactive elements into many of my courses.  However, the study also found gains as high as 60% or better in some interactive courses (on a scale that is described in the study; read it for more details).  My understanding from talking to people in the field of physics education research is that those gains are not terribly common today, and anything above 40% is really good and even above 30% is decent.  My hypothesis for the difference is that if you were using interactive teaching techniques in the 1990's you were an enthusiastic early adopter, and maybe even willing to plow ahead in the face of resistance.  If you are using interactive teaching techniques today you are following peer pressure, doing what is expected of you (in many but not all institutions) and not necessarily an enthusiast.  If the human element matters in the classroom, then you either restrict your hiring to early adopter types (assuming you can find enough of them) or you look for principles and practices that have a more essential, timeless element to them, fundamentals that will work even if the instructor is not possessed of an early adopter mindset.

What does this have to do with science vs. engineering?  The purpose of a good scientific study is to generate knowledge and understanding.  The purpose of engineering is to translate ideas into practice.  If I went to my lunch group and said "I just read this one study and the researchers made a photovoltaic with [insert great performance numbers here]!  Why don't companies just make that and sell it?" people would look at me funny.  If I went to my colleagues and said "I just read this one study and the researchers successfully treated tumors in preliminary trials, so why don't all doctors just do that?" they would wonder if I have a brain tumor.  The road from a scientific study showing proof of concept to something that can be adopted and work in widespread practice is a long one.  In the hypotheticals that I outlined, an engineer who said "Well, it's just a problem of consumer attitudes, we don't need to examine whether this [product, practice, drug, whatever] is robust" would quickly find himself at the unemployment office.  Making something robust is the essence of engineering.  However, if you look at This One Study and wonder if it will scale well outside of special environments or early adopter groups or whatever, people say that you have the wrong attitude.

I have a deep respect for social science.  I respect it enough to understand that the road from genuine scholarly insight to human practice in the field can be just as long as the road from laboratory physics innovation to product in your home.

Reflections on Hofstadter: Why should cultural subtext matter in educational debates?

First, let me make it clear that  the question in the post title does not refer to cultural contexts involving student-teacher interactions.  If a teacher raised in an Italian neighborhood in New York City finds themselves teaching in the rural Midwest, they should probably keep in mind that some of the people might be a bit more reserved than people back home.  Rather, I'm referring to the cultural context of the educators, officials, and members of the public debating over the purposes and methods of mass education.

The modern education reformers mostly make a genuine and sometimes very successful effort to be scientific in their approach.  That is to be applauded.  If they have reproducible data from carefully-controlled experiments, one might argue that it doesn't matter if they happen to be cultural heirs of Great Awakening preachers rather than Puritan essayists.  I nonetheless maintain that it does matter for 3 reasons:

1) Truly double-blinded, single-variable experiments are often impossible in social science.  The person who walks out of the workshop speaking of how "transformed" they are is not going to simply adopt a new teaching technique.  They are going to adopt it with enthusiasm and passion and come into class with a degree of energy and preparation that they might not otherwise project if they were doing the same-old-same-old.  The effect of the experimenter's interests on the experimental subjects is of old vintage in social science, being named the Hawthorne Effect after an experiment on work conditions at a factory in the 1920's.  It is probably similar in mechanism to the placebo effect in medicine.

I maintain that there are cultural factors intimate that determine which interventions will resonate with the instructor, cultural factors that are entwined with American views on democracy, equality, the role of the expert, the practical value of knowledge, and even religion.  You cannot understand the enthusiasm that American educators will have for a teaching method, and the way that they will transmit that enthusiasm to their American students, unless you understand American cultural history.

So far I've argued that most educational experiments are two-variable experiments (method and enthusiasm) rather than 1-variable experiments (method only).  But I would go farther and argue that there is often a third variable, one that is also laden with cultural baggage:

2) Adoption of a new instructional method is often accompanied by a change in what is taught and tested.  That is not a priori bad from a normative standpoint, but it certainly complicates interpretation of experimental data.  In physics, for instance, there are different schools of thought on the relative importance of "conceptual" understanding and more traditional emphases that might, for lack of a better term*, be termed "mathematical", "calculation", or "problem-solving."  The precise nature of the difference is less important than the fact  that if two classes emphasize different things AND use different instructional methods, and then both groups of students take the same test, a test that happens to better align with what was emphasized in one class but not the other, you cannot be sure what to attribute differences in performance to.

Now, the Puritans and the Awakeners were certainly not debating electromagnetic theory (excepting Puritan descendant Benjamin Franklin, of course), but the different approaches in physics do roughly correspond to a difference in how mathematics is used or emphasized, and in American culture mathematics is a subject that is often perceived as carrying heavy intellectual baggage. Americans believe that you have to be a real Einstein to "get" math. It's a subject for "smart kids" (and is often lamentably stereotyped as being for smart boys but not smart girls).  If a pedagogical divide happens to include a divide over the aspects of the subject that carry the heaviest intellectual and cultural baggage, it is fair to examine the ways in which these pedagogical schools might emulate different American traditions concerning experts, intellectualism, etc.

3) However scientific the founders of a movement might be in their scholarly approach, however empirically-driven their proposals might be, no movement can scale beyond small cohorts of enthusiasts without escaping the control of its founders and getting caught up in broader cultural currents.  If reformed pedagogy ultimately enjoys mass adoption it will become its own thing, subject to the biases and quirks of adopters, the pecuniary interests of publishers, the agendas of school administrators and public officials, and the wider culture of the Academy, to name just a few among a multitude of factors.  Indeed, Hofstadter's penultimate chapter notes that Dewey, like many a Great Man before and after him, ended his career shaking his head at the excesses of his followers, people who tried to institutionalize the Revolution.

Finally, one might note that I keep speaking of American culture, even though many college professors are immigrants and many of our students are immigrants or first generation Americans, and mostly do not have any ancestors who were either Puritans or participants in the Great Awakening.  My first answer is that there is nobody more American than an immigrant. Whenever I examine cultural strains that have been repeated again and again in American history I am examining events that included the participation of innumerable immigrants. My second answer is that culture is not transmitted solely by parents.  If America is good at anything, it is good at assimilating the children of immigrants into our culture.  Schools, entertainment, churches, neighborhood kids, employers, and other factors all conspire to take children with parents from every corner of the globe and make them into bog-standard Americans (for good or for ill).

I am thus comfortable in saying that American cultural history is an absolutely crucial background to educational debates, even if we should have a Middle Eastern immigrant professor teaching the theories of German physicists to the children of Latin American immigrants (a common enough occurrence in my department).

Indeed, let me gaze at my own navel: To my knowledge, I have no Puritan roots in any meaningful sense.  What English blood I have is from people who either moved to the South or came to Canada in the 1800's.  To the extent that I identify with any culture other than Midwestern White, I identify with southern Italians because of my very close bond with my grandfather, who was born in the Midwest but was raised in a family and community of southern Italian immigrants.  I attended a Franciscan grade school that was founded primarily by Slovak immigrants.  I attended college and grad school in California, not at a New England school of Puritan derivation. Even a lot of my academic mentors did most of their training at schools not founded in the Puritan tradition.  (Though there is nonetheless a bit of New England in my academic family tree.) Despite that, through my interest in books and science, and my subsequent sojourn in higher education, I internalized many cultural traits of America's intellectual classes, and thus there is a strong dose of Puritanism in the cultural patterns that I have internalized.  Much of this Puritanism comes from my teachers and relatives and friends who also have no direct ties to New England but have likewise absorbed those values through American culture.

*We traditionalists have, to our discredit, been bad at articulating precisely what we value and the ways in which it differs from the approaches termed "conceptual."