Science vs. Engineering, "This one study" vs. Practice

Any reader of this blog (assuming that I actually have readers...) can tell that I'm a curmudgeon who is skeptical of reports of Great New Ways Of Teaching.  My default is that teaching is largely a work of timeless issues and problems, and there are no easy substitutes for lots of time and attention and study and patient attempts to solve a problem many different ways, to understand something from many different perspectives.  I believe that the acquisition of skills and knowledge and insights is difficult, time-consuming, and painful, while also having sweet rewards and attractions that keep us coming back for more.  The difficulty of finding genuinely new ideas, genuine innovations, is not a new problem.

This tends to get me into a lot of arguments at lunch with colleagues.  One genuinely good thing about the people whom I tend to argue with is that they are optimistic and open-minded, and since I believe in balance I believe that it is important to have their perspective around.  I'm right and they're wrong and we thereby get balance :)  More seriously, my reason for dissenting from their optimism and open-mindedness, even while understanding its virtues, is that I think many of our disagreements ultimately come down to the difference between science and engineering.  (Bear with me for a few paragraphs before I get back to the science vs. engineering point.)

A typical lunch argument will involve somebody enthusiastically noting some recent study whose results they hope to incorporate into their own practice of teaching or mentoring in the near future (though we have also argued over studies of things like diets and lifestyle, perennial topics of news reports that begin with "According to a new study...").  I am generally skeptical that going from a small study to large adoption will yield the promised improvements.  Part Most of my skepticism is possibly obviously rooted in my own personality and prejudices, but there's more to it.  Studies of new techniques for solving some problem in human interaction are almost always either small-scale experiments undertaken by teams putting a lot of effort attention into it (notoriously vulnerable to the Hawthorne Effect), or else studies of cohorts that are larger but composed primarily of early adopters, who will tend to adopt things differently than the wider population.  When you are talking about teaching, where the human interaction is everything, the difference between an early adopter and a more cautious person is crucial to implementation.

One famous example from physics is the 1999 Hake Study (journal link) (free .pdf) which compared 62 different introductory physics courses and found that the more interactive courses tended to show better improvements on the Force Concepts Inventory (FCI).  That basic finding has stood the test of time, and I will note that I incorporate many interactive elements into many of my courses.  However, the study also found gains as high as 60% or better in some interactive courses (on a scale that is described in the study; read it for more details).  My understanding from talking to people in the field of physics education research is that those gains are not terribly common today, and anything above 40% is really good and even above 30% is decent.  My hypothesis for the difference is that if you were using interactive teaching techniques in the 1990's you were an enthusiastic early adopter, and maybe even willing to plow ahead in the face of resistance.  If you are using interactive teaching techniques today you are following peer pressure, doing what is expected of you (in many but not all institutions) and not necessarily an enthusiast.  If the human element matters in the classroom, then you either restrict your hiring to early adopter types (assuming you can find enough of them) or you look for principles and practices that have a more essential, timeless element to them, fundamentals that will work even if the instructor is not possessed of an early adopter mindset.

What does this have to do with science vs. engineering?  The purpose of a good scientific study is to generate knowledge and understanding.  The purpose of engineering is to translate ideas into practice.  If I went to my lunch group and said "I just read this one study and the researchers made a photovoltaic with [insert great performance numbers here]!  Why don't companies just make that and sell it?" people would look at me funny.  If I went to my colleagues and said "I just read this one study and the researchers successfully treated tumors in preliminary trials, so why don't all doctors just do that?" they would wonder if I have a brain tumor.  The road from a scientific study showing proof of concept to something that can be adopted and work in widespread practice is a long one.  In the hypotheticals that I outlined, an engineer who said "Well, it's just a problem of consumer attitudes, we don't need to examine whether this [product, practice, drug, whatever] is robust" would quickly find himself at the unemployment office.  Making something robust is the essence of engineering.  However, if you look at This One Study and wonder if it will scale well outside of special environments or early adopter groups or whatever, people say that you have the wrong attitude.

I have a deep respect for social science.  I respect it enough to understand that the road from genuine scholarly insight to human practice in the field can be just as long as the road from laboratory physics innovation to product in your home.