Science vs. Engineering, "This one study" vs. Practice

Any reader of this blog (assuming that I actually have readers...) can tell that I'm a curmudgeon who is skeptical of reports of Great New Ways Of Teaching.  My default is that teaching is largely a work of timeless issues and problems, and there are no easy substitutes for lots of time and attention and study and patient attempts to solve a problem many different ways, to understand something from many different perspectives.  I believe that the acquisition of skills and knowledge and insights is difficult, time-consuming, and painful, while also having sweet rewards and attractions that keep us coming back for more.  The difficulty of finding genuinely new ideas, genuine innovations, is not a new problem.

This tends to get me into a lot of arguments at lunch with colleagues.  One genuinely good thing about the people whom I tend to argue with is that they are optimistic and open-minded, and since I believe in balance I believe that it is important to have their perspective around.  I'm right and they're wrong and we thereby get balance :)  More seriously, my reason for dissenting from their optimism and open-mindedness, even while understanding its virtues, is that I think many of our disagreements ultimately come down to the difference between science and engineering.  (Bear with me for a few paragraphs before I get back to the science vs. engineering point.)

A typical lunch argument will involve somebody enthusiastically noting some recent study whose results they hope to incorporate into their own practice of teaching or mentoring in the near future (though we have also argued over studies of things like diets and lifestyle, perennial topics of news reports that begin with "According to a new study...").  I am generally skeptical that going from a small study to large adoption will yield the promised improvements.  Part Most of my skepticism is possibly obviously rooted in my own personality and prejudices, but there's more to it.  Studies of new techniques for solving some problem in human interaction are almost always either small-scale experiments undertaken by teams putting a lot of effort attention into it (notoriously vulnerable to the Hawthorne Effect), or else studies of cohorts that are larger but composed primarily of early adopters, who will tend to adopt things differently than the wider population.  When you are talking about teaching, where the human interaction is everything, the difference between an early adopter and a more cautious person is crucial to implementation.

One famous example from physics is the 1999 Hake Study (journal link) (free .pdf) which compared 62 different introductory physics courses and found that the more interactive courses tended to show better improvements on the Force Concepts Inventory (FCI).  That basic finding has stood the test of time, and I will note that I incorporate many interactive elements into many of my courses.  However, the study also found gains as high as 60% or better in some interactive courses (on a scale that is described in the study; read it for more details).  My understanding from talking to people in the field of physics education research is that those gains are not terribly common today, and anything above 40% is really good and even above 30% is decent.  My hypothesis for the difference is that if you were using interactive teaching techniques in the 1990's you were an enthusiastic early adopter, and maybe even willing to plow ahead in the face of resistance.  If you are using interactive teaching techniques today you are following peer pressure, doing what is expected of you (in many but not all institutions) and not necessarily an enthusiast.  If the human element matters in the classroom, then you either restrict your hiring to early adopter types (assuming you can find enough of them) or you look for principles and practices that have a more essential, timeless element to them, fundamentals that will work even if the instructor is not possessed of an early adopter mindset.

What does this have to do with science vs. engineering?  The purpose of a good scientific study is to generate knowledge and understanding.  The purpose of engineering is to translate ideas into practice.  If I went to my lunch group and said "I just read this one study and the researchers made a photovoltaic with [insert great performance numbers here]!  Why don't companies just make that and sell it?" people would look at me funny.  If I went to my colleagues and said "I just read this one study and the researchers successfully treated tumors in preliminary trials, so why don't all doctors just do that?" they would wonder if I have a brain tumor.  The road from a scientific study showing proof of concept to something that can be adopted and work in widespread practice is a long one.  In the hypotheticals that I outlined, an engineer who said "Well, it's just a problem of consumer attitudes, we don't need to examine whether this [product, practice, drug, whatever] is robust" would quickly find himself at the unemployment office.  Making something robust is the essence of engineering.  However, if you look at This One Study and wonder if it will scale well outside of special environments or early adopter groups or whatever, people say that you have the wrong attitude.

I have a deep respect for social science.  I respect it enough to understand that the road from genuine scholarly insight to human practice in the field can be just as long as the road from laboratory physics innovation to product in your home.

Sticking point

I am currently reading Make It Stick: The Science of Successful Learning, by Peter Brown, Henry Roediger, and Mark McDaniel. It's a book that, on one level, I ought to dislike: Heavy use of anecdotes (as illustration rather than an attempt at proof, I hasten to add), the mood that shows through is often more optimistic than critical, and it says nice things about Eric Mazur.  It's a light read, one that I often find myself skimming rather than deeply digesting.  Despite that, I like it.  Not as much as I like some of the other books that I've blogged here, and I won't have as much to say about it as some of the other books, but I nonetheless consider it a welcome addition to the genre.  Why?

First, whatever flaws it has in common with so many other books in its genre, it has them in far lesser degree.  It does not preach nearly as heavily.  It is optimistic but not breathless.  It uses anecdotes to illustrate but does not shy away from citing data and studies.  When it cites studies, it doesn't just cite the handful of famous studies that every good, decent academic has heard of if they've ever attended even one pedagogy seminar.  It is an easy read (at least for one accustomed to the conventions of this genre) but not a fluffy read.  I see a powerful case for recommending this book to brand new faculty* so that they can be socialized into what they will soon inevitably encounter in interactions with well-meaning (and sometimes not-so-well-meaning) faculty and presenters, but get it in a far better-reasoned form, from authors who (as far as I can tell) aren't selling anything besides this one book that you already bought.

However, Make It Stick is more than just "one of the good ones." There are key ways in which this book differs from the genre of books about teaching (or, at least, the higher ed sub-genre). It's a book that is adamant about the importance of frequent testing, frequent practice, and frequent (well-constructed) assignments. (Indeed, I am reading it specifically because one of the authors was recently interviewed about the importance of tests in The Chronicle.) The authors' strong endorsement of constant practice and testing is, admittedly, consistent with the embrace of popular things like clickers (though that's not necessarily bad; I use clickers in some classes), and also with some of the surface features of "flipped classes" (the topic of an excellent post today by my friend Xykademiqz), but there's none of the "Now, now, you have to understand, you're doing it wrong and need to be more progressive..." tone that usually comes from flipped class discussions and drives me and Xykademiqz and others up the wall.  (Indeed, the term "flipped class" appears nowhere in the index.)  I can think of at least one person who has gone on the public record to argue that progressive educational methods eliminate the need for reading or other substantive work or study outside of class time, and it is only my thin (though fading) residue of semi-civility that keep me from linking to this person's published statements on the matter.  The authors of this book would not be nodding along with that person.

Interestingly, Make It Stick is also a book that cautiously declines to join in casting stones at standardized tests as a category, an inclination that I equivocally share, and an inclination so outside the norms of polite academic tropes that seeing it in a book automatically warms my contrarian heart.  

I am about 2/3 of the way through this book, and I intend to finish it.  It is a worthy addition to the conversation.  I would love to live in a world where the most populated portion of the spectrum of pedagogical discourse stretched from historically literate curmudgeons like Hofstadter on one side to psychologists like Brown, Roediger, and McDaniel on the other side.

*This mention of recommended reading for new faculty brings something to mind:  In 2006, before I was about to begin my second adjunct gig, I read Ken Bain's What the Best College Teachers Do.  I don't know how I'd react to it now, after 9 years of encounters with edu-fads and their promoters, but at the time it struck me as non-preachy, non-ideological, and consistent with the instincts of a concerned teacher.  Perhaps Bain's book, Make it Stick, and Hofstadter's Anti-Intellectualism in American Life would make a nice trio of gifts for new faculty...

Testing 1-2-3

I haven't forgotten this blog.  I've been busy, but I haven't forgotten.  I will start blogging another book soon.  In the meantime, a few notes on an article in the Chronicle: The basic point of the article is that students learn more if they are tested frequently.  Given that constant practice and review and effort seem to be essential for success in any endeavor, I find this unsurprising and uncontroversial.  Although I don't give a large number of tests in my classes, I do give assignments before almost every class, so that students are always reviewing something or thinking about something or preparing something, and so that I always know how they are doing.  I think this practice is more or less in keeping with the point in the article.

In the physics community, one doesn't see a lot of attention to frequent tests in presentations on pedagogy.  It is, admittedly, generally noted that frequent practice is good, and regular "reading quizzes" are often specifically named as a good practice.  However, frequent testing doesn't get a lot of attention.  To the extent that something similar gets attention, it's generally called "Just In Time Teaching", which has the virtues of calling to mind a business buzzword (always important if your goal is to sell instructional materials and consulting services) and lending itself to an easy acronym (JITT).  The basic idea of JITT is to give online quizzes before class, so that you know coming into class what the students are getting and what they are struggling with, and can adjust accordingly.  It's an inoffensive, sensible concept.  It is not, however, as trendy as flipped classes, peer instruction (which is promoted by a famous guy who has his picture on the cover of a book), etc.

My hypothesis is that frequent testing and practice, while recognized as sound practices, get less discussion and attention for three reasons:
1) You can't make much money off of telling professors to give more tests and quizzes and homework assignments.  There's no manual to sell, no workshop to give.  We already write tests and quizzes and assignments, and we can write more.

2) Giving more tests and quizzes is more work for us (work that many but not all of us are willing to do) and more work for students (who will complain).  Admittedly, some of the more conscientious students will actually thrive with more frequent feedback and critique, but those who fall behind will stay behind.  The results will probably be more bifurcated, and sharply bifurcated results make it harder to justify the "Gentleman's C's" upon which the continuance of the system depends.

3) More tests and assignments may be a sound practice, but they don't inspire fervent discussion because they don't hit the same cultural resonances as questions of interactions and the role of the expert.  We can get passionate in debates about the role of the expert, but we don't have the same passionate views on frequent quizzes and homework assignments.  Admittedly, high stakes standardized tests arouse passions, but those tests are administered by outsiders, come with financial stakes, and visibly intersect with disparities that are rooted in thorny social problems .  That's very different from giving a weekly quiz in your freshman course. 

And thus we come back to some of the major themes of this blog.

Zhao, Chapters 5-7

In Chapters 5-7 of Zhao's book, I feel like the worst parts of China's education system are what America's would be if every bad part were turned up to 11.  Chapter 5 talks about the hitting of arbitrary targets:  Whatever the Emperor wants, the Emperor gets.  If it is decreed that a big wall will be built, then damn the expenses, the wall will be built.  If Chairman Mao wants to increase steel production, then steel production will be increased, even if it is to the detriment of every other activity in society.  If professionals (academic and otherwise) need to publish in order to get promoted, and the only thing that matters is the number of publications (because bureaucracies need numerical metrics) then fake journals will be utilized.  If k-12 schools are going to be rewarded for the number of patents filed by students (Zhao is apparently not joking) then k-12 students will file patents of zero quality, and if the system is going to reward this then the  patent office will be obliged to accept and approve those patents.

I feel much the same way about the paperwork demands of the modern American higher ed bureaucracy.  It has been decreed that reports on outcomes assessment are to be written, so those reports are written.  However, thoughtful, meaningful data collection and analysis are non-trivial activities, and the System has no ability to fund the massive efforts that would be required to produce thoughtful evaluation of anything other than the most mass-produced course delivery.  Mass production sounds good to some technocrats, but "Innovation! And! Transformation!" also sound good to them, so the system continues to tolerate the intellectual diversity that is anathema to standardized, mass-produced assessment.  As a result, the system accepts meaningless reports to show that the requisite numbers of reports were filed.*  Moreover, the only thing that The System values even more than the production of reports is ensuring that no boats are rocked, and a truly meaningful look under the hood might tell us...well, it might tell us that the phrase "Gentleman's C" refers to a problem that is timeless and universal rather than old, musty, and covered in ivy.

In chapter 7, Zhao notes that China's rulers are not and have not been oblivious to the problems of exclusive reliance on standardized testing.  The Imperial Exams were abolished a bit more than a century ago, but the competitive test prep habits inspired by those exams have not easily faded.  When the universities give weight to measures of creativity (an oxymoron if ever there was one) parents push this children to excel at those measures and bribe the evaluators.  When the universities give weight to patents (!) the children produce the requisite numbers of junk patents.  If they reward performance in artistic or athletic talent competitions, the parents prepare their children accordingly.

Some  of this is surely a consequence of 1300 years of test prep habits not fading easily.  Cultures do not change overnight, as evidenced by Marxism's utter failure to change Chinese culture. (I hasten to note that the Chinese people's refusal to replace their traditional social structures with Marxism is a very, very good thing!)  However, Chapter 6 offers another point for why these hyper-competitive academic habits have not faded easily:  In both the past and the present, the highest-status jobs in China have been and are government jobs with academic requirements, not private sector jobs.  When there are few paths to advancement, people will do anything and everything that it takes to get on one of those paths.

In the US, we most definitely do NOT have a problem of assigning low status to private sector jobs.  However, we may be seeing a narrowing and bifurcation of career paths.  In the alleged "service economy" there will be the people paid well for jobs at desks and the people paid poorly for everything else.  There are problems with the economic sustainability of this model (e.g. what, exactly, will we produce for sale to the rest of the world?) but if we leave that aside for now, and just focus on the social and educational elements, a bifurcation of opportunity will raise the stakes of elite education, and with those rising stakes we will see more intense competition, and more cutting of corners.

I want the reader to consider the hypothesis that the health of an education system relies on the economy more than the health of an economy relies on the education system.  I do not simply mean that higher GDP means more money to inject into the education system; you can inject cash into good or bad schools and get good or bad results.   Rather, I mean that a robust and sustainable economy with a breadth of desirable paths and opportunities will be conducive to a more robust and sustainable educational  system, with more intellectual breadth and a wider variety of approaches and paths for a wider variety of students.  On the other hand, an economy with bifurcated opportunities will see intense competition for the handful of pathways to the better opportunities.  Competition can be a very good thing, but intense competition for the wrong sorts of things does not lead to better learning or better innovation.

*Let me reassure you that I am, needless to say, merely reporting unfortunate tales related to me by colleagues at other, less conscientious, institutions.  My own institution is, of course, an exception, producing exemplary assessment reports of the highest caliber and most thoughtful analysis.

First thoughts on "Who's Afraid of the Big Bad Dragon"

The first couple chapters set the stage by talking about the Western infatuation with Chinese education and the history of the Imperial exams.  Apparently there is nothing new about Westerners proclaiming that we need to emulate Chinese education.  Jesuit missionaries were impressed by the effort that went into the Imperial Exam system centuries ago.  Zhao, born and raised in China before pursuing an academic career in the US, makes a historical case that our infatuation with China is misplaced, noting (among other things) that the Industrial Revolution passed China by because of an authoritarian education system that emphasized Confucian teachings over science and technology.  This resulted in China's humiliations at the hands of foreign powers in the 19th century.

The basic historical survey seems consistent with everything else I've seen, as far as the facts go, and well-argued as far as the analysis goes.  Nonetheless, I want to quibble with the author on two points where I think he's not pushing as far as he could with the implications of certain assumptions.

First, in the introduction, Zhao argues that we are mistaken to emulate China's exultation of effort over all else.  He says that it's an unrealistic expectation that carries the implication that if the poor fail it's their own fault for being lazy.  Certainly that is one way to apply such an attitude, and we often see the poor in the US castigated for alleged laziness.  However, I'm not convinced that the problem is our attitude towards effort, but rather a combination of (1) our attitude towards the poor and (2) elevating a theory of education to a theory of everything.

In the trenches of higher ed, the notion that failure comes from lack of effort has a brighter, shinier twin that says that anyone can succeed if they try.  This brighter, shinier twin eschews many traditional metrics of success and ability in favor of concepts like "growth mindset" and "grit" and argues for democratizing access to advanced study and elite institutions.  Maybe you share their perspective, or maybe you don't.  (For my own part, I believe that success is mostly effort but not solely effort, so I'm in a mushy middle.)  Regardless of your own view, the attitude that equates success with effort is one that can be harnessed in the service of progressive causes just as easily as it can be used to attack the poor.  Indeed, in the modern American context, those who argue that academic success has a strong connection to innate abilities are seen as playing on a slippery slope that equates academic failure with biological traits, traits that the less savory in our society then argue are more common in certain groups.

My point is that if you have a theory of what causes academic success or failure, you can, if you wish, use that theory to argue that the poor are lazy, or genetically inferior, or less virtuous, or whatever other trait you correlate with failure.  Likewise, you can use that theory to propose ways of uplifting the poor and disadvantaged, or at least ways of making their situation less bad than it currently is.  Some of those proposals may be more patronizing than others, but the point is that once  you have a theory of success and a theory of failure, how you apply it to the poor and disadvantaged depends on additional factors in how you view the poor and disadvantaged.  The problem is not your theory of education, it's your theory of the poor.

Of course, elevating a theory of education to a social and economic "theory of everything" is not a new thing in American society, as Hofstadter noted.  Our colonial predecessors wanted to use education to produce better Christians, the people of the Founding era saw education as the key to republican* virtue, social reformers from the 20th century to today have viewed education as the solution to social and economic inequality, and business and the defense establishment see it as the key to technological superiority.  Education solves all ills, we believe.  How you apply that notion to the poor depends on your attitude toward them, not your preferred educational theory.

Also, Zhao argues that the Chinese education system kept China from participating in the Industrial Revolution.  To a large extent that's true--certainly they did little to train a class of scientists and engineers.  However, all great technological advances ultimately require an enterprising business class with the flexibility, ambition, and fearless, stupid, naive over-confidence to go out and risk everything on that.  Imperial China didn't have an MIT or even a Tsinghua, but Cambridge and Oxford were scientific backwaters during the Industrial Revolution.  The UK nonetheless enjoyed the fruits of the Industrial Revolution because their society made room for entrepreneurship.  Perhaps China's real weakness was not its schools but rather a rigid social structure that allowed less room for technological and economic innovation?  Indeed, to this day, while America has some fine engineering schools that produce great entrepreneurs, most American entrepreneurship (at least outside the high-tech realm) does not come from universities.  In fact, most entrepreneurship cannot and should not come from the academic world.  The function of most academic disciplines is a conservative one, and the key to a healthy society is to maintain that important conservative function without building all of society around it.

*In the sense of a Republic, not a political party.

Next book: Who's Afraid of the Big Bad Dragon?

My most stressful recent deadline has passed.  I have many more to go, but the worst is over.  In the near future I will start reading and blogging about Who's Afraid of the Big Bad Dragon: Why China has the best (and worst) education system in the world.  It should be interesting to examine Chinese education a mere century after the end of a system that lasted 1300 years.

Final Thoughts on Miyazaki

The strangest thing that jumped out at me in the second half of the book is that young men who passed the imperial exams felt more loyalty to the people who passed them than the people who taught them.  The mindset seems to be that people who passed them recognized their merit* and gave them the ticket to success, whereas their teachers merely performed their duty in their job, as part of a transaction.

On the other hand, the concept of loyalty to the exam system, and identification with it, that is something that I get.  These men felt loyal to the academic system that they came up in, and one result is a back-and-forth political battle between men who gained influence through the exam system and bureaucracy, and men who gained influence by other paths. The winners of the high-stakes exam system would struggle amongst themselves, but unit against external threats. We see similar things today, and one could argue that my defense of sages on stages and skepticism of educational reform is in part a psychological response to perceived attacks on the legitimacy of the academic system that I have succeeded in.  Of course, I have tried to articulate my stance in other terms, and defend it by appeals to more factors than just "Well, it worked for me."  Nonetheless, I acknowledge that a reasonable person might make that inference.

One important thing to note about China's exam system is that it emerged from the efforts of Sui and Tang emperors to fight back against the aristocracy.  They wanted public servants drawn more broadly from the population, with loyalty to the national good rather than their own families, and they wanted the best talents that they could find.  It was a noble idea, and it had some noble features, being nominally open to almost anyone irrespective of birth (with a few narrow exceptions for men whose fathers or grandfathers had been involved in certain disreputable lines of business, e.g. brothels).  Also, they made a genuine and energetic effort to keep the system honest, working hard to fight cheating rather than accept it as something to be expected and forgiven in the sons of privilege, and the public outcries against cheating scandals (centuries before modern electronic media made public outcry easy to arouse) testify to the wider population's perception that the system could be a valid one if conducted fairly.

Of course, the emperors failed utterly in their effort to eradicate elite families from the social and political landscape of China.  There was turnover when the exam system first started, with some new elite families rising and old elites falling.  Nonetheless, the winners were able to pass on advantages to their children, and in time the system became as stratified as any openly aristocratic system.  Perhaps the most important lesson is that you cannot select your way out of inequality.  Other authors that I've critiqued on this blog have argued that you can't teach your way out of inequality, but I would go farther and say that you cannot select your way out of it either.  If you want to advance people from disadvantaged backgrounds you have to explicitly select them and promote them.  You cannot simply pick some other criterion, nominally neutral or perhaps even favorable, and trust that competition will not emerge to reinforce the advantages of having advantages.

I mean, even a Communist revolution failed to eradicate class distinctions in a nominally meritocratic society!  How could we expect better interview questions or admissions essays to accomplish that goal in a capitalist society?  If we want to address disadvantage we should look squarely at disadvantage and explicitly make allowances, not look at education as a proxy for whatever it is that we actually want (or claim to want).

*Yes, say what you will about merit and meritocracy, but that's how they saw it, and I'm trying to describe the mindset.