An article on ass-essment

 I don't have time for an in-depth review of this article on assessment, but I do enjoy it. I never thought I'd read an article that approvingly quotes both Dewey and Hofstadter. 

I want to quote two things. First:

As Michael Bennett and Jacqueline Brady point out in their 2014 article “A Radical Critique of the Learning Outcomes Assessment Movement,” learning outcomes is only a different term for lesson plans or course content. 

Yep. It's about asking us to restate something we already know (we'd better show up to class with an idea of what we're doing) with a word that they control, because they can always accuse us of not understanding their buzzword well enough.

Second, there are many great points about the pointlessness of "learning goals" but I will quote this one:

Trevor Hussey and Patrick Smith, in their 2008 article “Learning Outcomes,” distinguish between and among learning goals written for a lesson, for a course, and for a program, and the authors point out that the more remote the learning goals are from the classroom itself, the more irrelevant learning outcomes become. Robert Shireman, in his 2016 op-ed “SLO Madness,” amusingly labels student learning outcomes as “gibberish,” capturing Dewey’s point that for a learning goal to be an intellectual one it has to arise from the intelligence and experience of the teacher; otherwise it is meaningless. Similarly, one cannot function as a true scholar or an intellectual while being told what to do. Moreover, the sheer number of classes that need to be certified necessitates that learning goals be written in categories that may or may not be relevant to the material taught. For example, in the general education category at San José State University (SJSU), where I teach, a single learning goal can apply to ten to twenty different disciplines.

This machine-like nature stands in stark contrast to the ass-essment movement's commitment to "innovation." If I'm constantly doing new things then the goals will inevitably be shifting, at least in part. Yes, at the end of the day I still want them to learn optics or mechanics or whatever, but if I try new assignments then the details change, and the details of the assignment matter! The details of the project matter! Two classes can cover the same basic idea in such different ways that all you can really say is "Well, both classes talked about lenses."

The problem of ASSessment

This column at Inside Higher Ed elegantly summarizes the problem with assessment:

The faculty already have a language about teaching and learning. If you ask a math professor what students will learn in calculus, she may point you to the table of contents in the textbook, where dozens of topics are enumerated. If you ask how the class is going, you might hear that “they were fine with the derivative rules, but related rates problems are killing them.” This language is an integral part of teaching. 

The assessment bureaucracy -- those periodic checkboxy reports -- can only be justified if the formal learning outcome statements and their standardized assessments are superior to the native ways faculty know their students. Otherwise we could just ask faculty how the students are doing and use course registrations and grades for data. We could look at the table of contents to find the learning outcomes. 

These two worlds -- the report writing and the lived experience -- coexist, but not easily. While the assessment office depends on the informal channels of faculty knowledge to do meaningful work, in most regions of the country each program requires a formal report. These “cookie-cutter” reports fail miserably at generating new knowledge (something you couldn’t learn by just asking faculty members what they think) because they are based on faith in the special meaningfulness of the learning blurbs.

Exactly.  We already know how to talk to each other, and some of us also know how to talk to the people who hire our students.  What we don't know how to do is talk to people who need boxes checked so that they can justify their phoney baloney jobs.

https://www.youtube.com/watch?v=A2oaTxPufOg

Article on Assessment

I have an article at Inside Higher Ed titled "Some Questions for Assessophiles."  The comments are not entirely favorable, as you might expect.  I have since been pointed to this most interesting blog critiquing assessment, and this most interesting interview with Molly Worthen.

More on technocrats

Between the Devil and the Dragon, page 24:

To make of human affairs a coherent, precise, predictable whole, one must ignore or suppress man as he really is and treat human nature as a mere aspect of nature.  The theoreticians do it by limiting the shaping forces of man's destiny to nonhuman factors: providence, the cosmic spirit, geography, climate, economic or physiochemical factors.  The practical men of power try to eliminate the human variable by inculcating iron discipline or blind faith, by dissolving the unpredictable individual in a compact group, by subjecting the individual's judgment and will to a ceaseless barrage of propaganda, and by sheer coercion.  It is by eliminating man from their equation that the makers of history can predict the future and the writers of history can give a pattern to the past.

This reminds me of what I wrote about assessment several months ago:

OK, so we're up against a trifling force like human nature, but when has that ever stopped a technocrat?

I'm glad I picked up this book.

Shove it up your assessment

I have in my "drafts" folder a long post about why higher ed's ongoing struggle to implement "assessment" and all of the associated language of the managerial classes (e.g. "lists of program objectives", which I think are different from "learning outcomes", and alignment of these things with "mission statements", and other stuff that keeps bureaucrats and kool-aid drinkers happy) is completely useless.  However, I think that I can state the problem much more simply:

None of this is in any way related to what people actually do or how people actually talk about things when they aren't interacting with a very particular class of bureaucrats and consultants!

When a colleague from the private sector is looking for new graduates to hire, they don't ask me if some particular skill or topic is on our list of Program Objectives or whatever.  They just ask me how much experience my students have with that particular skill or topic.  Ditto when a colleague in academia (a world in which we are all under administrative orders to embrace assessment) asks me if I have any good students to recommend for graduate study.  They don't ask me about our program objectives, they just ask me if I have smart, reliable students who have extensive experience with a particular topic or skill.  When I answer their question, I don't consult a list of Student Learning Outcomes.  I know what is covered in the courses, and what students do in various research labs, so I give an informed answer.

Likewise, when a colleague embarks on teaching a course that they haven't taught before, they usually go to colleagues and ask questions like "What do you cover?" or "What book do you use?" or "What level do you cover this at?" or "In your experience, what do the students have difficulty with?"  Nobody uses the language of assessment bureaucrats in these conversations.  My department actually has a few faculty with extensive training in educational research, and while some of that jargon might appear in their publications they do not (in my experience) use that language when interacting with people in face-to-face environments and discussing things that they will actually do.

A tempting rejoinder is that this is all well and good for higher education insiders when interacting with each other, but how could an outsider know what is going on without documents describing what is going on?  That's a fair point, except that in reality outsiders rely on contacts and networks and experience, not documents.  Part of the reason is that reliance on networks and informal contacts is the way that humans have done business for a very, very long time.  Everyone knows that the next village over has some excellent copper smiths, and their apprentices are well-trained.  If you need somebody to work copper, you get one of them.  Silk traders know which of the villages along the Silk Road have particularly reliable guides and bodyguards, so they know where to go to hire help.

Ah, but we aren't in the bronze age anymore!  We're in the 21st century!  Well, yes, but even in the 21st century the various tech industry sectors in Silicon Valley make extensive use of headhunters.  You'd think that if there were just one place that would embrace transparent documentation for identifying human talent it would be Silicon Valley.  However, Silicon Valley makes extensive use of headhunters who know individuals, and also relies on the reputations of schools.  I can assure you that when Silicon Valley firms recruit from Stanford and Carnegie Mellon it isn't because somebody published an impressive list of Program Outcomes.  I can assure you that even my own university, not in the leagues of Stanford and Carnegie Mellon but nonetheless a widely-respected school for engineering, architecture, and agriculture, acquired its reputation via something other than impressive lists of Student Learning Outcomes.

OK, so we're up against a trifling force like human nature, but when has that ever stopped a technocrat?

Well, we're also up against the need for the very sort of innovation and "disruption" that the technocrats claim to love so much.  In the past few years several of us in my department have done a pretty wide range of innovative things in project-based classes.  From the adoption of new simulation software to novel formats for final projects to the incorporation of peer review into laboratory courses to the introduction of "learning assistants"*, we've seen a lot of innovation in my department.  I sincerely think that some of our advanced courses are much better-structured than the analogous courses that I took.  However, none of this would have happened if each innovation were accompanied by submission of paperwork outlining revised Course Objectives, Student Learning Outcomes, etc.  Can you imagine submitting new paperwork to the massive university bureaucracy every time somebody decides to take their lab class to the next level?  It would be the most insane thing ever.  No innovation would ever happen if we stopped to produce extensive paperwork and revise the Strategic Plan or whatever!

I have no easy answer to the question "How should academic programs be evaluated?"  What I can say is that the managerial mindset that they've been trying to push down our throats for more than a decade is completely disconnected from what actually happens when educators interact with each other, with their students, and with those who would like to employ their students.

*Think TAs, except undergrads, and with much more supervision and far narrower scope.  I'm not always sure that the LAs are doing as much for the students as we would like, but at the very least we know that when you teach a subject you learn it far better than you would have otherwise, so we can be certain that the Learning Assistant is learning.

Zhao, Chapters 5-7

In Chapters 5-7 of Zhao's book, I feel like the worst parts of China's education system are what America's would be if every bad part were turned up to 11.  Chapter 5 talks about the hitting of arbitrary targets:  Whatever the Emperor wants, the Emperor gets.  If it is decreed that a big wall will be built, then damn the expenses, the wall will be built.  If Chairman Mao wants to increase steel production, then steel production will be increased, even if it is to the detriment of every other activity in society.  If professionals (academic and otherwise) need to publish in order to get promoted, and the only thing that matters is the number of publications (because bureaucracies need numerical metrics) then fake journals will be utilized.  If k-12 schools are going to be rewarded for the number of patents filed by students (Zhao is apparently not joking) then k-12 students will file patents of zero quality, and if the system is going to reward this then the  patent office will be obliged to accept and approve those patents.

I feel much the same way about the paperwork demands of the modern American higher ed bureaucracy.  It has been decreed that reports on outcomes assessment are to be written, so those reports are written.  However, thoughtful, meaningful data collection and analysis are non-trivial activities, and the System has no ability to fund the massive efforts that would be required to produce thoughtful evaluation of anything other than the most mass-produced course delivery.  Mass production sounds good to some technocrats, but "Innovation! And! Transformation!" also sound good to them, so the system continues to tolerate the intellectual diversity that is anathema to standardized, mass-produced assessment.  As a result, the system accepts meaningless reports to show that the requisite numbers of reports were filed.*  Moreover, the only thing that The System values even more than the production of reports is ensuring that no boats are rocked, and a truly meaningful look under the hood might tell us...well, it might tell us that the phrase "Gentleman's C" refers to a problem that is timeless and universal rather than old, musty, and covered in ivy.

In chapter 7, Zhao notes that China's rulers are not and have not been oblivious to the problems of exclusive reliance on standardized testing.  The Imperial Exams were abolished a bit more than a century ago, but the competitive test prep habits inspired by those exams have not easily faded.  When the universities give weight to measures of creativity (an oxymoron if ever there was one) parents push this children to excel at those measures and bribe the evaluators.  When the universities give weight to patents (!) the children produce the requisite numbers of junk patents.  If they reward performance in artistic or athletic talent competitions, the parents prepare their children accordingly.

Some  of this is surely a consequence of 1300 years of test prep habits not fading easily.  Cultures do not change overnight, as evidenced by Marxism's utter failure to change Chinese culture. (I hasten to note that the Chinese people's refusal to replace their traditional social structures with Marxism is a very, very good thing!)  However, Chapter 6 offers another point for why these hyper-competitive academic habits have not faded easily:  In both the past and the present, the highest-status jobs in China have been and are government jobs with academic requirements, not private sector jobs.  When there are few paths to advancement, people will do anything and everything that it takes to get on one of those paths.

In the US, we most definitely do NOT have a problem of assigning low status to private sector jobs.  However, we may be seeing a narrowing and bifurcation of career paths.  In the alleged "service economy" there will be the people paid well for jobs at desks and the people paid poorly for everything else.  There are problems with the economic sustainability of this model (e.g. what, exactly, will we produce for sale to the rest of the world?) but if we leave that aside for now, and just focus on the social and educational elements, a bifurcation of opportunity will raise the stakes of elite education, and with those rising stakes we will see more intense competition, and more cutting of corners.

I want the reader to consider the hypothesis that the health of an education system relies on the economy more than the health of an economy relies on the education system.  I do not simply mean that higher GDP means more money to inject into the education system; you can inject cash into good or bad schools and get good or bad results.   Rather, I mean that a robust and sustainable economy with a breadth of desirable paths and opportunities will be conducive to a more robust and sustainable educational  system, with more intellectual breadth and a wider variety of approaches and paths for a wider variety of students.  On the other hand, an economy with bifurcated opportunities will see intense competition for the handful of pathways to the better opportunities.  Competition can be a very good thing, but intense competition for the wrong sorts of things does not lead to better learning or better innovation.

*Let me reassure you that I am, needless to say, merely reporting unfortunate tales related to me by colleagues at other, less conscientious, institutions.  My own institution is, of course, an exception, producing exemplary assessment reports of the highest caliber and most thoughtful analysis.