The following is an open letter to my Fall 2020 students.
November 30, 2020
Dear Students:
In every semester of my 15 years at Ball State, I have made time to talk to my students about course evaluations. This presentation has grown over the years as I have learned more about the history and impact of course evaluations. In recent years, the discussion has touched upon four major themes, and I will describe each in turn.
The first theme is that I take the evaluations seriously. I give more consideration to written responses than the numeric ones. Written comments provide more context, which help me identify affordances for action. I have made changes in my courses based on student feedback, including clarifications and modifications to my grading scheme, sequencing of assignments, and team accountability systems. One significant example that should resonate with Ball State Computer Science students is expanding the final project in CS222 from six to nine weeks, even though it meant shortening the early-semester instructional period.
The second theme is that course evaluations are generally misused in contemporary higher education. These instruments emerged in the middle of the 20th century to provide formative feedback to faculty. Formative is a key word here: the feedback was meant solely for a faculty member to be able to improve their teaching. This is like the written feedback that I might give to you on an assignment, pointing out areas of particular strength while pointing out directions to remediate weaknesses. Predictably, administrators made the mistake of looking at these data from a Taylorist, scientific management point of view, using them as summative evaluations of teaching. A summative evaluation is like the grade you get at the end of the course: impersonal and lacking in context. Hence, we see a process designed for one end (formative evaluation) being misapplied for a different end (summative evaluation) without apparent regard for fitness of purpose.
I maintain antipathy for the summative ends for personal and professional reasons. It doesn't matter to me where my department or my college rates me with respect to other faculty. I only care about the formative ends; that is, I want to use the teaching evaluations as one of several sources of data to help me improve my teaching.
There is a relevant corollary to this second theme: I believe that many students believe that my teaching evaluations primarily serve the summative role, despite my exhortation to the contrary. I have noticed that many written comments over the years have been composed as if they are going to my supervisor rather than to me. This manifests, for example, in talking about me in the third person rather than addressing me directly. This is pertinent because it supports my argument that this is a systemic failure, not an isolated one.
The third theme of my presentation to students is that they are always welcome to give me private, personal feedback on my course, both during and after. I follow a white box pedagogy in which the inner workings of my course designs are available for any who are curious. For example, my course plans frequently include explanations of my reasoning for assignments and grading schemes, and I post both my course plans and my reflections upon them in public. I inform my students that I am happy to talk to them now, but I also acknowledge that there is a power differential: even if I claim to be unbiased and open, I am still the one who determines their grades. Hence, I welcome and encourage them to talk to me about the course after it's over—whether it's that week, the next semester, or years after graduation. I am happy to hear their stories because I know we can learn from each other through real and honest discourse.
The fourth theme is that the direct beneficiaries of an honest evaluation are the next generation of students. That is, I try to impress upon my students the idea that completing a course evaluation is a form of charity, a service to those students who will come after them whom they may never meet. I don't like the term "giving back" since it implies an obligation of reciprocity that does not exist, so I frame it instead as an act of volunteering for the good of the community.
With those themes articulated, we can address the current situation. Several years ago, the university switched from paper-based teaching evaluations to online. Response rates plummeted. I have spoken with a few faculty who saw negligible decline in response rates because they continued to have students complete the evaluations during class time. However, most switched to an asynchronous model and had concomitant reductions in response rate. Informal conversations reveal that all faculty and administrators are aware of the severe reduction in response rate, and yet there has been no observable change in how these teaching evaluations have been used. That is, in discussions of promotion, tenure, and merit, I hear people making decisions based on the teaching evaluation data as if they were meaningful.
The problem, of course, is that they are not.
A scientific epistemology reveals the problem clearly. If all students submit honest teaching evaluations, then we can clearly conclude that the results are meaningful. If a majority submit evaluations, then the results are probably meaningful. This was the case with paper evaluations, although one ought to consider that potentially valuable feedback from those who stopped attending class is missing. With the online asynchronous approach, I consistently see only a minority of students completing evaluations. This has been consistent despite my allocating more time and attention to the matter and providing numerous reminders in person and via Canvas.
It is theoretically possible for a minority of responses to validly represent the population but only when using random sampling. However, evaluations are not completed by a random sample, so this is a non-starter. Indeed, the sampling problem is worse than that: it is only those who are motivated to complete evaluations who do so, and that motivation is often emotional rather than rational. We all know what happens when you combine strong emotions, anonymity, and the Internet.
Reading non-representative student evaluations is far from innocuous; it contains a subtle and serious danger. Scholarship of learning tells us that people effectively cannot "unlearn" ideas. Once you learn something wrong, it is very difficult to overwrite that idea with something correct. Misconception seems to prowl about the subconscious, continuing to direct thought and action. Hence, even if a faculty member knows that the teaching evaluations are not representative of the whole class, the ones they read will still impact their future course designs—potentially for the worse. The feedback loops are insidious when considering the power of confirmation bias, that one reads into the evaluation what one wishes to see, and then uses that to affirm rather than interrogate existing patterns. Garbage in, garbage out.
My conclusion, then, is that I ought not to read your course evaluations this semester for any course that has low response rates. I expect this to be all of my courses, given the ineffectiveness of exhortations and reminders in my past experience, combined with the fact that all my classes are asynchronous and online this semester: I don't even have a time when I can casually discuss the four themes with my students. I want to be explicit that my decision not to read the evaluations is not because I don't care but precisely because I do care. Reading unreliable data has the potential to cause more harm than good. I take my scholarship of teaching too seriously to allow a flawed system to potentially damage my work.
As always, I welcome feedback from you. Feel free to reach out to me now or in the future. Whether you want to discuss content, pedagogy, or philosophy, know that I am happy to talk with you. Those are the conversations that I want to be formative to me and my practice, for it is there that I believe we can find goodness, truth, and beauty.
Sincerely,
Paul Gestwicki, Ph.D.
Professor
Computer Science Department
Ball State University
No comments:
Post a Comment