An open letter to my students on the topic of course evaluations

The following is an open letter to my Fall 2020 students.

November 30, 2020

Dear Students:

In every semester of my 15 years at Ball State, I have made time to talk to my students about course evaluations. This presentation has grown over the years as I have learned more about the history and impact of course evaluations. In recent years, the discussion has touched upon four major themes, and I will describe each in turn.

The first theme is that I take the evaluations seriously. I give more consideration to written responses than the numeric ones. Written comments provide more context, which help me identify affordances for action. I have made changes in my courses based on student feedback, including clarifications and modifications to my grading scheme, sequencing of assignments, and team accountability systems. One significant example that should resonate with Ball State Computer Science students is expanding the final project in CS222 from six to nine weeks, even though it meant shortening the early-semester instructional period.

The second theme is that course evaluations are generally misused in contemporary higher education. These instruments emerged in the middle of the 20th century to provide formative feedback to faculty. Formative is a key word here: the feedback was meant solely for a faculty member to be able to improve their teaching. This is like the written feedback that I might give to you on an assignment, pointing out areas of particular strength while pointing out directions to remediate weaknesses. Predictably, administrators made the mistake of looking at these data from a Taylorist, scientific management point of view, using them as summative evaluations of teaching. A summative evaluation is like the grade you get at the end of the course: impersonal and lacking in context. Hence, we see a process designed for one end (formative evaluation) being misapplied for a different end (summative evaluation) without apparent regard for fitness of purpose. 

I maintain antipathy for the summative ends for personal and professional reasons. It doesn't matter to me where my department or my college rates me with respect to other faculty. I only care about the formative ends; that is, I want to use the teaching evaluations as one of several sources of data to help me improve my teaching.

There is a relevant corollary to this second theme: I believe that many students believe that my teaching evaluations primarily serve the summative role, despite my exhortation to the contrary. I have noticed that many written comments over the years have been composed as if they are going to my supervisor rather than to me. This manifests, for example, in talking about me in the third person rather than addressing me directly. This is pertinent because it supports my argument that this is a systemic failure, not an isolated one.

The third theme of my presentation to students is that they are always welcome to give me private, personal feedback on my course, both during and after. I follow a white box pedagogy in which the inner workings of my course designs are available for any who are curious. For example, my course plans frequently include explanations of my reasoning for assignments and grading schemes, and I post both my course plans and my reflections upon them in public. I inform my students that I am happy to talk to them now, but I also acknowledge that there is a power differential: even if I claim to be unbiased and open, I am still the one who determines their grades. Hence, I welcome and encourage them to talk to me about the course after it's over—whether it's that week, the next semester, or years after graduation. I am happy to hear their stories because I know we can learn from each other through real and honest discourse.

The fourth theme is that the direct beneficiaries of an honest evaluation are the next generation of students. That is, I try to impress upon my students the idea that completing a course evaluation is a form of charity, a service to those students who will come after them whom they may never meet. I don't like the term "giving back" since it implies an obligation of reciprocity that does not exist, so I  frame it instead as an act of volunteering for the good of the community.

With those themes articulated, we can address the current situation. Several years ago, the university switched from paper-based teaching evaluations to online. Response rates plummeted. I have spoken with a few faculty who saw negligible decline in response rates because they continued to have students complete the evaluations during class time. However, most switched to an asynchronous model and had concomitant reductions in response rate. Informal conversations reveal that all faculty and administrators are aware of the severe reduction in response rate, and yet there has been no observable change in how these teaching evaluations have been used. That is, in discussions of promotion, tenure, and merit, I hear people making decisions based on the teaching evaluation data as if they were meaningful.

The problem, of course, is that they are not.

A scientific epistemology reveals the problem clearly. If all students submit honest teaching evaluations, then we can clearly conclude that the results are meaningful. If a majority submit evaluations, then the results are probably meaningful. This was the case with paper evaluations, although one ought to consider that potentially valuable feedback from those who stopped attending class is missing. With the online asynchronous approach, I consistently see only a minority of students completing evaluations. This has been consistent despite my allocating more time and attention to the matter and providing numerous reminders in person and via Canvas. 

It is theoretically possible for a minority of responses to validly represent the population but only when using random sampling. However, evaluations are not completed by a random sample, so this is a non-starter. Indeed, the sampling problem is worse than that: it is only those who are motivated to complete evaluations who do so, and that motivation is often emotional rather than rational. We all know what happens when you combine strong emotions, anonymity, and the Internet.

Reading non-representative student evaluations is far from innocuous; it contains a subtle and serious danger. Scholarship of learning tells us that people effectively cannot "unlearn" ideas. Once you learn something wrong, it is very difficult to overwrite that idea with something correct. Misconception seems to prowl about the subconscious, continuing to direct thought and action. Hence, even if a faculty member knows that the teaching evaluations are not representative of the whole class, the ones they read will still impact their future course designs—potentially for the worse. The feedback loops are insidious when considering the power of confirmation bias, that one reads into the evaluation what one wishes to see, and then uses that to affirm rather than interrogate existing patterns. Garbage in, garbage out.

My conclusion, then, is that I ought not to read your course evaluations this semester for any course that has low response rates. I expect this to be all of my courses, given the ineffectiveness of exhortations and reminders in my past experience, combined with the fact that all my classes are asynchronous and online this semester: I don't even have a time when I can casually discuss the four themes with my students. I want to be explicit that my decision not to read the evaluations is not because I don't care but precisely because I do care. Reading unreliable data has the potential to cause more harm than good. I take my scholarship of teaching too seriously to allow a flawed system to potentially damage my work. 

As always, I welcome feedback from you. Feel free to reach out to me now or in the future. Whether you want to discuss content, pedagogy, or philosophy, know that I am happy to talk with you. Those are the conversations that I want to be formative to me and my practice, for it is there that I believe we can find goodness, truth, and beauty.

Sincerely,

Paul Gestwicki, Ph.D.
Professor
Computer Science Department
Ball State University

The Most Terrifying Story in Fairyland: The Scarecrow of Oz

 Several years ago, I read through all fourteen Oz books by L. Frank Baum with my second son. They are fun fairy tales, although few of them are very interesting. Most have a loose plot that is summed up in chapters of the form encounter a situation, talk a bit, resolve a situation. 

For the past several months, I have been reading the series again, this time with the younger boys. I was excited to start The Scarecrow of Oz the other day because my recollection was that this book, which has almost nothing to do with the eponymous Scarecrow, contains one of the most chilling stories in all of literature.

A few chapters in to the book, the protagonists enter the kingdom of Jinxland, which is part of the Land of Oz that is geographically separated from the rest by impassable mountains. Jinxland is currently ruled by the paranoid King Krewl. On their way to the castle, our protagonists meet Pon, a distraught young man who works for the royal gardener. Their conversation reveals that Pon is in love with Princess Gloria, who loves him as well and wishes to marry him, but is banned from doing so by her uncle, King Krewl. When the protagonists point out that a princess ought not marry a commoner, Pon reveals his lineage: he is the son of the previous king! His exposition lays out the story:

“My father used to be the King and Krewl was his Prime Minister. But one day while out hunting, King Phearse—that was my father's name—had a quarrel with Krewl and tapped him gently on the nose with the knuckles of his closed hand. This so provoked the wicked Krewl that he tripped my father backward, so that he fell into a deep pond. At once Krewl threw in a mass of heavy stones, which so weighted down my poor father that his body could not rise again to the surface. It is impossible to kill anyone in this land, as perhaps you know, but when my father was pressed down into the mud at the bottom of the deep pool and the stones held him so he could never escape, he was of no more use to himself or the world than if he had died. Knowing this, Krewl proclaimed himself King, taking possession of the royal castle and driving all my father's people out. I was a small boy, then, but when I grew up I became a gardener. I have served King Krewl without his knowing that I am the son of the same King Phearse whom he so cruelly made away with.”

The fact that the Oz natives are immortal is well established in the earlier books. Mortals such as Dorothy who come to Oz do not share this quality, which is occasionally a plot point in the earlier books. In Ozma of Oz, the central plot is about a group of good Ozites going to the neighboring Land of Ev to rescue their royal family from the Nome King, who has magically transformed them into ornaments. Ozma and company would never want the rightful rulers of Ev to spend eternity as a nome's baubles. 

Here, in Jinxland, we get the first mention in Oz of a character being essentially killed. Keep in mind, however, that even as Pon points out, they are not actually killed. 

Right after Pon gives this explanation, he is asked about Princess Gloria's father, to which he responds,

“Oh, he was the King before my father,” replied Pon. “Father was Prime Minister for King Kynd, who was Gloria's father. She was only a baby when King Kynd fell into the Great Gulf that lies just this side of the mountains—the same mountains that separate Jinxland from the rest of the Land of Oz. It is said the Great Gulf has no bottom; but, however that may be, King Kynd has never been seen again and my father became King in his place.”

This is how Pon relates the story. Ten chapters later, after Ozma has sent the Scarecrow to Jinxland, the Scarecrow describes the succession a bit differently:

[The Scarecrow] told how Gloria's father, the good King Kynd, who had once ruled them and been loved by everyone, had been destroyed by King Phearce, the father of Pon, and how King Phearce had been destroyed by King Krewl.

It seems that King Kynd's falling into the Great Gulf was no accident. 

By the end of the story, Krewl is deposed, Pon and Gloria are wed, and the two are installed as the rightful rulers of Jinxland, to much celebration and fanfare. The Scarecrow explanation that both King Kynd and King Phearce were "destroyed" may be true, since the fiction does not clearly define the boundaries Ozites' immortality. Indeed, perhaps The Scarecrow of Oz is the canonical explanation. Being held underwater and crushed by rocks does seem like it would be hard to come back from, but we know the Nome's Magic Belt let him turn people into ornaments with no lasting effect on their well-being. Endlessly falling similarly does not sound lethal, especially since in Tik-Tok of Oz, the main characters fall through the Hollow Tube, which ran from Oz all the way through the center of Earth to come out on the other side in the land ruled by Tititi-Hoochoo. Incidentally, that tube was created by a Magician...

“...who tumbled through the Tube so fast that he shot out the other end and hit a star in the sky, which at once exploded.”

“The star exploded?” asked Betsy wonderingly.

“Yes; the Magician hit it so hard.”

“And what became of the Magician?” inquired the girl.

“No one knows that,” answered Polychrome. “But I don’t think it matters much.”

Even the Magician who shot into space and blew up a star with the force of his impact is not necessarily dead nor destroyed.

Maybe the Scarecrow is correct in his pronouncement that both Kynd and Phearce have been destroyed, but maybe this is, in fact, a coup. Ozma rules over all of Oz, but she knows that the geographically-isolated region of Jinxland is prone to rebellion, and King Krewl is not one to bow to her wishes. On the contrary, Pon and Gloria both seem amenable to maintaining Ozma's preferred political order. So, Ozma sends her emissary—the charismatic and clever Scarecrow—to help arrange the coup. The people cheer at Krewl's elimination, for he was an unjust ruler. This allows the Scarecrow to emerge as the hero and subsequently to install Pon and Gloria as the new rulers, both of whom vow fealty and gratitude to Ozma. Pon and Gloria both know their claims to the throne depend on their fathers' disappearances. Despite the prevalence of magic in the kingdom—in characters such as the Wizard of Oz and Glinda the Good, or in artifacts such as the Magic Belt—they damn their fathers to eternal suffering and isolation while their people rejoice at their ascendancy.

That is the most terrifying story in fairyland. It brings an entirely different light to the claims made throughout the series that all the subjects of Oz love Ozma, how they all believe her to be kind and good, and how people throughout fairyland believe her the greatest, wisest, and most beautiful ruler. They all know what may happen if they don't toe the line. 

Beware ye to whom Ozma sends the Scarecrow.

NaGa DeMon 2020 Status Report: Two out the window, One in the oven

At the end of October, I shared some preliminary plans for my NaGa DeMon 2020 project. Once the calendar flipped to November, I started work on a roguelike. After a little over a week of work (off and on), it looked like this:

I made this in Godot Engine using Kenney's 1-Bit roguelike pack. As I mentioned before, I have never made a classic roguelike, and it was fun to build this project up this far. Getting into using BSP for dungeon generation was quite interesting, but I hit some serious technical and algorithmic frustrations in creating useful tunnels between rooms. This took some of the wind out of my sails, in large part because it made me face the fact that, following this path, I would end up with basically "NetHack but worse." I love NetHack, but I didn't really have any inspiration for my roguelike that would make it stand apart. It didn't have a hook, as Josh Ge put it in his inspirational talk.

I let my wander into the realms of hooks, and a thought crossed my mind that's been flitting about intermittently: what happens if you combine PbtA-style action resolution with a CRPG? Regular readers may recall that my NaGa DeMon project last year was a PbtA campy superhero RPG. I am not aware of any game projects, even experimental ones, that attempted to put PbtA mechanisms into a video game; small experiments may be out there, but I think if there was a major one I would have heard about it. In any case, this line of thought combined nicely with the roguelike mindset I was already in, and I found myself thinking about the interface and style of Hero Generations. I have also been reading about Dungeon World and prepping to play the Unlimited Dungeons hack with my family, so that was on my mind. 

Working along these lines, I built a prototype in about a week that looked like this:

It actually looked a little better than that, but not much. It seems I abandoned this one in the middle of working on a feature branch, so you cannot see the text indication that there are some goblins in the dungeon. 

What made me abandon this project was a similar sense of being directionless that hit me a few days into the development of the earlier prototype. I implemented action selection for actions akin to Dungeon World's Hack-n-Slash and Discern Realities, as well as Escape and Advance moves for dungeon-delving. Each of these supported the three traditional PbtA roll outcomes of failure, partial success, and success—an important and symbolic part of the game that I have written about before. Middling successes resulted in a variety of options that depended on the contents of the current room.

This all worked in a technical sense, but it was just a mechanical toy: I didn't have a goal besides seeing what happens if I did it. Indeed, I think both this prototype and the previous one were fun implementation projects that merited more attention to paper prototyping. I think part of me knew for both of these that, if I had spent more time paper prototyping, I would come to the clear conclusion that I could make these things but that I didn't have much reason to do so.

After stopping work on this prototype, then, I went back to first principles: what project is exciting enough to me that, with only half the month left to go and another pile of looming work, would keep me inspired to finish strongly? I decided to go back to the first idea that I had written about in October, that idea that has been pulling at my interest for some time: a card game about manipulating the relationships of powerful characters who are not controlled by any player directly. 

I started work on that other project in earnest late last week, and today, I published v0.1 of the rules as a print-and-play game. The working title is "Intrigue," and you can download the rules and cards on the Web site. The graphic design is rudimentary, and there is a complete absence of illustration. That said, I've been happy with the game in playtesting. The story of this game's development, including twists, turns, and technology, will be the subject of a future post.

Teaching Game Design with Player Practices: A Blog Letter to Chris Bateman

The following is a blog letter to Chris Bateman, in response to his "Write to Me" post on October 27. We had a brief email exchange, based on which I have moved the question here.

I heartily enjoyed reading your three-part serial on game dissonance on the International Hobo blog [1, 2, 3]. Your theory of Player Practices is intriguing, and it certainly is appealing in that it bridges gaps left by systems-focused or story-focused approaches.

My question for you is a practical one: what would it look like to teach an introduction to game design using player practices as the frame?

For background, I teach an introduction to game design course that is terminal and elective: it is not a prerequisite to any other course, nor is it a formal part of a games-related curriculum. Although the course is hosted in a Computer Science department, the course involves no programming—a point that sometimes surprises students who enroll! Instead, we focus on analog games, which means anyone with markers, dice, and a deck of cards can be productive. My emphasis in the course is in helping students learn the value of rapid iteration and player-centered philosophy. That is, they should move quickly from idea to testable prototype, and they should evaluate their games with real players. Incidentally, this is something that is of special value to the Computer Science majors and minors who take the course since it brings a People focus in a curriculum that overwhelmingly emphasizes Things.

How might you bring Player Practices to bear in such a course? My thoughts turn, for example, to how I have had students pitch their final projects. Usually I have them write a one-page concept document in Tim Ryan's format. This format includes a Background, in which I expect them to cite how their work is inspired by or based on other ideas they have seen; clearly, Player Practices may be applicable here. I also have them write a second-person narrative describing the player's experience playing the game. I have touched upon a peculiar phenomenon in a few other blog posts, but I don't think I ever devoted one post to it; namely, that almost all of the students struggle with writing about the player's experience and instead write about the character's experience. This seems to happen regardless of the amount of frontloading I do to explain the difference. It struck me as I was preparing to write this letter that, I think, this is a case where player practices may provide a language to help the students.

Thanks for your willingness to share your thoughts on the matter. I am always revising my courses, and I look forward to considering how your perspective on Player Practices may be incorporated.

Cheers!

UPDATE: Here is Chris' thoughtful response.