I wasn't going to write up this post this morning—plenty of other writing and course planning activities to do. However, the Fall 2018 student teaching evaluations were just published, and I was kind of blindsided by the feedback from my Honors Colloquium on Serious Game Design. I'm engaged in some discussions on the Facebook about it right now, and I am grateful to those who are talking through some of the issues with me.
I want to give a high-level view of the course first. The students pursued a great variety of projects, and both me and my colleague at Minnetrista agreed that the overall quality of projects was improved from last year. I believe this is in large part because we already had the experience of last year. As I wrote about in September, I think having Fairy Trails as a lens helped the students see more opportunities than when we were greenfielding in 2017.
This was the first time I had Architecture majors in the class due to a change in the time I taught the class. I told many people throughout the semester how one of the great blessings here was that they were used to the idea of giving feedback. Some classes struggle to get the first idea out when giving feedback to students' work, but this is just part of the process for CAP majors. They were almost always the first to give feedback, which then got the ball rolling for the rest. By and large, they also took peer feedback well, which also created a good environment.
One thing I was aware of during the semester was how I got very few questions from this class. During the semester, only one student came to my office unsolicited to seek my advice on their classwork. There was one student with whom I had email exchanges about his project as he had a hard time getting his feet under him, but we sorted that out and his project turned out quite nicely. Aside from that, it was quiet. The problem is, quiet can signify both diligent focused work and gross misunderstanding of expectations.
All that said, this group of students gave me the worst teaching evaluations I've ever had--by far. It's patently clear in reading them that a group of students regularly conversed about me but never spoke with me about the course. Their feedback all covers the same ground, which also makes it clear that this group were my Architecture majors, or they were at least a vocal minority. It seems they harbored grudges and misconceptions all semester. Most of these issues could have been addressed by reviewing the course plan or, without a doubt, by talking with me.
In the Facebook discussion, a colleague posted this excellent quotation from Eckhart Tolle: "When you complain, you make yourself a victim. Leave the situation, change the situation, or accept it. All else is madness." That's a nice sentiment, and I certainly don't want to use my blog as a place to complain. (Well, maybe just a little bit about how students don't read course plans and don't take notes.) Moving forward, I want to point out a few areas of friction that these students brought up, what I think actually happened, and what I can do about it in the future.
Accounting for Grades
The students comments made it clear that, despite the course plan clearly laying out the evaluation scheme, many students were relying on Canvas to report their grade to them. I told them in the first week not to do this, but this didn't stop them. (Hm, not reading, not taking notes.) Unfortunately, they also didn't talk to me in any way during the semester to reflect this error; that is, I had no evidence until now that they were internalizing their grade incorrectly.
Canvas' grading system is rudimentary at best. In fact, it's naive to the point of being dangerous: I think it's one of those design decisions that pushes faculty toward bad teaching practice, but I digress. What I need to do is remember to mark all assignments as not counting toward a final course grade. I just verified, and if I did this consistently, students would see their grades as 0 out of 0 points, with "N/A" in the letter grade column. That sounds perfect. It would serve as a reminder to the students not to trust the automated system.
Consulting Grades
In the course plan, students could earn a few points for consulting with me or another game designer on their final project. In the course plan, it says that this has to be done "during the production period." The schedule for the final project included two rounds of pitches, five status reports, a practice final presentation, and an actual final presentation to the community partner. To me, it was obvious that "the production period" would be the time from the pitches to the end of Status Report 5. After that, production should be done, and we are giving presentations.
It seems that students did not see it that way. I received roughly four requests from students in the week leading up to the practice final presentations. I responded to these emails that I would be happy to meet with them, but I also pointed out that the production period had passed. Three of them then were not interested in meeting any more; one of them still met with me, and we had a productive discussion. This is important: the students contacted me to meet not to get my feedback on their projects, but only to get points. The points were, of course, supposed to incentivize them to get feedback, but it seems they were the end in themselves.
In week 2 or 3 or so of production, I reminded them in class that they could earn Consulting Points. It was not like I wanted to trick them out of it. I did note, though, that as I made that reminder, no one wrote it down. (Hm, not reading, not taking notes.)
What I can do in the future is to lay out the calendar more clearly with named periods. This will not require them to actually think about the process, but instead to match keywords. I don't mean that in a snarky way: thinking is hard, pattern-matching is what our brains do automatically. It's an easy way to reduce friction.
Also, a few students either did not understand or completely dismissed the notion of "practice final presentations." I figured the language was clear, but a few students had not actually finished their work by this date. I can make it more clear in the future that "practice final presentation" means that you should actually have your work done and be focused on practicing your presentation. How could it mean anything else? The devil's advocate position is, I suppose, that students don't understand that you play like you practice so you should practice like you play. That is, they see "practice" and think "not real" instead of "preparation for excellence."
CAP Trip
It seems that the students in the College of Architecture and Planning have a week during the semester when they go on a field study. I gather that, during this time, no CAP faculty have any expectations that they will do any work for their other classes. That's fine for CAP, but of course, my class doesn't stop because a subset of students is going on a trip. Indeed, I think the students don't recognize that I can hear, because in the weeks leading up to the trip, I heard one tell the other what a great vacation this trip is: walk around a historic city until lunchtime with your class, and then have the rest of the day to yourself.
When the students gave me their travel forms, I filed them away and told them it would not be a problem. These are university forms that say quite clearly that the students recognize they are not excused from class responsibilities. All my assignment deadlines were posted from the first day of the semester, and I figured the students would either travel with their laptops to get their other classes' work done, or just get everything done before they left. Leaving this assumption unstated was a mistake, I gather, since in the course evaluations, the students interpreted it differently. In their minds, they gave me the forms, and I did not tell them what to do until just before they left, when they asked what they should do. Again, I thought it would be obvious: they should do what everyone else is doing. After some emails back and forth, where one student insisted that they are forbidden from bringing expensive items like laptops on their trips, I extended the deadline for these students. And they complained about this in the student teaching evaluations. It still boggles my mind: I made an assumption, it turned out to not match theirs, and I gave them an extension. And I'm the bad guy... because I expected them to do work? Because they had to do something that required being online? I really don't know. I wonder about other faculty who regularly have CAP students in class, do they actually waive requirements or something, the way that the other CAP faculty seem to?
This is one where it's not clear to me what I could do differently in the future, except perhaps to go back to the normal time I taught the class, which time prevented CAP students from enrolling.
Donuts and, more generally, Food (and, more generally yet, Culture)
There was one day that I was late to class. On the next class meeting, I brought donuts by way of apology. No one mentioned that in the course evaluations.
There was a girl in the class who, every morning, brought her breakfast into the classroom and ate it before we began. This would make the whole room smell like pancakes and syrup, which I found annoying. I asked her if she would please eat her breakfast elsewhere so as not to make the room smell like food. She decided to eat her breakfast sitting on the floor of the hallway. She could have eaten it wherever she bought it, but she chose the floor outside the room. She never complained about it. In the course evaluations, a few students pointed to this as my showing favorites, that I would not let this poor, hungry student who had been in classes since 8AM eat her breakfast in the classroom. They pointed to my not wanting the room to smell like food as flippant. Now, I didn't point out to them at the time that there is actually a university rule against anyone having food in the classroom. I thought about bringing this up the day I asked the student not to eat in the classroom, but I didn't want to be "that guy" who throws the rulebook, when instead I figured I could just appeal to a sense of shared space and community. Well, that clearly backfired.
What can I do differently? Again, it's not clear. I think what happened in terms of students interpreting my decision as one of bias or callousness, it all happened in discussions where I was not involved. They were convinced they understood my rationale, and so there was no reason for them to ask or question it. It's the dark side of human tribalism: I was the enemy, and they were the valiant underdog heroes. Indeed, this fits exactly into what Lukainoff and Haidt talk about in The Coddling of the American Mind as one of the great untruths we are teaching students, that the world is made up of good guys and bad guys—homogenous tribal thinking.
I had not thought of that before, but this also leans into what I thought was an even better treatment of these themes by Campbell and Manning in The Rise of Victimhood Culture. The students did not engage with me on any of the topics they found challenging, even those that I chose as intentionally provocative, such as Harry Potter, with which people have a kind of religious devotion. Instead of engaging in dialectic with me, they wrote lengthy arguments to administrators in course evaluations, wrote about how I hurt their feelings, and threatened to write to the dean. That's exactly in keeping with Campbell and Manning's description of Victimhood Culture: easily offended and seeking justice by appeal to authority. Yikes. As I dig into why I am so upset by these evaluations, perhaps I have found my answer here.
Biased Participation Grading
Another common theme in the negative evaluations was that I was biased in my grading, that I would grade people I liked better than people I didn't. Again, I struggle to understand how they saw it this way. The way that participation grades worked is that students could earn up to three points in a day for participation. If someone said anything on topic, I wrote their name down in my notes, and they got three points. If they didn't, but they showed up, they would get one of the three points, in keeping with my usual triage grading scheme—3 meaning "essentially correct" and 1 meaning "essentially incorrect". Whenever I gave less than full credit, I included a note along the lines of this: "I have no record of your participating in class today." Frequently, I would add, "My records aren't always perfect, though, so let me know if you think I missed something."
As I said, I struggle to understand their comments here. They claimed that I would give students I "liked" more praise in class for their comments and that I would push back on those I didn't like. To some extent, that may be true, since I tend to like students who are prepared and give thoughtful feedback. However, it had no impact on the grading. I do not know where the miscommunication here was.
I will note that there was no time during the semester when any student contacted me and appealed their participation grade. No student said, "Actually, I mentioned in Zach's presentation that he should have less randomness." In truth, I would have believed anything they sent me, but they sent me nothing. How could they, when they took no notes? (Hm, not reading, not taking notes.) Even when I reminded them that the final exam would include questions about each others' projects, no students took notes from the others' presentations. It was rare that a student would even take notes of feedback during their own presentations.
What's to be done? One thing that I am very hesitant to do, but student behavior seems to be driving me toward, is grading students' in-class notes. I believe they have no idea what the value of notes is, and further, I think most of them have no idea how to effectively study. I have a blog post in draft about my reading of Make It Stick earlier this semester, and that (combined with Dorothy Sayers' "The Lost Tools of Learning") really got my wheels turning about how little students are prepared for lifetime learning. Where should the buck stop?
Book?
Another bit of the student conspiracy was a set of complaints that I had based my course on the Game Design Concepts course taught by my friend Ian Schreiber. I told them at the start of the semester, his online materials are as good or better than any books I have read on game design, so we would use that as the basis for our readings. The cabal of angry students turned this into complaints that all I had done was use his material. They did not mention the additional readings, nor the fact that I had saved them a few bucks by using a free online text. I chalk this one up to another complete disconnect from reality, that in their echo chamber of complaint, they had not realized that "basing a course off of someone else's work" is exactly what the whole textbook industry is based on. It would be interesting if we could all only teach in areas where we had written the textbook, but that clearly doesn't scale, not like amazing free online content prepared by well-respected instructors.
Project Grading
The final complaint that I want to address was one that I graded the projects in a biased manner. This is really fascinating because I did not grade the projects as such at all. The course plan is very clear about this, that I only graded their process, not their products. It is a point I emphasize in a few places in the course plan and regularly in my presentations. For each status report, students had to update a design log and address four questions: what was the primary design goal you pursued since the last status report? How did you prototype your ideas? How did you evaluate your prototype? What conclusions did you draw, and what do they imply for your next steps?
The design log was a new step for me. I added that as a way to have more evidence about whether students were actually following the process. A few students hit some rough spots in understanding the role of the design log in the process, and I could have added more guidance here. That said, they also worked, in that they showed me who was not really doing anything between status reports. I had one student essentially lie to me about her progress, since she did not know that I could check the document history in Google Docs. She was smart, though: she was careful not to explicitly lie in her email to me, but instead to imply something that was untrue. I called her out on this in my feedback, and she never responded to it, electronically or in person. I wonder if, in retrospect, I should have pursued an academic dishonesty case; maybe she would have gone away of her own accord. As you would expect, a liar is not going to be a great contributor to the course atmosphere and goals.
Talking with my friends on The Facebook, I got to wondering if the CAP majors in particular struggled with the tight iterations required of game design. They study design, ostensibly, and they believe they are good at it. However, their loops are very long, and their feedback does not come from end user evaluation, but from discussions around models. In game design, as in software design, we work in tight loops. It's possible that even though we talked about the importance of iteration, that because they believed they already understand design thinking and iteration, that they were incapable of seeing that they were not following the processes I required. If not incapable, then perhaps at least unprepared. I don't have direct evidence of this, it's merely a hypothesis. It's hard to believe that I could have been any more clear in my course plan, in my grading scheme, and in my feedback that I expected short iterations complete with evaluations and reflection; yet, this set of students complained about my grading their "projects" rather than helping shape their process.
Fun?
This one goes a bit sideways from the others, because it was some honest feedback from a student outside the Angry Gang. This student commented that they wished we had spent more time on how to make fun games. I wanted to add this here, because I've gotten this kind of feedback in my game design courses before, and it's really fascinating. It's like the student is just laying down their cards and saying, "I believe you can just tell us how to make something fun, and you didn't. I really did not learn anything this semester at all." This is another area where I'm not sure how to make it more explicit. We talk about this a lot, and the course plan, as mentioned above, is all about the process of game design for exactly these reasons. That said, I understand how someone who only goes through the motions of the course could come out without understanding this. That is, if you see a course as just being a series of work items and don't actually think about them, then you could come out saying that I didn't teach how to make something fun; to realize that this is impossible requires thinking, listening, and perhaps even reading and taking notes. (Hm.)
Conclusions
I love teaching these courses on game design. Reflecting on them, they do actually produce some of the worst course evaluations I receive, and I think there are a few reasons for that. One is that many colloquia are blow-off courses. Many alumni have told me such, that students expect colloquia to require minimum effort, a bit of BS, and then you walk away with an A. Mine is designed to be a rigorous challenge, and this perhaps sets us up for conflict. Another is that the Honors Students generally, and some subcultures like CAP in particular, are fed a line by the university that they are elite and special. My class is something none of them have ever done before, and so they make mistake, and they harbor misconceptions... and I tell them. Cognitive dissonance kicks in as follows, "If I am special and smart and successful, but this guy is saying that I am wrong, then he clearly must be wrong." Add homogeneous tribal cultures, and you have a recipe for disaster. I've never had a sizeable minority in my class all from my major before since these colloquia are open to all, but looking back on this, a lot of factors point to an inevitable conclusion that the students would circle the wagons rather than engage.
I think I have done a decent job in this essay of identifying where I made mistakes, where I might have known better, and what I can do differently. I'm still trying to sort out my own reaction to their feedback. There is certainly an element of pride, but there's also a sense of treachery, that these students had harbored these grudges and misconceptions all semester, all while smiling at me and responding politely to my questions. Writing this helped me identify elements of victimhood culture, and this helped in two ways: I can step back from the phenomena and understand them in an empirical, research-oriented way; and I can understand why it would upset me so deeply, as reading those books I referenced also did. They lead to a sort of deep, existential dread around education and society.
Incidentally, the evaluations on my other courses were par for the course, echoing some of the thoughts I've shared about strengths and weaknesses already. The one pleasant surprise there was that one student commented positively about my framing of the required statement on diversity (discussed here), the only comment that I received on that all semester. I'll keep it in there.
Thoughts? Comments? Suggestions? Criticisms? Perspectives? Share them below. No need to let Zuckerberg hold all our conversations. Thanks for reading!
No comments:
Post a Comment