Reflecting on the Fall 2019 CS315 Game Programming course

My students are currently taking their final exam, so this seems like a good time to start my end-of-semester blog post about CS315 Game Programming. This was my third Fall semester teaching this upper-level elective course using Unreal Engine 4. The semester ended up consisting of four "mini-projects", each about two weeks, and one six-week final project, which was completed in two iterations. By and large, I am happy with how the semester went: students learned how to work with some contemporary tools, including Perforce Helix for centralized version control, and they made interesting final projects. What I want to document here are some of the struggles, because it will do me more good when planning next year's class than focusing on the successes.

It turns out that almost all of my frustrations from the semester stem from my decision to use Specifications Grading again. For people who are not familiar, you can easily hop over to the course plan's projects page to see what they are. Briefly, I laid out all the criteria for which student work would be evaluated ahead of time, and like last year, I asked students to submit self-evaluations in which they graded their own work.

This leads quickly into the first problem: students did not seem to understand how to use checklists. It feels so strange to even type that, but it's true. As part of their submission, the students had to complete a checklist, and then based on what was satisfied, they could know—and had to say—what their grade would be. However, more often than not, I would read through the student's submission and have to point out that they didn't actually satisfy some criteria. I designed a little leniency for students who legitimately did not understand a criterion or two, but what I didn't expect was that several students made the same mistakes again and again and again. I forced the students to rotate partners during the Mini-Projects, thinking that this would ensure that mistakes would be caught by the partner; instead, what I saw was that the misunderstanding (not the understanding!) spread to new partners.

I suspect that a major reason for the checklist problem is that students are so deeply brainwashed into the "turn this in and hope for points" model that they cannot conceive of an alternative. Certainly, in my years of teaching, I've had plenty of push-back on unconventional things I do. (I continue to do unconventional things, partially because I want students to learn to question conventions.) I can work on clarifying the language around the specifications themselves of course, but I feel like this is treating a symptom rather than a cause.

There is one place where my instantiation of specifications grading contrasts, as I recall, against the presentation in Nilson's well-known work. She describes making a choice between more hurdles and higher hurdles, but my version of specifications grading is both more hurdles and higher hurdles: students have to do more and better work to earn higher grades. This is sensible to me, but I wanted to mention it here because it is a lever that I could pull in an experimental assignment or section.

Another problem I encountered with the specifications grading this semester was that a minority of students were able to follow the specifications I provided to earn relatively high marks but, in my professional opinion, without really meeting the learning objectives. For example, I had a B-level criterion which was, basically, that the project should have all the parts of a conventional video game: a title screen, gameplay, an ending, and the ability to play again. An alarming number of teams did not handle mouse input controls properly, so that once you click in the game, the mouse is captured and the cursor made invisible. This means that technically you can still navigate a UI menu, but without being able to see the cursor, it's awfully difficult. Their conventional solution seemed to be to use Ctrl-F1 to release the cursor so they could see it again: an editor kludge for a runtime problem. Did such teams satisfy the criterion? Well, yes, but also no. I liberally allowed it, leaving notes in my review that they should fix this, which almost nobody did. I could, of course, add text to the already-wordy criterion to say "If you are developing for a Desktop application, and you are using mouse navigation, make sure etc." That's just one special case of a particular environment, though. What I think I'm really running in to is the problem if specifications grading in the face of creative, wide-open projects.

Several students took my examples wholesale, brought them into their projects, and then submitted them as satisfying the relevant criteria. For example, I showed C++ code to count how many shots a character fired; student teams put this into their game and then checked the box saying that they included C++ code. Technically yes, but without any semblance of understanding. Another student took my dynamic material instance example wholesale and put it into his final project. Again, no indication of understanding the pieces, just copying and pasting it into his project and claiming that he included dynamic material instances. Yes, they're in there; no, there's no evidence of understanding. Some of this could, in theory, be cleaned up by changing the specifications, but then it gets into the same kind of problem as measuring productivity in programming. Exactly how different from my example does a student's work have to be to demonstrate that they understand the comments? "Exactly" is the key word here if the specifications are going to be objective.

I'm left with this sinking feeling that specifications grading are not worth the effort and that I should return to my tried and true minority opinion on grading: use triage grading for everything. This allows me freedom to say something like this: "Use dynamic material instances in your project in a way that shows you understand them." Then, I can fall back on saying that a student's work either clearly shows this (3/3 points), clearly doesn't (1/3 points), or is somewhere in between (2/3) points. This clear and coarse-grained numeric feedback can be combined with precise and crystal-clear written feedback to show students where they need more work, which appeals to me much more than my grimacing at the student's submitted checklist, then at the student's code, and then saying, "Yeah, I guess."