Reflecting on the Fall 2012 Game Design Colloquium: Expectations and Reality

Background

In Fall 2012, Ronald Morris and I team-taught an honors colloquium about serious game design. At the start of the semester, I wrote about the achievement-based grading system we intended to use and how it fits into a larger academic-year game production project. To make a long story short, we had a team of students from a variety majors, and each was expected to make an educational non-digital game for the Indiana State Museum. I took the day today to compose my reflection on the course, which is now shared with you in four parts: my expectations when we designed the course; what actually happened when we ran the course; insights gleaned from the final exam; and what I might do differently next time.

This is my third attempt at composing this reflection. The first was a collection of notes and observations that were too raw to share. The second was a narrative that attempted to map expectations, execution, and lessons learned into nice triads. This ended up being a mess, as the relationships are much more rich than simple one-to-one mappings. I mention this because it strikes me as significant to my reflective practice: in taking the time to write carefully, I have been able to understand the semester a bit better and come to peace with some things that had been bothering me.

Expectations

Students would enroll because they were interested in either the topic or generally in immersive learning experiences. By explaining all of the expectations in the course description, including an explicit statement regarding nine hours of attention per week, students would know that they needed to take the course seriously and commit themselves appropriately. Achievement-based grading would motivate students to engage with the material. Given that this was an honors colloquium—and hence only open to students in the Honors College—they would be responsible enough to set a good pace of achievements throughout the semester. The achievements leaderboard was posted in a shared location, which would lead to positive peer pressure for everyone to keep up. Some achievements rewarded students for reading and commenting on each others' essays, and this would lead to interesting discussion, deeper thinking, and better critical analysis.

Since the students have set up their own schedule based on interest and other personal commitments, their independent activity during the first third of the semester would complement the in-class discussion of game design fundamentals. This activity would then strengthen their work in the next third of the semester, which would be devoted to iterative prototyping. By the final third, the students would be focused wholly on producing their games, with all the other achievement-based background work done.

To help students with the Socializer achievement—as well as getting through the canon of genres—I would announce all of the meetings of the Society for Game Design and Development as well as the Game Days at the Muncie Public Library. Students would approach these events with some expected trepidation, but once they realized the friendliness of the community, they would continue to attend these events, both for personal pleasure and to gain exposure to new game design ideas.

At the end of the semester, we would showcase our games to the Indiana State Museum. Some of the games may be usable right away, but all would serve as media for communicating important design ideas. Given their expertise in production, the ISM could very likely produce better looking or more durable game bits. This means that clarity would be a key concern in the students' designs.

Each student would make their own game deliverable, but the designs themselves would not be graded. This could be too subjective as well as dependent upon prior experience and extracurricular digital production skills. Instead, we would give students a mark based on their following a good iterative process. As explained in the course description, we expected each design to go through several iterations, and failure to present a prototype in each iteration would result in grade reduction. Hence, the only source of grades for the student were the achievements and the iterative design process, meaning that students could really choose their grade based on their level of commitment. Because we did not need any other grades, the final exam would be a reflective exercise.

Reality

We have little information about why the students enrolled in the colloquium. Most of the students had little to no background knowledge in game design, and little passion for the topic. One student told me that she enrolled because "design" was outside of her major and experience: she identified it as an area for personal growth during her undergraduate studies. This is a great reason to take a course, and she proved herself capable during the semester. Another confided that he took the course only because of the time it met and to satisfy the Honors College's two-colloquium requirement.

For the first third of the semester, almost no achievements were completed. Ron and I gave the students some class time without us to collectively develop a plan, and this partially worked: they started holding meetings with their classmates to play the games of the games canon. Unfortunately, this happened after the transition from fundamentals to prototyping in the course schedule. Because they were completely unfamiliar with many established genres, they could not draw upon these in their own prototypes.

As we got into the prototyping phase of the semester, we started by formally scheduling students every two weeks or so. The students preferred a more ad hoc approach in which individuals would bring prototypes as they were ready. This caused a conflict with the course description, in which we intended calendar-based iterations with students presenting a new prototype each cycle. The students were right, however, that some prototypes took longer than others, especially in those cases where students completely changed topics. We agreed to let the students follow this process, entrusting them to hold each other accountable, even though it meant our iteration-based grade penalty system would become unenforceable. 

As the semester went on, we witnessed design stagnancy. Many students brought essentially the same prototype after having had a week or two to work on it, and it became clear from their discourse that there was little to no playtesting happening outside the class meetings. A few students were able to recover from this with some not-so-gentle pushing by Ron and me, ending up with good designs but only after having fought against our feedback.

At midsemester, there was still very little progress on the achievements, though with a few exceptions. Ron and I held individual conferences with the students, during which we provided some feedback on their designs and also encouraged them to enact a plan for meeting the achievements. There was a lot of smiling and promising, but very little material evidence that these meetings were worthwhile. Most of the achievements ended up being completed in the last two weeks of the semester, many appearing rushed and certainly not contributing to the students' all-but-finalized designs. Because we had these individual meetings, though, we know that it was procrastination and a failure to plan on the students' part that led to this situation, as opposed to a fundamental flaw with our expectations.

Some of the achievements simply took a long time to complete: students made progress on playing the games in the canon for several weeks before marking the achievement as complete. However, the one that was delayed the longest, which caused me the most frustration, was the Socializer. Only one student completed this achievement before the last two weeks of the semester. In their reflective essays, students acknowledged—after the fact—that it would have been better to go earlier and more often.

Near the end of the semester, we engaged in two rounds of external playtesting. The first was with Motivate Our Minds, and the second, College Mentors for Kids. Our students highly regarded these opportunities, even in those cases where the young playtesters were not quite in the target age group for the game. I would have liked to see more preparation on the part of my students: some had not defined all of the terminology in their games, and others were missing key physical pieces of their games—which, again, betrayed a lack of playtesting.

We had three in-class presentations in the last two weeks of the semester for the Scholar achievement, one each on A Theory of Fun for Game Design, The Art of Game Design: A Book of Lenses, and Homo Ludens. I was surprised by how easily the students slipped into discussion mode, engaging each other and voicing opinions. At the same time, it was disappointing at the end of the semester that some of them were still falling prey to very naive notions of game design and learning. I opted against steering these conversations, instead listening for evidence of who had integrated knowledge from their experience and who had not. I would like to claim that those in the former group made better games as well, but I think my recollection of the conversation is too colored by my post hoc opinion of their games.

The students made an impressive display of their games for the community partners. We invited representatives from the ISM to campus, and each student explained their game's theme and demonstrated gameplay. The ISM staff responded most positively to those that were simple and effectively described. This is predictable but unfortunate for the students, since the games that actually contained the best designs were not necessarily given the attention they deserved. What I mean by "best designs" are those that reflect what is known about games and learning, and not coincidentally, these were mostly the ones that went through the most revisions.

Most of the games do good design elements, and I believe that we can call this a successful outcome. Comparing them to the students' first critical analyses and initial designs, we can see a general transition from novice toward advanced beginner. However, there are also those corresponding design decisions that appear to have been left without critical analysis, but of course, learning how to ask these questions is part of learning the craft. We are currently considering which of the designs to move forward with into digital production in the Spring, and there are several that I think would make great starting points.

Final exam reflection

The final exam consisted of the following questions:

1a: Pick a specific exhibit at a museum you visited this semester, not including the one for which you made a game already. Now consider that the museum approaches you to create a game for use in a summer camp, where students engage with thematic ideas (history, science, etc.) in a day camp format. Describe the process by which you would deliver such a game. Note that we are not asking you to design the game, but to describe the design process for the game.

1b: Explain, in detail and drawing on your personal experience, why you believe the answer to 1a to be true. 

2: What was the relationship between the course structure and your learning this semester? Be sure to specifically address peer evaluation of prototypes and essays as well as the achievement system. You might also consider what elements you would change or add. 

3: How many hours did you spend on this class each week? Did this change during the semester, and why? 

4: Excluding yourself, who did an outstanding job this semester? Consider both their contributions to the class and their product.

The first was designed to get them thinking about what they did this semester and consider what they have learned. Part 1b is more important to the learning process, as this requires metacognition. The answers to these questions were all similarl: the students would do it the same way we did it this semester. While the students were able to summarize all the various steps we had taken, and they appeared to have faith that this was a good way to proceed, I was a little disappointed that there was not more critical discussion of what worked and what didn't work. In particular, only one student nailed the importance of research for serious game design. That is, to make a game that matches a museum theme, the designer needs to have a good understanding of that theme.

The second question was designed to get specific feedback on course structures. Almost every student mentioned that they liked the achievement system but that they would have liked some deadlines in order to help them manage their time more effectively. In particular, they wanted the Scholar achievement moved earlier in the semester to provide more foundation for their work. If we had one more meeting, I would have pointed out to them that this course of action was available to them all semester, and that it was their procrastination that pushed this to the end of the semester. I hope that they recognize this and that it was a good learning experience, but in the future, I would rather prevent the students from having to learn this lesson, so they can focus on course ideas instead. One of the best ideas shared by a student was to have achievements for playing more than one game in a genre, rather than just having them for playing different genres.

The third and fourth questions were Ron's ideas, and I am glad we added them. I assume that their answers were honest, since there was nothing to be gained at this point—remember that this "exam" did not contribute to the students' grades at all. According to their answers, about three students spent the expected nine hours per week on the course, and the rest spent less, some much less. This is interesting, as there was some complaining during the semester about how long it takes to learn and play some of the games in the canon. Several students admitted to spending no time on the course during the first third, as we expected. I think this shows that the level of rigor we expected of them was appropriate, even only some of them really rose to the challenge, and it indicates that students may do better with more scaffolding.

As for the fourth question, it reflected a similar phenomenon to the ISM's visit: the students seemed to value highly their peers who were most vocal, even in cases where—in my expert opinion—they had nothing to say. I take this as an indication that we needed more opportunity for the students to talk about specific design details: this would prevent the generalities and platitudes that students are so willing to share.

Next time

With an ounce of luck and a grant, I should be able to teach a similar course next Fall. With this in mind, here is a summary of what I would consider doing differently next time.

• I need to plan on a class full of partially-interested novices rather than hope for a few with passion and background knowledge. Should such students show up, I'm sure I can capitalize on it, but I need to make sure I'm not quite so surprised when initial critical analyses are all about kids' games.
• The students should do more smaller designs as a way to explore the design space and get used to the designer's mindset. In particular, I should have them make and share one-page design documents on shared themes, a trick I picked up during the semester from a conversation with Lucas Blair at Meaningful Play.
• Keep the achievement system, but add deadlines. Scott Nicholson—who I also met at Meaningful Play—recommended a system in which students have a few dates during the semester by which they must complete a threshold number of achievements in order to earn a certain grade. For example, completing three achievements by each of the dates would earn an A. I will likely do something similar, probably sending Scott a friendly request for more details as the hypothetical future course gets closer.
• Decompose and decouple achievements. Rather than have one that takes weeks and five designer games to complete, make this five different achievements. My colleague Evan Snider has explained to me how he chains together smaller achievements into "quests," which is something I might consider, although I do worry about taking the metaphor too far.
• Only use positive grading, not punitive grading. Our original model had grade penalties based on student infraction of policies, but this implies that students have points to lose. I prefer a model where students cannot lose credit, only gain it.
• Push them harder, individually if possible. The best products, and the best learning, came about when Ron and I contacted students individually to push them to do better. Each rose to the challenge.
• Include a request for feedback as part of the final exam. Since the university moved to digital course evaluations, their usefulness has plummeted. Most students do only perfunctory evaluations and do not provide the kind of useful feedback as when done on paper and in class. The comments from the exam were much more useful to me than the course evaluations, even though they ostensibly asked the same questions.