Playing the Mob Programming RPG in CS222

I did something new in Monday's meetings of CS222 Advanced Programming: I had my students play Willem Larson's Mob Programming RPG. A colleague in the games industry turned me on to the idea of mob programming by way of the Mob Mentality Show.  More specifically, the RPG came up in this interview with Joe Justice about mobbing at Tesla. It was the first Mob Mentality Show that I listened to, and it was interesting to jump in with a discussion of how mobbing, which is usually associated with software development, was used for hardware development around electric cars.

Willem Larson was inspired to make a free PbtA-style RPG to teach Mob Programming. Now that scratches a lot of my itches! I encountered a few concerns while reading the rules, including the fact that the introductory script is very long: the amount of ideas presented orally to the team far exceeds what they can remember, especially when you consider that they also are handed three different role descriptions to process. There are certain parts of the instructions that appear incomplete or ambiguous, but rather than start by copy-editing and opening a pull request, I figured I should present the game as-is. Shuhari and all that.

I printed off all the playbooks, reviewed the rules, grabbed the handful of other required supplies, and set off to CS222 Section 1. I figured it was not feasible to plan on everyone sitting in a line, which is recommended in the rules, since I never know how many students are going to show up. I ended up using mobti.me, the first online mob timer I found when searching online for a web-based one. The rules recommend the people on the left and right of the Driver and Navigator help those two manage their XP acquisition, and so to adapt this to haphazard seating, I added the roles "Driver-Helper" and "Navigator-Helper" to the rotated roles.

We used FizzBuzz as our example. Before both classes and in accordance with the game recommendations, I set up separate starter projects. In both cases, I gave a failing unit test as follows:

@Test
public void testFizzBuzz() {
  FizzBuzz fizzBuzz = new FizzBuzz();
  String actual = fizzBuzz.valueOf(1);
  Assertions.assertEquals("1", actual);
}

Of course, without any implementation of FizzBuzz, the test does not compile and hence is "Red." The mobs then started with a failing unit test.

Section 1 proved to be a trial run. I presented the rules as written, answered all the questions, and let them go. Unfortunately, three problems quickly showed up. One was that I had entered the roles in an unhelpful order, which meant that players did not move as intended from Driver to Navigator. After seeing this go poorly, I understood why this sequence is preferable. The other problem was that I did not have the sound configured properly for the online timer. I assumed the problem was that the room's speakers were off (they were) or that my laptop volume was too low (it was). After doing this, though, we still didn't get sound, so I ended up running a parallel timer on my phone. The fact is I had forgotten that the audio buzzer is a configurable option in mobti.me, not a default. Finally, I also noticed that people started putting their badges up, but no one had called out the fact that they had earned XP. Yes, I had announced that when someone earns XP, they should say why; no, the students had no memory of this. It actually took me a while to recognize this error mode, and that put me in a sticky situation: should I interrupt and re-explain the rules? And if I interrupted, should I invalidate or keep the existing badges? I ended up taking the laissez faire approach, although in retrospect, I wish I hadn't.

By the end of one rotation, they had about six or eight badges on the wall, but none of them were authentically earned since no one called out their XP gains; crucially, this means that any student didn't know what valuable thing any other student had just done. Also, throughout the exercise, very few people in the mob participated. The Navigators never really did anything except, occasionally, ask for ideas. They mostly watched the Driver type in reaction to one or two voices in the mob as they talked through a proposed solution at the code level. When these people became Drivers, they just implemented their visions without any Navigator intervention, and as Navigators, they never interacted with the mob.

After the rotation, we had about fifteen minutes left in the class period, so we switched into discussion mode. I explained how I had come across the game, and that I had never played it, nor had I any personal experience in mob programming. I also admitted up front that I noticed them missing that important rule about claiming XP out loud, and I had held back on interrupting their flow. They seemed happy with my candor, and one student even thanked me afterward for running the game. When I asked for their comments, a student pointed out that it didn't feel right, that they didn't think there was a single point of failure, but that at the same time, the ideas weren't going smoothly from the mob to the driver. I pointed out that, if this was what was going wrong, doesn't that mean the Navigator was the single point of failure consistently? After a moment's hesitation, I think the class agreed, although the unwillingness to hold anyone accountable was palpable.

This led to good but brief discussion of the role of the Driver and the Navigator. I pointed out how what they learned in CS120 was, very likely, an antipattern: the Driver works and the Navigator watches. (Maybe we should call this the "Napigator Antipattern".) I pointed out how different the form is when the Navigator has to speak their ideas out loud in such a way that the Driver has to listen and translate it into code. Someone suggested that changing Navigators means that there is not a unified direction for the mob; I made the counterpoint that only changing the Navigator can mean that everyone understands the direction of the mob. Again, I got the sense that they understood it was more complex than it seemed at first blush, but it's hard to say if these particular ideas stuck with the students or not. (In writing this blog post, I did a little research, and it seems some people call it The Strong Technique of the Driver/Navigator pattern when the driver can only do what the navigator suggests. This gives me some nomenclature and resources for talking about how I expect my teams, and the teams in prerequisite courses, to work.)

I was more confident to start the game with Section 2. The same number of students showed up—eight—although more of them late for this section. (Tardiness always irritates me, but cases like this are the most egregious, when someone comes in and, while I'm speaking, loudly asks a friend, "What are we doing?") This time, when explaining the game, I made it clear and explicit, repeating the point that you can only mark the XP if you tell the mob that you earned it. During the pre-game Q&A, students twice asked questions about how they know if they have earned the XP, for cases such as "Listening on the edge of your seat." The answer was easy, and I repeated it verbatim from the instructions: it's up to you to decide.

This section launched right into action, and they clearly were enjoying themselves. Occasionally, someone would say with a smile, "I am listening on the edge of my seat!" There was much more clarity about the roles, as the Drivers acknowledged getting XP for ignoring directions from the mob that didn't come through the navigator, typed things they disagreed with, and asked clarifying questions. Indeed, I think it was in large part the diligent driving that held the Navigators to high standards. Of course, the having the right sequence in place helped too, with the Drivers moving into the Navigator roles before returning to the Mob. For example, a Driver who thought they were typing toward a dead end could, as Navigator, direct the next Driver how to fix it.

There was a lot more excitement and energy in the second section, with people moving around, taping up badges, jotting ideas on the whiteboards, and laughing as they called out their XP earnings. One emergent problem though was that, as roles changed, the new people would wait for things to settle down before starting the next round. That is, a round would end, conversation would stop, and folks would be moving around, cutting out badges, looking at higher level roles, and so on. Even though I encouraged them to just get started, stating that the mob was simply mobbing, the students consistently wanted to wait until everyone was settled and facing front before starting the timer. I wonder if this was because they wanted it to look like a class discussion rather than, well, a mob. This had the effect of significantly interrupting the flow,  killing the energy. It also had the practical effect of using up all our time: so much time was wasted during role rotation that we only had about two minutes for discussion at the end. Fortunately, the students stayed about two minutes after to wrap up the discussion. They also were very helpful in cleaning up the room, for which I am grateful, since we had a colloquium getting set up in the same room for the next hour.

Still, Section 2 did quite well, and with a final score of 31.

During our brief discussion, the students commented that one of the biggest problems with the game is that they spent most of their time looking at their role sheets to figure out how to get points rather than solving the problem before them. This is a serious game design problem, by which I mean it is a serious problem and it is a problem with a serious game. The mechanics of the game here are working against, rather than with, the learning objectives. Indeed, it tempts me to mold what Larsen has so generously provided from his intuition into something that is more rooted in theories of teaching and learning. 

Another student made a good observation about a hesitancy to interrupt someone else's flow. They said that in their final project team, they often felt like others knew what they wanted to do, and so the inclination was to not interrupt even if the direction was not understood. "I don't want to interrupt, and I'll figure it out later," is how the reasoning goes. Spoiler alert: later never came. This allowed us, as the previous section has done, to discuss the Navigator and the fascinating wisdom of rotating that role through the whole team. If everyone on the team can take a turn explaining where the team is going, then everyone understands it, and that means everyone is rowing in the same direction.

This leads me to the most startling observation of the exercise: my students did not come near to solving the problem. I've heard reports and studies about FizzBuzz is still an effective filter since most people who apply for a programming job cannot solve it. Surely, I thought in my naïveté, my students would be able to deploy TDD to come up with at least most of a solution. It was not the case.

Again, we must begin with a disclaimer that they were working under the distractions of the game. They were told, specifically, to try to get as many points as possible. In theory, the points should be earned while moving toward a solution to FizzBuzz, but especially in Section 2, many points were earned by doing the corresponding actions in a vacuous or even antiproductive manner. It's possible that their working memories were completely overloaded in thinking about the game rather than the problem. 

That said, I don't think this explains away all the different error modes that I observed, which suggested more fundamental problems. Both sections showed an alarming lack of rigor around TDD, a practice that is supposed to be a focal point of our work this semester. They are all supposed to be doing Beck-style Test-Driven Development throughout their nine-week projects. Judging from their submitted artifacts, many appear to be doing so. During the game, though, nobody wrote out a plan of test cases that would lead to success: they all started in with coding without any real plan. Indeed, despite the instructions specifically saying that Mob Programming should feel like a bulldozer rather than a race car, the game fought against this: the only way to get points fast is to work fast, so my students throw discipline out the window. I only noted one instance in which a student called out for a failing test to be written before production code. Otherwise, it was a kind of erratic hopping between writing tests and writing production code, with refactoring never really touched. Indeed, in one of the sections, a student said, out loud, in front of me, "Let's just skip refactoring and do it later." Spoiler alert: later never came. However, the horror of saying such a thing in front of the professor who has been proclaiming the importance of refactoring for 13 weeks also points to something subtle and important: I believe I was witnessing authentic student practice. That is, even though I was sitting in the back row, saying nothing, I got the sense that they were not performing any show for my benefit. I was seeing how they really work.

Common definitions of the Driver/Navigator pattern specify that the Navigator should be describing an approach, and the driver turns that into code. In the RPG, it is presented more bluntly: "... [T]he Driver’s job is to type what the Navigator instructs them to type. The Navigator’s job is to sift the ideas of the mob and instruct the Driver what to type." My students took this very literally: the Navigators told the Drivers what to type, keystroke by keystroke. Indeed, one student typed in exactly what the Navigator said, even though it was clearly a syntax error, and then claimed an XP for typing something they disagreed with. The clear problem with this literal interpretation is that it keeps all of the discourse at the level of keystrokes rather than ideas. No wonder they could not solve the problem if they were trying to do so at the level of parentheses! In other cases, however, it wasn't clear if students were being overly literal or if they really just couldn't translate an implementation idea to code. For example, one student said, "We need an 'if' statement: if 'i' is 3." The Driver then keyed in "if (i is 3)". It took over ten seconds for the Navigator to explain that what they meant was "equals equals." Notice here that they did not mean "equals equals" at all. They meant "i is 3," but that turns into "i==3" in languages like Java. This was a breakdown caused by the inability of the students to have discourse about the code at a reasonable level. Not knowing the names for punctuation symbols used throughout programming languages didn't help them either. It seems like a case where the masters have been lecturing them about the Parthenon, but they should have been lecturing about the optative.

I had hoped that my initial failing unit test would push the students to see a clear model-view separation in the problem, but that didn't happen either. I thought it would be obvious from our various class discussions that there is a model that computes the answer and a view that prints it out. The first group, after getting the initial test to pass, agreed to change the test case itself so that FizzBuzz's valueOf method should return an integer. I am still uncertain how they thought this would help. Truly, I cannot get my mind into a state where I can see this as useful, neither the specific change nor the idea that the test I would give them is somehow flawed at the level of static typing. Their implementation of valueOf just ended up being to return the passed-in value. They then added a second method, called isDivisible, that took an int and returned the String that should be printed for it. If you have any experience programming, this should boggle your mind, yet my students happily moved forward with this. Suffice it to say that the rest of their code made no real progress toward a solution, not in any deliberate, structured, or sensible way. If they were ever to get to a solution, it would have been essentially by chance.

In the second section, they started more sensibly, getting the first case to pass in the obvious way and then adding a test for the third word being "Fizz." In doing so, they (seemingly unknowingly) removed the code that solved the first test. However, when they went to run the unit tests, they only ran the new one, the case for three, which passed. I saw a clear regression defect, but they did not, as they immediately went on to writing the code for the "Buzz" case—without having run all the tests, without refactoring, and without writing a failing test first. At some point, several minutes later, someone in the mob saw that only one test was being run, and they instructed the Navigator to ensure all the tests were run. This, then, caught the regression. I suspect that the student who noted the oversight would have noticed it earlier had they not been face-down in role sheets. Of course, in a semester where we learned "Always run all the tests" on the second day of class, I hoped that this knowledge would be more widely distributed, but this seems not to be the case.

Perhaps the most important thing I gained from the Mob Programming RPG were the insights into how my students think and work. Techniques that I've shown them and required on their projects for months are clearly not part of their normal operating mode. That suggests that there really are a few anchor people who are ensuring the right things get submitted on a team, but that this is not being done in a way that the ideas are being disseminated. During the game, I saw how some strong voices could derail the whole team, moving in directions that were not just contrary to our principles, but not even syntactically valid. It makes me want to pull Mob Programming earlier into the semester, although I recognize a logistical problem with that: we're at a point now where about half the class does not show up and a few students have withdrawn. What started as 20-some students in the room has turned into about eight who show up regularly. We could not possibly finish a whole mob cycle with 20 in one class session, which means I would have to split it over multiple days and then deal with absences from the rotation. Mobti.me would not support this, but of course, I'm already imagining whipping up a more robust replacement in Flutter. I need to think about how to deal with this since the benefits of the exercise certainly seem worth the team, even if just for the discussion and my observation of real student practice.

Making my own Mob Programming RPG sounds like a great scholarly endeavor. I think using PbtA playbooks is a clever idea, but the mental load on my students was clearly too much. I am sure that professionals would approach the game differently than undergraduates. Anyone with a few years under their belts would be more adept at programming, communication, and coordination. Crucially, they would also be more familiar with holding each other accountable. My students are trained, well before they get to me, that they should never criticize their peers for anything they do or say. That is completely self-destructive for collaborative software development, of course, but my pointing that out does not change a cultural phenomenon. Students are also swimming in a sea of points, and so if they see that a particular action earns points, they will pursue it regardless of prudence. A good example is in Larsen's Rear Admiral role, in which a player can earn XP for speaking quietly in the Navigator's ear. Twice in the second section, students walked up to the Navigator, whispered something, and walked away, claiming XP. Yet, although what was whispered was actually good advice at the time, they did not regard whether or not the Navigator was in a position to hear or understand, and what the Rear Admiral had to say ought to have just been said aloud as part of the mob. My own hypothetical version of a Mob Programming RPG could capture not just particular learning outcomes of my curriculum, but it could also account for the spaces, headcounts, and population that I deal with regularly.

Looking to the immediate future, I would like both sections of the class to have the opportunity to deploy mobbing to actually solve the FizzBuzz problem. I plan to use another class period this week to repeat the exercise but without the game. That is, we can just do "regular mobbing", using the mob timer with a Driver and Navigator, and see if we can make progress. We will have to start again from scratch since the code from the end of each session is not worth saving. I am concerned that Section 1 may not have seen enough of the practical execution of mobbing, but I think I will try to keep the two exercises in sync rather than allow the two sections to diverge. I will need to start by pointing out the difference between the Navigator telling the Driver what to type vs. describing an approach that the Driver turns into code. Also, I will tighten up role transitions by ensuring that everyone knows the sequence of roles, either by finding a wireless keyboard to pass down a literal line or by copying the name sequence to the board so it's not hidden in a browser window. I am tempted to remind them about Red-Green-Refactor and model-view separation; my hope that they simply remember these may be in vain.

I know that we are all creatures of habit, and I do not expect that any team will make radical changes to their practices in the last two weeks of a nine week project. However, I hope that these exercises will get them thinking critically about their team experience, giving them something concrete to compare it to, and give them a vision for what might be possible. The role of this course in the curriculum is to prepare students for team- and project-oriented upper-division courses, and so I am eager to see how I can leverage this exercise myself as I start thinking about Fall.

This story continues in the next post.

The Launch of Traveler's Notebook: Monster Tales

I am pleased to announce the release of my Spring Studio team's game, Traveler's Notebook: Monster Tales. It is a two-player, narrative-rich game in which you encounter monsters from various world cultures. The game grew out of my consideration of learning through games like Tales of the Arabian Nights, which I wrote about back in January, and which led to the team's identification of their goals and early design work. I plan to write a more thorough retrospective of the semester once it's all behind us. For now, check out the game at travelersnotebookgame.com!

If you're in the vicinity of Ball State University, members of the team will be at the Immersive Learning Showcase on Monday, 4-6PM in Cardinal Hall at the Student Center. They would be glad to give you a demo and talk about their experience.

Symposium on Games in Academia Redux

I wrote about planning the Symposium on Games in Academia about two weeks ago. The Symposium took place last week, and I would definitely call it a success. Approximately 50 attendees came to the symposium to hear fourteen speakers talk about the intersection of games and higher education.

The crowd before the afternoon coffee break

I had the opportunity to give the opening remarks, so I talked about some of the inherent problems of games in education: that we know that games can teach, but we don't really know what they teach, or how they teach, or in some cases, what they even are. It's a perfect storm for future research! I shared one of my favorite stories from my own work, how students playtesting a Morgan's Raid paper prototype concluded that George Ellsworth was a wizard: it wasn't the designer's intent that affected what they learned but the individual constructive and social play experience.

We had a variety of door prizes donated by Wizard's Keep, Game Stop at the Muncie Mall, and Aw Yeah Comics, as well as other local philanthropists. One of my favorite talks of the day was given by Joel Bozell, an acquaintance of mine who is immortalized as Brian van Hoose in Knights of the Dinner Table. He spoke about the Muncie and Ball State connections of the long-running comic.

Joel Bozell talks about Knights of the Dinner Table

Another personal highlight was Carisa Lovell's presentation about her experience as an English major on the Collaboration Station team. Most of our discussions have revolved around shared interests and experiences, but in this presentation, she eloquently tied the studio experience to her major.

Carisa Lovell demonstrates Collaboration Station

I took a leadership role in planning the event, and I am grateful for the assistance of Eva Grouling Snider, Jennifer Grouling, and Scott Reinke, who were instrumental in making it a success. My department also generously supported the event both with refreshments and the help of administrative staff. I should mention that Eva designed this clever little fellow, who I think needs to become a mascot for the Serious Games Knowledge Group.

Once we get the attendee list transcribed, I will be sending out a post-symposium survey to gather some assessment data. Some people expressed interest in running another event in Spring; while I am sure there are more people who would be willing to speak, I also worry about gathering too much low-hanging fruit too quickly. Others have mentioned a desire to have a bigger, more formal event next year, perhaps with outside speakers. That, of course, requires a budget, which we don't have. I am hoping that the post-symposium survey will get us a better picture of what our audience thinks, and we can use this to do some planning and perhaps some politicking.

Symposium on Games in Academia

I founded the Serious Games Knowledge Group at Ball State University about two years ago, I think it was October 2013. The premise of the group was to gather faculty from across campus who are interested in games as learning tools, defined quite broadly. From a small core, we grew to about a dozen faculty and staff. The university's concept of a knowledge group, as I understand it, is to bring faculty together to produce scholarship and, potentially, grant proposals. Our group has met roughly once a month over lunch, where we share our stories about games we are studying, games we are developing, conferences we have attended, and articles we are reading. There is great diversity in our group, but this diversity has meant that we have not collaborated with any shared purpose: while the sharing of our individual work is enriching and enlightening, it has not yielded any scholarly artifact that we can point to as our own. In some ways, we have operated more like special interest group than a knowledge group.

This came up in our first lunch, and we agreed to do something about it. This Friday, the Serious Games Knowledge Group is sponsoring a Symposium on Games in Academia, 2:00–5:00PM in BL104.

https://sites.google.com/site/symposiumongames/

We put out a call for presentations at the end of September, and I was pleased to receive fifteen high-quality proposals—enough that we had to reserve a bigger space! The presentations represent an even wider variety of interests than the Knowledge Group itself, and presenters include tenure-track faculty, contract faculty, faculty who teach at the Indiana Academy, university staff, and undergraduate students. The event will truly be a celebration of our diversity of interests, a chance to share our stories in a more formal and thoughtful environment. Refreshments are being sponsored by my department, and there is a wide selection of door prizes that have been donated by area companies and individuals.

The event is free and open to the public, so if you're in Muncie this Friday, please feel free to come join us. Eva designed this flier to advertise the event, and you are welcome to post it and share it. I'll make some time to post a follow-up on my blog after the event.

Spring 2015 Game Studio project announcement: Collaboration Station

It's been a busy time here, and it seems I missed January entirely on my blog. One of the reasons for the tight schedule is that I am once again mentoring a major immersive learning project, working with an multidisciplinary team of ten undergraduates to create an original educational game. We're partnering with the Children's Museum of Indianapolis, who are helping to provide feedback as we work on Collaboration Station, a cooperative game about the International Space Station.

Our social media presence just went live, so please visit the team's blog and check out our Twitter feed. Right now, there's just an introduction to the project, but you can expect more content in the coming weeks.

Quiz Games and Learning Taxonomies

A few days ago, I posted about the SOLO Taxonomy and my desire to incorporate its ideas into educational game design. Yesterday's serious game design class provided just such an opportunity.

This week was my students' first opportunity to present game concepts. Over the coming weeks, they will be presenting additional concepts as well as multiple iterations of prototypes, as I described in my course planning post. This was their first real opportunity to show something original, and most of them chose to make games based around the International Space Station, a topic of particular interest to the class' partner, The Children's Museum of Indianapolis. The designs varied in genre, scope, and clarity, as to be expected from a first pass. One particular design was essentially a series of questions, where each question was worth a point. That is, this design was essentially a quiz, with no interesting or relevant decisions being made by the player.

I told the students how, in general, I don't have much interest in quiz-style games. Coincidentally, they had just read The Education Arcade whitepaper, “Moving Learning Games Forward,” and I was able to draw on some of the ideas in that whitepaper to justify my indifference. Looking at quiz-style games from the perspective of SOLO, they clearly fall into either the unistructural or multistructural levels of the taxonomy. That is, either the player is recalling a single idea (identify or name) or sets of ideas (list, enumerate). SOLO makes it clear that these games exist at the lower level of learning activity. By contrast, a game that requires analysis or application of domain knowledge would place the learning activity at the relational level, and a game that requires invention or theory-building would be at the extended abstract level. These levels inherently reflect what Burgun calls endogenously meaningful ambiguous decision-making, which I find to be a useful heuristic for evaluating whether or not a game concept will be worth prototyping.

It seems to me that identifying quiz games as low-level learning is just as easy with Bloom's Taxonomy of the Cognitive Domain: quiz games are usually situated at the Remember or possibly the Understand level, the very lowest levels of the taxonomy. The applicability of SOLO vs. Bloom seems to get murkier once when adds meaningful choices to the game. This gives me a direction in which to consider these taxonomies as I move forward with this semester's serious game design efforts.

Reflection on The Spring Game Studio

Introduction

In the Spring semester, I was the mentor for an immersive learning project through which a multidisciplinary team of students developed an original educational video game. This post is a reflection on the semester. This post begins with a description of the studio's charter and its performance. With this background, I describe some specifics of my mentoring, focusing on the formats of the iteration and summative (end-of-semester) retrospectives. I close with some of my own personal reflections and a few acknowledgements.

I have peppered some pictures throughout. Some are related to the surrounding text, and some are not. However, I have found that when I need to prepare a presentation about my work, it's often easiest for me to search my own blog posts, since this gives a good curated collection.

An Overview of The Spring Game Studio

The Spring Game Studio—having officially been named Purple Maize Productions—had two formal meetings during finals week. On Monday, the team delivered Children of the Sun, their educational video game based on the Middle Mississippians, to our community partners at the Indiana State Museum. This was to mark the end of the team's production focus, and on Friday, we held our last formal meeting: a semester retrospective focused on what the team learned during their fifteen weeks together.

The team was able to overcome significant human, technical, and design barriers in order to create this game. On the human side, the team was working in a shared space but without a shared time. I had to decide during recruitment whether my priority was to pick the best student composition or those who could work at the best times. I chose the former, and so my individually-talented students had to work around the lack of consistent, whole-team synchronous communication. We used Google Groups and Google Drive for digital communication and, in physical space, several shared whiteboards. On the technical side, we were developing a networked iPad game without being able to have a persistent server: we ended up using Bonjour to discover local machines,  with one having as both server and client. None of the students had network game development experience, and most had no game development experience at all—much less educational game development. Finally, regarding design, the team was working within a rare combination of constraints: an educational game, about Middle Mississippian culture, for upper-elementary school students, that they would play only once, on iPads, together, in a formal educational program about archaeology.

Start screen

In the village, giving orders to 300 villagers.
The layout is based on archaeological evidence from Angel Mounds.

Planning, Reviews, and Retrospectives

In the past, I have always aligned the three Scrum-style meetings of Planning, Review, and Retrospective to align with weekdays: Planning on a Monday, Review and Retrospective two Friday's later, for a two week sprint. Unfortunately, the only time the Spring Studio reliably had to meet as a whole was Tuesday mornings, so modifications were in order. We held Planning meetings at the beginning of a two week sprint, and at the end, we held short Review meetings with whoever could be available on the Monday afternoon ending the sprint. The Retrospectives were not aligned with the end of sprints, instead taking place on Tuesdays that were not planning meetings. We did not have a retrospective in the middle of the first sprint, and another was cancelled due to my planning error; hence, we had only three iteration retrospectives during the semester.

I articulated and prioritized user stories based on my understanding of the team, the project, and the community partner. The Sprint Planning meetings went in two phases: planning poker to assign story points, then the team's breaking down the stories into tasks. I often skip planning poker in a three credit-hour studio course, but since these students were earning six, we had time to include it. I'm glad we did, since the conversations that arose from this activity helped the team to see what needed to be done; this was especially true later in the semester as the team developed a better vocabulary and better understanding of the game, the tools, and our processes.

Promotional art for Children of the Sun Infinite.

I have been in the habit of using a Start-Stop-Continue format for iteration retrospectives, and it has served my teams well. However, I read Kua's The Retrospective Handbook: A Guide for Agile Teams in preparation for the semester, and I decided to deploy some ideas from it. For this semester's iteration retrospectives, I wrote four categories on the board:

• What did we do well?
• What did we learn?
• What should we do differently?
• What still puzzles us?

At the beginning of each retrospective, I read the prime directive and then asked the team to write down contributions on sticky notes for each of these categories while I did the same. After ten or fifteen minutes, we came to the board and posted them all, and I encouraged the team to read each others' to find clusters. Then, I led the team in a left-to-right reading and discussion of the notes, looking in particular for potential action items, which were recorded on a separate board. After going through all of the items, we culled the list of potential action items into just those items that the team agreed were important. Individuals committed to these, and the meeting was adjourned with another reading of the prime directive.

We were rarely all in the same place at the same time, but this artist's rendering shows what it might have looked like.

One thing I missed in the early retrospectives—honestly, because I forgot—was to triage troublesome notes into a separate column. There were a few cases where we probably had too deep a discussion about singular notes, and this broke our stride: we should have postponed that discussion, since the rest of the exercise was going to reveal more details about it anyway.

There's a good story here about community learning. In the early sprint planning meetings, I served my usual role of directing both the planning poker and task articulation processes. However, it was revealed in a retrospective meeting that my articulation of a particular UX design process did not make sense to the team. (Their reasons for not seeking clarification during the planning meeting or when the miscommunication was recognized are not known to me.) One of the action items from that retrospective meeting was that one team member was responsible for ensuring that when I wrote a task, the team understood what it meant. However, the more I thought about this, I realized that a better approach would be to give the articulation more directly to the students. Surely, I had modeled the expected behavior well enough for them to do no worse than I did! At the next planning meeting, after planning poker, I told them it was up to them to take charge, and I sat down. To my great delight, two of them jumped up, ready to grab the sharpie and sticky notes. They did a great job, and I don't think the team had any more of these task-articulation defects.

Sprint Planning: Student-Directed Task Articulation

The End of Production

Our original production schedule called for the game to be feature-complete four weeks before the end of the semester, allowing ample time for polish, quality assurance, evaluation of efficacy, and consideration for distribution on the App Store. The team missed this goal, as well as the revised goal of being feature-complete two weeks before the end of the semester. I set a firm production deadline of 4PM the Friday before our final meeting, which the team also missed. Work continued through the weekend, though by only one or two developers who claimed it was all clean-up. This "clean-up" required additional clean-up Monday morning before the ISM meeting. The ostensibly final version that was delivered at the meeting was revealed, in the demonstration, to have a defect that prevented the game from starting. The team was duly embarrassed and, to my embarrassment, pulled out a laptop and tried to fix the game during the meeting. As another part of the meeting continued, a small group was able to put in a kludge to get the demo working, at least partially.

One of the team members agreed to drive down to the ISM on Thursday to deliver the fixed version of the code. I became concerned about the radio silence after Monday's meeting, and on Wednesday, I posted a message to the team stating, "I trust that the changes made between Saturday and Monday have been subjected to rigorous quality assurance." This was a manipulative lie, of course. One developer (or perhaps I should say, only one developer) came in and spent several hours in the studio Wednesday, where he rooted out the actual defects and fixed them—despite his being nearly overwhelmed by his senior capstone project, which was due at the same time. I have to wonder, what would have happened if he had not done this?

Final presentation to the ISM. Notice the nifty team polo shirts!

Semester Retrospective

In the past, I have used a two-phase, divergent-then-convergent approach to semester retrospectives: the team brainstorms a list of things they learned during the semester, and then votes on the top collectively-important items. Recently, however, I read "Groupthink: The Brainstorming Myth," which reminded me of the dangers of purely-divergent, criticism-free brainstorming. I decided to ask the students, several days before our final meeting, to reflect personally on what they learned. In particular, I asked them to produce an artifact such as an essay or list based on this reflection, arguing that articulating their ideas would help their understanding—a classic encouragement of writing for metacognition. In hindsight, I should have asked them to send a copy to me: one student did, of his own volition, and looking at it now, I can trace his personal learning into the collective learning.

For the retrospective meeting itself, I was strongly inspired by Alistair Cockburn's recent technical report on knowledge acquisition during product development. In this report, Cockburn identifies four areas in which teams build knowledge, quoting:

1. How to work together, and indeed, whether these people can get this job done at all.
2. Where their technical ideas are flawed.
3. How much it will cost to develop.
4. Whether they are even building the right thing.

I paraphrased these four categories on one whiteboard and gave each a stack of colored sticky notes. Comparing my shorthand to Cockburn's articulations, I realize I should have used his nomenclature: it is more elegant though more verbose, but I had the room:

Four categories of knowledge acquisition.

I drew a timeline across the adjacent whiteboard, highlighting significant events from the semester, including: the three week orientation and design workshop; the six subsequent two-week sprints; our presentation at the Building Better Communities Showcase, the Butler Undergraduate Research Conference, the Ball State University Student Symposium, and Science Day at the Fairgrounds; and our meetings with the ISM. The team reviewed the timeline and agreed that I had all the major events in the right places.

After explaining the four categories, I invited the team members to translate their individual learning (from their personal reflections) into collective learning, to choose a category and write down something the team learned, and to mark it on the timeline. They seemed a bit hesitant, and when I asked if they wanted me to start, they agreed. I picked up a blue sticky note and wrote, "Dr. G cannot be here in the studio all the time," explaining that I learned that I could not be there with the team as often as I wanted to. Although I knew this, cognitively, from the start of the semester, its implications didn't really hit me until about a third of the way through production. They agreed that this was a good estimate of when the team recognized that my interventions would be necessarily periodic, and so I stuck the note there and sat down. (In hindsight, I should have written one of the implications of this, namely, that I wouldn't know what people were working on.) They still seemed hesitant, so I stood back up and grabbed an orange note, marking on it that it we were making a game about the Middle Mississippians; I actually knew this before the semester started, but it was announced to the team at the end of the first week of the semester, so that's where it went.

After this, team members began coming forward and adding to the timeline. However, to my dismay, one of the first people to approach the board, grabbed a pad, and then sat down in a chair by the board. I held out hope that they would write quickly and get back up, but sadly, this became the pattern: people grabbed the pad they wanted, sat down to write, and then put the note on the board. In fact, after a time, people just wheeled their chairs forward and grabbed a pad. I suspect that this was a significant loss of momentum, and I regret not having moved that chair away. Several times, it was clear the student didn't really know what to write until he or she had grabbed a pad and sat down and started to talk through it. Despite this, I am glad with what the students contributed, although I had hoped for more items.

Annotated timeline from the semester retrospective

The image above demonstrates that a majority of articulated learning outcomes dealt with how the team should work together (blue). While there is a good spread throughout the semester, there is a cluster in the rightmost side, which is marked in the timeline as "today." These are items that came out of the personal reflections and collective retrospective, and they include the idea that the studio space is important, that technical debt needs to be paid off regularly, that we had a particular focus on "making" (in contrast to, say, "understanding") in part because it's what the team was explicitly recruited to do, and that it's easy to fall into the human failure mode of inventing rather than researching.

Here are a few of my personal highlights from the timeline. Clustered at the middle and end of the penultimate sprint are "Test Early and Often," "Playtests Reveal Defects," "SRP," and "The importance of Clean Code Principles." This is really where the team came together and recognized the value of what I had been preaching from the beginning: it is important to have an iterative design process that incorporated real end users, and having a solid technical foundation is necessary for being able to adapt to these changes. Unfortunately, by this point, the team was locked into their game design and software design, so there were not many significant design changes even where it would have been beneficial. However, it's important here to remember that ours are academic risks, and the fact the team learned these ideas before the end of the semester is a great win for studio-based immersive learning.

The team had a booth at Science Day at the Fairgrounds

I was disappointed that the team misconstrued the cutting of features as a team failure. Early in the semester, as I described in an earlier post, the team defined an ambitious set of features for the game. Of course, many of these were not included in the final game; more specifically, the features that were included were those with the highest priority based on our shared understanding of technical feasibility, the potential players, and the community partners. However, during the retrospective, the team described this outcome as a failure, as if cutting these features was bad or a sign of failure. It is good that the students recognized that some of these features revealed important cultural aspects of the Middle Mississippians, and that cutting these meant a lost opportunity; however, I think they failed to understand that this was normal in game development. All those features we described at the beginning of the semester were risks—unknowns. We never knew if they would actually teach what we wanted them to teach. By having a smaller, tighter game, I would argue that we have a better game. However, for the retrospective, I was more interested in seeing what they learned than in using the moment to teach any more about game development processes, so I left it alone.

Four team members at the 2013 Ball State University Student Symposium

I particularly wanted to push the students to consider the nature of commitment in a team where sprints were not completed and there was very little accountability for following the methodology. (As examples, the team "committed" to holding code reviews, but none were held; the team "committed" to Clean Code, but a majority of the code is in violation of these standards.) When I brought this up, I was surprised at the dissension. Some of them clearly understood my question: that the team had made commitments to each other and to me, and had not fulfilled them, and that this pattern became status quo—so what was learned? Others, however, argued that there was no problem, because the team was composed of amateurs, did not know how to estimate, and set their own goals: that is, in my nomenclature, they argued that these were not commitments at all despite being called them. I have as much trouble accepting this now as I did when it came up, in part because when I told them they were committing to user stories, I meant "committing" in the strict sense, not in a relativist sense. It became clear that this was something of a religious issue for some team members, and the best I could do end this debate was to suggest that the team learned, that day, that they still didn't agree on what the word "commitment" meant. They accepted this, but when I reviewed the board later, I was upset at how one of the students had articulated the outcome: the card reads, "Commitment is a fuzzy word." I suggest that it is not, and rather that the students have a fuzzy understanding of commitment, since they have so little experience with it. This raises a puzzling question for mentoring future projects: how will I identify and deal with future team members who don't share my definition? 

Post-Retrospective

After this group exercise, I asked the students to write personal essays based in response to a prompt: if they were put in charge of a similarly-skilled team to make an educational game for the ISM, how would they do it? I was surprised that most of the essays assumed that the team would be undergraduates working for credit during a semester, when this was neither in the prompt nor my intention. The essays revealed a consensus that iterative development should be beneficial, and that getting digital prototypes into the hands of players and sponsors earlier is better.

Testing on the iPads

Several pushed for a longer production schedule, at least two semesters, so there would be more time to incorporate feedback from playtesting and the community partner, as well as to recover from design mistakes. It is notable that several of the students had been involved in two-semester Computer Science senior capstone projects, so they were not just guessing that more time would help—they had seen it for themselves. The Spring Studio had been designed as a six-credit experience, which I tried in part so that they would have time to revise their designs. However, after these fifteen weeks together, I think the students are right: the issue may have less to do with hours-per-day and more to do with calendar time. It reminds me of my frustrations with teaching five-week summer courses in programming: one may have the same number of contact hours as a fifteen-week semester, but there's just not enough calendar days to develop a sense of the craft.

One of the students pointed out that the team had agreed to use pair programming, at my encouragement, even though they had little experience with it and some didn't believe in it. Despite this consensus, most of them did not actually practice pair programming for the first several weeks. After significant frustration with the project, some began pair programming—lo and behold, they saw that they were more productive in pairs than individually! He asked me, then, how can a leader get the team to see the benefits of pair programming earlier, or is it the case that a team must fail first? I laughed and pointed out that I have been dealing with that exact problem for several years! It was good to see that several of the students' responses mentioned pair programming a particular practice that they would adopt in a follow-up project, which suggests to me that they collectively felt its value.

The poster shows the evolution of a UX design from sticky notes to digital prototype.
The laptop shows the final version.

The team members were novices in almost all areas of serious game design and development, and they had struggled considerably with two related concepts: prototyping and iterative development. As a whole, though they recognized that physical prototyping was faster and cheaper than digital development, the team had trouble seeing the connection between physical prototypes and digital prototypes. When they finally did produce a digital prototype, there was some confusion about how to effectively evaluate and how to integrate playtesting results into development. This was undoubtedly related to the team's troubles with iterative development, which has been challenging for every student team I have mentored. Student experiences generally come from trivial individual projects or very small scale team projects—projects with no clients besides the students and a professor. This particular team did not successfully complete any sprint: there were no sprints in which all the user stories were completed and a potentially shippable product was created. Notably, the team rejected sprint renegotiation, preferring an optimism that resulted in sprint failure. This is unique in my student teams, as I recall. However, I also stopped recommending renegotiation after they rejected it in early sprints, and I wonder if I should have come back to it again.

Testing an early paper prototype

By university regulation, I must assign grades to the students, and so I try to do so in the most fair and unobtrusive way that I can. Each student had a midsemester progress report meeting with me, during which I tried to give useful, private, individualized feedback. At the end of the project, I gave the whole team a grade based on their collective achievement, although individual grades varied based on peer commendations. Such grading is rather stressful for me, and it is also divorced from the day-to-day activity that I want to encourage. Yet, I tend to be conservative in my interventions as Grade-Giver, preferring to offer mentor's advice that the team can choose to follow or not. Indeed, the team chose not to follow my advice to either complete all the user stories or renegotiate, and this did not doom the project: the game is still acceptable to the client, despite their collective dismissal of my suggestions. I talked informally with one of the team members who happened to also be in my Fall game design colloquium—my first experiment in badge-based grading—about whether a badge-based approach would work here. What if there was a badge such as "Completed all stories in a sprint" or even "Renegotiated a failing sprint"? He agreed that it would have to be very carefully designed. The best outcome would be that the student see more clearly what kinds of behaviors I value and promote, in such a manner as might lead to a performance bonus in industry; the worst outcome would be that students lose autotelic focus on the project and start to game the system for points instead of project success.

Looking Forward

I am proud of the work of Purple Maize Productions. They overcame significant obstacles to produce an original educational game, using tools, technologies, and processes that were new to all of them. They emerged from the semester with powerful, significant reflections—evidence that the learning outcomes of the studio were met. This was the also first time that one of my student teams attempted a multiplayer game. Should future teams do the same, I have some great stories to share with them, and I will know to push the team much earlier and more forcefully toward an end-to-end prototype. Indeed, given the complexities of testing and the tight fifteen-week time limit, I may simply advise students against any networked multiplayer, sticking with hot-seat or single-player games.

To experiment with potential villager animations, the team projected the village view onto the board and traced it. Brilliant!

I intend to reuse both of the retrospective formats from the Spring Studio experience. I like how the iteration retrospectives are oriented toward concrete action items and will likely keep the format as-is. I am considering two changes to the semester retrospective format. First, I would have the students submit their personal reflections to me before the meeting. This would serve two purposes: it would demonstrate their importance to the students, since I asked to see them, and I would have more data with which to triangulate actual team learning. The other change is that, after modeling how to write a team learning statement, I would give the students a few minutes to write out their own before posting them. This should avoid the problem of individuals coming to the board without really knowing what to say, since it would already be written.

There are still some puzzles regarding Purple Maize Productions. For example, why was the team hesitant to seek my advice in my areas of expertise, or from Ronald Morris, my collaborator on this project? Why was the team not fulfilling its commitments to each other regarding retrospective action items? Why did they continue to underestimate tasks sprint after sprint? Fortunately, this studio was also used to conduct research on immersive learning, and one of my colleagues from the English department collected field notes and two interviews from the students. I look forward to digging into this data with him and seeing what stories they tell.

In reviewing the semester, including the students' final essays, I find myself wishing I could have been more proactive in helping team formation earlier in the semester. With several past teams, I have hosted game nights and pizza parties, which help people get to know each other. At the start of the Spring—the critical time for breaking the ice—I had a newborn who was not sleeping through the night, an 8AM class that had me ready for bed when some of my students were just hitting stride, and a case of the flu that had me in my bed for almost a week. I hope that I never again have to deal with these factors along with a team that does not have adequate face-to-face meeting times!

I don't know what this is, but I love coming into the studio and finding things like this.

The ideal relationship between me and teams like this one merits further reflection. How do I balance the roles of professor, mentor, coach, designer, developer, and researcher? This semester, I opted to emphasize mentor, in part because I knew I could not work with all the students due to scheduling conflicts. Children of the Sun is a fine game, but it's very much the students' design. I can't help but look at the fascinating constraints my student team was given and be a little jealous: I wish I had fifteen weeks to work on such a project! Would it be better if I ran the team more like a research group, using student labor to realize my own designs? It would be different, and I would argue that it would no longer be "immersive learning," which enshrines student-direction and faculty-mentoring in its definition. This gets quickly into issues of community partner versus client as well as teaching versus consulting. There is a significant difference between what I can do and what I can mentor students to do, but I worry sometimes that the immersive learning model is presented as if these are the same. Immersive learning is not about providing free or cheap consulting services: it is about welcoming external partners to become co-teachers and co-learners with us.

Writing this post helped me recognize a critical difference between this project and Morgan's Raid, another course where I had separate design and production courses. In Morgan's Raid, the initial prototyping was done in the Spring, and I was able to refine these over the Summer. This also led to my selection of an appropriate software architecture, choice of a stable and appropriate toolchain, and design of interventions to help students learn these. I was able to start Fall's production course by saying, "Here's what we're building, and here's how we will do it!" By contrast, the Spring Studio inherited an unrefined (indeed, unplayable) prototype from Fall's design colloquium, and they had to create the game design, choose the toolchain, and design the software architecture. I provided recommendations of course, but I delegated the decision-making authority and process to them. I do not doubt that the Spring Studio learned much more about these items than the Morgan's Raid team, but I am unconvinced that it was worth the time and attention. Given that it is logistically impossible for me to recruit a two-semester team at this point, I will take a more hands-on approach regarding the transition between the design course and production studio next time.

Speaking of which, I am pleased to report that I have received internal funding through the Provost's Immersive Learning Initiative to teach another series on serious game design and development next academic year. I will be teaching a game design colloquium in the Honors College in the Fall and leading another six-credit multidisciplinary development studio in Spring. We will be working with my friends at the Children's Museum of Indianapolis to develop an educational video game. This upcoming collaboration is one of the reasons it was worth more than a day's effort to write this post: I want to ensure that I emerge from this past academic year with a good understanding of what went well and what went poorly so that next year, we can be even more successful.

Almost everyone was there for the team photo. This is a photo of that photo.

Acknowledgements

I would be remiss if I did not acknowledge some of the people, offices, and agencies who made this project possible. Thanks go out to: Ronald Morris, who was co-mentor of the project; Ball State University's Computer Science Department, History Department, Honors College, Building Better Communities office, Provost's Immersive Learning Initiative, and Digital Corps; Motivate Our Minds; College Mentors; the Indiana State Museum; and Angel Mounds.

Reflecting on the Fall 2012 Game Design Colloquium: Expectations and Reality

Background

In Fall 2012, Ronald Morris and I team-taught an honors colloquium about serious game design. At the start of the semester, I wrote about the achievement-based grading system we intended to use and how it fits into a larger academic-year game production project. To make a long story short, we had a team of students from a variety majors, and each was expected to make an educational non-digital game for the Indiana State Museum. I took the day today to compose my reflection on the course, which is now shared with you in four parts: my expectations when we designed the course; what actually happened when we ran the course; insights gleaned from the final exam; and what I might do differently next time.

This is my third attempt at composing this reflection. The first was a collection of notes and observations that were too raw to share. The second was a narrative that attempted to map expectations, execution, and lessons learned into nice triads. This ended up being a mess, as the relationships are much more rich than simple one-to-one mappings. I mention this because it strikes me as significant to my reflective practice: in taking the time to write carefully, I have been able to understand the semester a bit better and come to peace with some things that had been bothering me.

Expectations

Students would enroll because they were interested in either the topic or generally in immersive learning experiences. By explaining all of the expectations in the course description, including an explicit statement regarding nine hours of attention per week, students would know that they needed to take the course seriously and commit themselves appropriately. Achievement-based grading would motivate students to engage with the material. Given that this was an honors colloquium—and hence only open to students in the Honors College—they would be responsible enough to set a good pace of achievements throughout the semester. The achievements leaderboard was posted in a shared location, which would lead to positive peer pressure for everyone to keep up. Some achievements rewarded students for reading and commenting on each others' essays, and this would lead to interesting discussion, deeper thinking, and better critical analysis.

Since the students have set up their own schedule based on interest and other personal commitments, their independent activity during the first third of the semester would complement the in-class discussion of game design fundamentals. This activity would then strengthen their work in the next third of the semester, which would be devoted to iterative prototyping. By the final third, the students would be focused wholly on producing their games, with all the other achievement-based background work done.

To help students with the Socializer achievement—as well as getting through the canon of genres—I would announce all of the meetings of the Society for Game Design and Development as well as the Game Days at the Muncie Public Library. Students would approach these events with some expected trepidation, but once they realized the friendliness of the community, they would continue to attend these events, both for personal pleasure and to gain exposure to new game design ideas.

At the end of the semester, we would showcase our games to the Indiana State Museum. Some of the games may be usable right away, but all would serve as media for communicating important design ideas. Given their expertise in production, the ISM could very likely produce better looking or more durable game bits. This means that clarity would be a key concern in the students' designs.

Each student would make their own game deliverable, but the designs themselves would not be graded. This could be too subjective as well as dependent upon prior experience and extracurricular digital production skills. Instead, we would give students a mark based on their following a good iterative process. As explained in the course description, we expected each design to go through several iterations, and failure to present a prototype in each iteration would result in grade reduction. Hence, the only source of grades for the student were the achievements and the iterative design process, meaning that students could really choose their grade based on their level of commitment. Because we did not need any other grades, the final exam would be a reflective exercise.

Reality

We have little information about why the students enrolled in the colloquium. Most of the students had little to no background knowledge in game design, and little passion for the topic. One student told me that she enrolled because "design" was outside of her major and experience: she identified it as an area for personal growth during her undergraduate studies. This is a great reason to take a course, and she proved herself capable during the semester. Another confided that he took the course only because of the time it met and to satisfy the Honors College's two-colloquium requirement.

For the first third of the semester, almost no achievements were completed. Ron and I gave the students some class time without us to collectively develop a plan, and this partially worked: they started holding meetings with their classmates to play the games of the games canon. Unfortunately, this happened after the transition from fundamentals to prototyping in the course schedule. Because they were completely unfamiliar with many established genres, they could not draw upon these in their own prototypes.

As we got into the prototyping phase of the semester, we started by formally scheduling students every two weeks or so. The students preferred a more ad hoc approach in which individuals would bring prototypes as they were ready. This caused a conflict with the course description, in which we intended calendar-based iterations with students presenting a new prototype each cycle. The students were right, however, that some prototypes took longer than others, especially in those cases where students completely changed topics. We agreed to let the students follow this process, entrusting them to hold each other accountable, even though it meant our iteration-based grade penalty system would become unenforceable. 

As the semester went on, we witnessed design stagnancy. Many students brought essentially the same prototype after having had a week or two to work on it, and it became clear from their discourse that there was little to no playtesting happening outside the class meetings. A few students were able to recover from this with some not-so-gentle pushing by Ron and me, ending up with good designs but only after having fought against our feedback.

At midsemester, there was still very little progress on the achievements, though with a few exceptions. Ron and I held individual conferences with the students, during which we provided some feedback on their designs and also encouraged them to enact a plan for meeting the achievements. There was a lot of smiling and promising, but very little material evidence that these meetings were worthwhile. Most of the achievements ended up being completed in the last two weeks of the semester, many appearing rushed and certainly not contributing to the students' all-but-finalized designs. Because we had these individual meetings, though, we know that it was procrastination and a failure to plan on the students' part that led to this situation, as opposed to a fundamental flaw with our expectations.

Some of the achievements simply took a long time to complete: students made progress on playing the games in the canon for several weeks before marking the achievement as complete. However, the one that was delayed the longest, which caused me the most frustration, was the Socializer. Only one student completed this achievement before the last two weeks of the semester. In their reflective essays, students acknowledged—after the fact—that it would have been better to go earlier and more often.

Near the end of the semester, we engaged in two rounds of external playtesting. The first was with Motivate Our Minds, and the second, College Mentors for Kids. Our students highly regarded these opportunities, even in those cases where the young playtesters were not quite in the target age group for the game. I would have liked to see more preparation on the part of my students: some had not defined all of the terminology in their games, and others were missing key physical pieces of their games—which, again, betrayed a lack of playtesting.

We had three in-class presentations in the last two weeks of the semester for the Scholar achievement, one each on A Theory of Fun for Game Design, The Art of Game Design: A Book of Lenses, and Homo Ludens. I was surprised by how easily the students slipped into discussion mode, engaging each other and voicing opinions. At the same time, it was disappointing at the end of the semester that some of them were still falling prey to very naive notions of game design and learning. I opted against steering these conversations, instead listening for evidence of who had integrated knowledge from their experience and who had not. I would like to claim that those in the former group made better games as well, but I think my recollection of the conversation is too colored by my post hoc opinion of their games.

The students made an impressive display of their games for the community partners. We invited representatives from the ISM to campus, and each student explained their game's theme and demonstrated gameplay. The ISM staff responded most positively to those that were simple and effectively described. This is predictable but unfortunate for the students, since the games that actually contained the best designs were not necessarily given the attention they deserved. What I mean by "best designs" are those that reflect what is known about games and learning, and not coincidentally, these were mostly the ones that went through the most revisions.

Most of the games do good design elements, and I believe that we can call this a successful outcome. Comparing them to the students' first critical analyses and initial designs, we can see a general transition from novice toward advanced beginner. However, there are also those corresponding design decisions that appear to have been left without critical analysis, but of course, learning how to ask these questions is part of learning the craft. We are currently considering which of the designs to move forward with into digital production in the Spring, and there are several that I think would make great starting points.

Final exam reflection

The final exam consisted of the following questions:

1a: Pick a specific exhibit at a museum you visited this semester, not including the one for which you made a game already. Now consider that the museum approaches you to create a game for use in a summer camp, where students engage with thematic ideas (history, science, etc.) in a day camp format. Describe the process by which you would deliver such a game. Note that we are not asking you to design the game, but to describe the design process for the game.

1b: Explain, in detail and drawing on your personal experience, why you believe the answer to 1a to be true. 

2: What was the relationship between the course structure and your learning this semester? Be sure to specifically address peer evaluation of prototypes and essays as well as the achievement system. You might also consider what elements you would change or add. 

3: How many hours did you spend on this class each week? Did this change during the semester, and why? 

4: Excluding yourself, who did an outstanding job this semester? Consider both their contributions to the class and their product.

The first was designed to get them thinking about what they did this semester and consider what they have learned. Part 1b is more important to the learning process, as this requires metacognition. The answers to these questions were all similarl: the students would do it the same way we did it this semester. While the students were able to summarize all the various steps we had taken, and they appeared to have faith that this was a good way to proceed, I was a little disappointed that there was not more critical discussion of what worked and what didn't work. In particular, only one student nailed the importance of research for serious game design. That is, to make a game that matches a museum theme, the designer needs to have a good understanding of that theme.

The second question was designed to get specific feedback on course structures. Almost every student mentioned that they liked the achievement system but that they would have liked some deadlines in order to help them manage their time more effectively. In particular, they wanted the Scholar achievement moved earlier in the semester to provide more foundation for their work. If we had one more meeting, I would have pointed out to them that this course of action was available to them all semester, and that it was their procrastination that pushed this to the end of the semester. I hope that they recognize this and that it was a good learning experience, but in the future, I would rather prevent the students from having to learn this lesson, so they can focus on course ideas instead. One of the best ideas shared by a student was to have achievements for playing more than one game in a genre, rather than just having them for playing different genres.

The third and fourth questions were Ron's ideas, and I am glad we added them. I assume that their answers were honest, since there was nothing to be gained at this point—remember that this "exam" did not contribute to the students' grades at all. According to their answers, about three students spent the expected nine hours per week on the course, and the rest spent less, some much less. This is interesting, as there was some complaining during the semester about how long it takes to learn and play some of the games in the canon. Several students admitted to spending no time on the course during the first third, as we expected. I think this shows that the level of rigor we expected of them was appropriate, even only some of them really rose to the challenge, and it indicates that students may do better with more scaffolding.

As for the fourth question, it reflected a similar phenomenon to the ISM's visit: the students seemed to value highly their peers who were most vocal, even in cases where—in my expert opinion—they had nothing to say. I take this as an indication that we needed more opportunity for the students to talk about specific design details: this would prevent the generalities and platitudes that students are so willing to share.

Next time

With an ounce of luck and a grant, I should be able to teach a similar course next Fall. With this in mind, here is a summary of what I would consider doing differently next time.

• I need to plan on a class full of partially-interested novices rather than hope for a few with passion and background knowledge. Should such students show up, I'm sure I can capitalize on it, but I need to make sure I'm not quite so surprised when initial critical analyses are all about kids' games.
• The students should do more smaller designs as a way to explore the design space and get used to the designer's mindset. In particular, I should have them make and share one-page design documents on shared themes, a trick I picked up during the semester from a conversation with Lucas Blair at Meaningful Play.
• Keep the achievement system, but add deadlines. Scott Nicholson—who I also met at Meaningful Play—recommended a system in which students have a few dates during the semester by which they must complete a threshold number of achievements in order to earn a certain grade. For example, completing three achievements by each of the dates would earn an A. I will likely do something similar, probably sending Scott a friendly request for more details as the hypothetical future course gets closer.
• Decompose and decouple achievements. Rather than have one that takes weeks and five designer games to complete, make this five different achievements. My colleague Evan Snider has explained to me how he chains together smaller achievements into "quests," which is something I might consider, although I do worry about taking the metaphor too far.
• Only use positive grading, not punitive grading. Our original model had grade penalties based on student infraction of policies, but this implies that students have points to lose. I prefer a model where students cannot lose credit, only gain it.
• Push them harder, individually if possible. The best products, and the best learning, came about when Ron and I contacted students individually to push them to do better. Each rose to the challenge.
• Include a request for feedback as part of the final exam. Since the university moved to digital course evaluations, their usefulness has plummeted. Most students do only perfunctory evaluations and do not provide the kind of useful feedback as when done on paper and in class. The comments from the exam were much more useful to me than the course evaluations, even though they ostensibly asked the same questions.