Reflecting on the Spring 2020 CS490 Game Production Studio

I remember that there were many times in the past months where I thought, "I really should be blogging more about the CS490 class," and yet, I was surprised when I looked through the archives and saw that I hadn't really written anything at all. The only reference is in my post about how we moved to distributed teams. That being the case, let me lay a little foundation here.

I have been teaching a game production studio course in the Spring semester for several years. This has been done through the university's Immersive Learning program, which connects faculty, students, and community partners to build solutions to interesting problems. I have been having students apply to the experience so that I could get the best mix of talent. This led to my feeling rather guilty in Spring 2019, when I had over thirty people apply for the ten or so positions I could fill. This told me that there was much more demand than I had supply. One one hand, it's good for the partners to get the best students they can on the project; on the other hand, this means a lot of students who want to learn about multidisciplinary game production do not have the opportunity.

In response to this, and as part of my continued consideration of how to scale up games scholarship at Ball State, I changed the format of CS490 for Spring 2020. This time around, it was not an Immersive Learning class—no community partner meant lower stakes. I decided to welcome any students who wanted to join, as long as they were past the foundations courses of their major, and that I would let them pursue whatever creative games projects they desired.

I ended up with twenty-one students in the course. A few came from my Spring courses on game programming and game design, but most were new to the domain. I gave them a crash course and starter methodology, mentored them through a pitch and selection process, and we were off to the races. Four teams formed around four ideas.

I seeded each team with a starter methodology similar to the one I have been using for some time. I added a "student producer" role, and to support them, I tried to make some of the parts more explicit. This documentation, along with other pieces I shared during the semester, is in a GitHub repository. My plan was to make this public and reusable by future teams, but I think I should do an audit on it first.

The intention was that each team would be autonomous, and indeed, they were—almost to a fault! Whereas I usually am acting as a producer and director on the teams, I was now removed from some of the day-to-day details. This left me feeling out of the loop, and I am sure I missed some opportunities to help. The biggest danger was certainly second-order ignorance: the students don't know what they don't know. When I am embedded in a team, I can detect the signs of problems and move to identify or correct them. Outside of the teams, I suspect it is impossible for me to write down for students, in a clear and actionable way, all the possible things I might notice as potential problem indicators.

We moved forward in two-week sprints. The teams all struggled with the concept that the sprint should produce an executable release. However, I don't think they know that they struggled with this. Yes, each sprint produced something that ran, but it was months before anyone produced anything that was remotely testable. Indeed, the biggest failing of the semester was that, as far as I can tell, none of the teams ever really playtested their games. In a high-functioning team, like the best of my past Immersive Learning teams, we produce something that we can play with other people each sprint—ideally conducting the playtesting within the sprint, since we don't have separate "design sprints." As a result, the games all contain what I would consider significant impediments to playability that a modicum of playtesting would have made obvious.

The shift to distributed development went fairly smoothly for this class. There was an uptick in the use of our existing Slack. Teams did their daily  (MWF) stand-up meetings via Zoom, which I then also used to host the Sprint Review meetings. Zoom's remote control feature was valuable for when I was working with students on technical problems as well, although for the Sprint Review, it was really just about screen sharing.

By the end of the semester, all the teams had been able to build playable games that matched their original vision. Some took a slow and steady route to get there, and others had to twist, turn, and reboot their efforts, but they all got there. I am proud of their work, and any criticisms I have offered here are primarily aimed at helping me be a better mentor.

Each of the teams made some kind of public release, even if it's just a public GitHub repository. I'm happy to share those links here:

• Adventuring Supply Company, a game about running a shop for fantasy adventurers while paying off a debt to the dragon mafia
• Axil, a game about growing plants in space
• Teddy Bear Wars, a 2D platformer in which the demo level hints at a strange mystery
• Title Pending, a still unnamed combination rhythm game / deckbuilder

During our final Sprint Review meeting yesterday, I invited the students to share any observations they had from the semester. The results were interesting, so I want to summarize a few of them here. One student mentioned how, when they had gone to distributed development, they originally decided to become lax on synchronous development, and they saw productivity plummet as a result. They quickly changed direction and went back to enforcing that the team should work together at 9-11 MWF, just as we had done in person, and they found their productivity shoot back up. Another student noticed that "group productivity" was higher before the campus closure, but that afterward,
"individual productivity" went up. That is, if I understand it correctly, individual team members had to be more accountable to their work rather than relying on the collective in the room to solve problems. Several students offered closing comments about how much they learned and how they enjoyed this as being a very different kind of university experience.

At this point, I want to turn from talking about what happened toward what I should consider doing differently next time.

One of the mistakes I made early in the semester was that we formed teams before I set up a common Slack server for the studio. In the meantime, each of the teams had set up their own communication channels—primarily Discord. Once I realized this, I created the Slack as a place to have all the teams in one place: this made it clear that we are one studio with multiple teams, and it also allowed me to track all the teams in one place, watching for places where I could offer encouragement or advice. I requested that the teams dismantle their other, ad hoc communication systems, but one of the teams did not. They continued using Discord because it was more familiar to them, despite its being inferior for our purposes. I suspect, as any reasonable person might, that this was also a matter of pride: they felt ownership over the tool they liked and so assumed it was the right tool, the same way people do for operating systems and text editors. In any case, I share this vignette here as a reminder to myself to set up the Slack first, before the semester even starts, to make it clear where we go to share information with the team.

We did not have a good system for assigning grades to students. I deployed my usual approach here, which was to require individual reflective submissions at the end of each iteration. These took different forms and offered different perspectives, but they were also easy to game. For example, if I asked a student to share something they contributed over the past two weeks and a reflection on what it means, anybody can talk about some small contribution: it doesn't mean the contribution was valuable, that it was communicated well, or that it mattered. In some cases, students who clearly were doing a bang-up job were just overwhelmed with other obligations and forgot (or deprioritized) the submission of the reflection. Clearly, if I just used these, it would be insufficient and inaccurate. However, the intuition I would normally use when embedded in a team runs the risk of being unfairly biased with distributed teams. Indeed, I am not even entirely sure how I want to assign grades even now!

A different approach is clearly needed to deal with managing multiple teams, whether they are distributed or not. Last year, I wrote about the idea of paths for students; I like that idea, but I have not pursued it. Something more like contract grading or specifications grading may be in order. This approach would also let me make it more obvious which concrete elements of the methodology I value the most. For example, all the teams this year were supposed to be using burndown charts to track their progress, but only one did, and only that team reaped the benefits. The other producers either forgot or didn't care. If this were part of a grading specification for the student producer, or the team writ large, then you can be sure they would do it. Similarly, if there were a grading specification that said you had to playtest with people outside your team and document the results, you can be sure students would do it with much more gusto than if I continue to give them periodic reminders.

The problem with this is, of course, that you can actually make a great game without using burndown charts or without using a particular form of playtesting. Is this course about making a great game or learning my particular process? I cannot rightly tell the teams that they have authority and then also say they have to do things my way. This past semester, though, I leaned toward a libertarian approach and the result was something like a tragedy of the commons: students who I am sure did not carry their weight are grazing on the high grades of those who did.

Another option is to use peer evaluations. During the final meeting, one of the students mentioned another professor's course that uses a scale of percentage contribution, which he found useful. I took issue with that, since I'm not sure that percent contribution makes sense on a multidisciplinary team: what is a percent of the project anyway? The low-hanging fruit here is, of course, for me to just use the self- and peer-evaluation strategy that I have used in CS222 for years. Really, I could just deploy this almost as-is and see how it goes. Thinking back on the semester, I think it was mostly inertia that prevented me from doing it. The switch to distributed development was, in retrospect, probably the right time to do this. So, if nothing else, I can try this as a starting point next time around.

As I reminded my students yesterday, the game production studio course is hands-down my favorite thing to teach at Ball State. I am grateful to be able to do it. I hope to run another studio in Spring 2021, and I am continuing to invest energy in finding ways to scale up these experiences without sacrificing quality. Thanks for reading!