Morgan's Raid Postmortem

Morgan's Raid enjoyed its public release at the Conner Prairie Educator Event last Thursday. With that event, the project is essentially "done," modulo the inevitable maintenance. For the past few weeks I've been sketching out notes for a postmortem in the style of Game Developer magazine, and now I'd like to share these with you. In truth,  I vary a bit from their style, looking more at structural elements of the team than technological decisions, but this is natural given the dual nature of the project, which was designed to create something significant while also meeting learning objectives for the developers. (As I think about this, it would be interesting to write one in their style too, but that's a task for another day.)

What went well

Scrum

The application of Scrum was excellent, especially as we refined our process in the Spring. The students and I appreciated the public nature of the task board, and they healthily struggled with what it meant for a team (not an individual) to commit to a task. The impact of the burndown charts was evident in how students discussed their work and our progress, and the sprint renegotiation process was relatively smooth. The sprint reviews and retrospectives gave the team a chance to analyze and discuss the product and the processes, and their empowerment helped fuel their continued motivation.

Our modifications to Scrum were relatively minor. Because the team only met three times per week, we did not have daily stand-ups, but our "once-per-workday" stand-ups were sufficient. One aspect that we completely dropped was the product burndown chart. With our already low number of collaborative, collocated working hours, I decided against adopting formal planning poker sessions. In the Fall, I made up story points based on my experience as a developer and a teacher, and in the Spring, we skipped these entirely. I do not doubt that they would have been an academically interesting feature to keep, but aside from academic intrigue, I think I made the right call to slim Scrum down a bit.

Due to scheduling issues we could not control, our artist could not meet with the developers in the Fall. As a result, his work was rather undirected and his contributions relatively few. In the Spring, he was able to meet with the developers, creating a real cross-functional team. The results were amazing, and I need to be sure to promote cross-functional teams in future experiences. This is not directly related, but I'm sure that our separation of "designers" and "developers" in the Scourge of Orion project was a major contributor to its lack of productivity.

Playtesting

Our external playtesting experiences were fundamental to the success of the game, and I express again my gratitude to those teachers and schools who shared their time with us. I know that I learned a lot by seeing how the end users interacted with our various prototypes, but more importantly, I could see how it helped the developers build empathy for the end users.

It was especially useful to playtest with different demographics. "Indiana fourth grade students" encompasses a wide variety of skills and dispositions, and even in our relatively limited playtesting, we were able to witness this range. I suspect that this perspective not only helped the team to design the game but also to develop empathy for the challenges of the elementary school system.

Every child who played Morgan's Raid gave us some feedback, and often they had feature requests. Some were clearly outside of our scope, such as adding multiplayer support and leaderboards. Many others were based on the students' experiences with school and educational software. At some point later, I plan to write a more detailed piece on why there are no quizzes in Morgan's Raid, despite this suggestion being proffered by many playtesters. What was more salient to this post is that some of the team members jumped on these recommendations, an unspoken assumption being that we should provide what the intended end users wanted. There's a deeper anthropological lesson here that I tried to explain to the team, that while the users are always right, they also aren't experts. This brings us back to the design issue mentioned earlier, that experience design is difficult, and it's important that someone have the power to veto.

Development methodology: Entity system architecture and Mercurial

I have written before about the entity system architecture, and I list it here as one of the great successes of the project. Adopting this architecture allowed us to separate tasks rather seamlessly, certainly better than I had experienced before in working with teams of undergraduates.

The fact that this was a team of amateurs is significant. The ideals of proper object-orientation can and should lead to modular, robust, scalable software systems. However, because the students lack experience, perspective, and technical understanding, it is insufficient to give them a software design task and expect production-quality results. Even with the entity system architecture, there are parts of the code that are ugly to behold, unnecessarily coupled across disparate systems. That said, the fundamental idea of the architecture—decoupling behaviors into components, which are acted upon by systems, and combining components into entities—was simple enough that it could be explained with a few examples and then applied by the team.

Maybe I'm cheating by putting two in one here, since the version control system is orthogonal to the software architecture, but I'm going to mention it here anyway. This was my first time using a distributed version control system, and Mercurial more than proved itself. The students were able to learn it quickly and became quite savvy with the workflow. It was a little distressing that even at the end of development, some team members were mad at Mercurial for making them merge, where I would have liked them to adopt the more enlightened perspective that merging is a necessary part of teamwork, and Mercurial made it infinitely easier than doing it by hand.

Spring Space

The use of a dedicated space in Spring was fantastic. This is not easy to come by in a university setting, and on behalf of the entire team and our stakeholders, I am grateful to my department for granting us the use of the room. Everything one would expect to improve was improved. While the Google Docs approach to managing the Scrum artifacts was reasonably effective when working with the 25-member team in Fall, it pales in comparison to the power of three giant whiteboards and piles of sticky notes in the Spring.

The incredibly impact of dedicated space has made me an evangelist for more responsible use of university resources. It got to the point where, on the Future of Education Task Force, people would just put "space" down for me in any list of things we need to change. Space is a deeply political issue on any college campus, but one important lesson I learned is that I should pay more attention to it during the planning stage of any future activity. If I can find any partner who is wiling to pony up space as part of a grant proposal, for example, I'll jump on it.

Team formation

It was great to see how the team gelled, especially the smaller group in the Spring. They really were a functional independent game development team in every way that matters. There was some work that I did to try to foster this sense of team, such as adopting agile methodologies, bringing goodies for Sprint Review and Retrospective meetings, and hosting board game sessions. These definitely helped, but I also could tell that the students were hungry to make something important, to do something that mattered beyond a credited learning experience. Even if we all had our own idiosyncratic views of what Morgan's Raid should be, we shared this vision of being able to work together to make something significant that none could do on his or her own.

The only distressing part about this success is that the team is now gone. Usually, at the end of fifteen weeks, I am ready to push my students out to face their next challenges. With this team, a part of me wishes I could keep them here so that we can continue to face challenges together. I build an emotional bond with these undergraduates. They were my teammates and coworkers, my partners in a difficult endeavor, my colleagues, and my friends, and I will miss them.

What didn't go well

Code reviews were not in our methodology

We did not have formal code reviews, and in retrospect, our code quality suffered because of it. It is easy at the time to ignore reviews in favor of "productivity," but I think this was a classic managerial blunder. I am not certain  what methodology we could have used in the Fall to promote this, but we certainly could have inserted some more formal code reviews in the Spring. This would have directly helped us meet some learning objectives while improving the code base, since as I and others have said before, software development is a learning process.

Because of the generally poor code quality from Fall, I made sure to introduce a formal verification step into the task approval process. When I had the time and energy, this meant that I would not just check that the task was working, but that the underlying implementation was reasonable; however, working under our significant time pressure, I often did not have the time and energy.

In truth, I do not have much experience with code reviews. Sure, I've graded countless assignments, but these have primarily been summative assessments, not formative assessments. This is not an excuse but rather an opportunity for me to include code reviews into a future experience, so that my students and I can learn together how best to use them in our environment.

No technical specification

I invested a significant amount of time in Summer 2010 working the results of the Spring design colloquium into a functional design specification. This took the form of a wiki on Google Sites, and it was shared with all the team members and product owners. This was a colossal effort, since the output from Spring was nowhere near the quality we expected, and so really I was redesigning the whole game from disparate prototypes—and even then, what I wrote up was not that good and ended up being significantly changed before shipping. This does not mean it was wasted effort: it was still a necessary step along the way. However, because this took all of my spare cycles in Summer 2010, I was out of time when it came to the technical specification. I wrote up a few notes about methodology (inspired by Cockburn's excellent book that I read the same Summer), but for the most part, the software architecture was only in my head.

I suspect that the lack of a technical specification was a major factor in the relatively low productivity of the Fall semester. I believe that the productivity of Spring would not have been possible without the learning of Fall, but at the same time, and as pointed out by a team member the other day, much of the Fall was undirected learning. That is, the students were learning, but they didn't always know where they were going with this. Being able to rely on, for example, a software architecture specification, could have helped them see how their little piece was actually going to make a big difference.

Undergraduates, by and large, do not have the perspective necessary to design large software systems. After all, having wide perspectives and the ability to integrate across disciplines is an outcome of higher education, not an input. I'm sure a technical specification would not have improved Fall's productivity by an order of magnitude, but it would have helped a little. Most importantly, it would have helped the students see, from early in the process, the value of documentation. About two-thirds through the Fall semester, we did a massive architectural overhaul, and to get this done, I turned to de facto standard UML. Once the students understood how this language was part of my bigger process, they could see the value in it and showed interest in learning it. I suspect this transformative experience could have been brought about earlier through more formal technical specification.

IP

Dealing with the university's intellectual property policies has been an exercise in frustration. I suspect that mine is like a lot of other institutions of higher education, where tight budgets are causing the administration to jump on the commercialization bandwagon, and yet the staff has little or no experience dealing with software innovations. As a point in case, I have yet to meet anyone at the university who seems to understand software licensing, and all of the forms and procedures are aligned with physical invention.

In the university's defense, I put off dealing with IP until we were 90% done with the project. I just didn't want to deal with it, because I knew it would be a quagmire. Sure enough, it was, and still is, since some of our issues are still not sorted out: I was told that I would receive official written documentation on some issues over a month ago that I has still not arrived. I suppose I should have started the internal processes earlier.

On the positive side, all of the programmers have agreed to license their code under an OSI-approved license, and so I hope to make a source release of the game and announce official licensing policy by the end of the month.

The takeaway on the IP front is that, in future projects, I should deal with it earlier on, ideally during the proposal process. This would lead to a traceable paper trail and explicit acknowledgement of the funding agency of the intended licensing strategy, and we would only have to go through the current headaches if things changed.

Undergraduate game designers

I hesitate to mention this here since several of my undergraduates may read this, but understand that I write this with respect and affection.

We put too much faith in undergraduate designers. Game design is hard, and I posit that legitimate educational game design is even harder. I think what we asked of our Spring design colloquium class may have been impossible for students at that level of experience. Not only were they lacking game design knowledge specifically, they had essentially no prior experience with design thinking at all.

I recently took the time to reflect on the game designs I have come up with during my lifetime. I've always been a hobbyist, and specifically, I remember writing up rules for a tabletop RPG that got some playtesting when I was an undergraduate, and I sketched out at least one CCG rulebook that was circulated among my friends in grad school. When I look back at these, I made classic amateur mistakes, prioritizing the wrong elements and trying to do big design up front rather than incremental and iterative design. That is, I knew nothing formally of design thinking. As I reflect on what my students designed, in the design colloquium and during the evolutionary design of the development process, I see the same kinds of amateur mistakes.

For those who were on the team for more than one semester, I'm sure that they saw how I made changes to cope with these observations. The design colloquium was very open-ended, but in the Fall and Spring, I positioned myself as the design authority. While anyone could pitch any idea to me—and indeed the great design successes of the game are primarily student-generated—I was the gatekeeper, and there were many directions that I forced to be shut down. As benevolent dictator, I needed to ensure that energy was being spent fruitfully, even if it was perceived as less glamorous by the team.

Trying again to look forward, I think this will have to be the perspective I take in future endeavors. We were probably not critical enough in the Spring, and there wasn't enough time in a three-credit course for the students to develop healthy levels of design self-criticism.

Attention management

In the Spring, we had an excellent space in RB368, and I am sure that the success of the project rested in having access to the space. That being said, I'm not sure that I managed it correctly. As I have read more about agile software development spaces, I realize that we did not have a good management system in place for dealing with distraction. Our shared workspace was regularly interrupted by both related and unrelated events. I know there were several times where I was in the zone and then got broken out of it by well-intended interruptions.

For many students (at least it is said), working in a high-distraction environment is their modus operandi. The sad result is that they have no idea how much productivity they are losing, since they are not intentionally managing their attention. This would have been a good learning opportunity for them to see what being productive really felt like. Next time around, such as next Spring's VBC seminar, I will be more explicit about working conditions. I am confident that this can be done without negatively impacting the social bonds of the team, and in fact, I suspect it will lead to an increase in mutual respect.