A Project Mentor's Reflection: Q&A

A colleague sent me four questions in response to yesterday's post. With his permission, I'm responding publicly here, so that others may also benefit.

1. How does the methodology document get created? Do you start out with an empty document, a document with topic headings only, a draft document that is then refined, or something else?

I started with a seed of a methodology years ago, based on what my teams had been doing successfully and what I knew to be best practices from industry. Now, each semester I revise what the previous semester left behind. I've made a public copy of last Spring's methodology document here. There's a published skeletal methodology and a bit of theoretic background as part of my and Brian McNely's 2016 TOCE article.

There are a few things that I am considering changing for next time, in part from writing yesterday's reflection. The Anzeneering link was a late addition, and I talked about but did not make Stop Work Authority cards. Thinking more about the principle of safety, I think I should require that reading and make the signs myself and have a few scattered around the room. This resonates with the Psychological Safety rule I linked in my post, which comes from Google's studying of team dynamics.

The Crystal Clear principles are good but only if people read and embrace them. I'm not sure that they do, although we are able to come back to them during our periodic retrospectives. As I mentioned, I've been watching some Robert Martin talks, and this one on professionalism brings up some interesting ideas not explicitly mentioned in Cockburn's list. The particular one that I want to add somewhere is Never Be Blocked. I think that's a piece that hindered a few team members, as they decided to wait for someone else to act rather than take the initiative to get things done. It's one of those things that I have fairly well internalized, which means it's hard for me to remember to teach my students about it! Some of the other pieces that Uncle Bob talks about tend to be tricky for students working in games development, particularly the bits about test coverage and manual testing, since so much of game development is experimental. I think it's hard for students to distinguish between code that should have 100% coverage and code that is not worth writing unit tests for; of course, they too often bias toward the latter!

I also note that the conversation around version control is stuck in the "for programmers" section, with only the value of frequent integration mentioned for everyone else but not the process. I need to make this more clear that it applies to everyone. Obviously, this would have to be couched in discussion of psychological safety, since this whole notion will be foreign to people from outside CS.

2. My sense is that you have students write reflective essays throughout the semester, and that this is what you use for assessment for their final course grade. What else, if anything, figures into their grade?

That's right, and this too comes right out of the Academic Studio pedagogy that McNely and I documented in TOCE. We borrow from Agile practices the idea of short iterations, incremental development, and reflective improvement. That last point manifests explicitly in two ways: collective team retrospectives and individual reflective essays. As project mentor, I lead the retrospectives and document the results, reflecting approved changes into the methodology document as well. The essays are framed with essential questions that are designed to help students see beyond the project to deeper academic issues. That's the good kind of "academic" of course, not the dismissive one. We only had two for this studio: how do teams coordinate activity to address interdisciplinary research and development problems, and what is the role of theory in research and development projects like ours?

In addition to each end-of-iteration essay, students make at least one entry into an individual portfolio, which is usually a link to a resource and one or two sentences of context. I started doing this a few years ago after having had a bad experience: I had a team with some slackers appeared to produce nothing all semester, but with so much collective work, it was hard to have evidence that they didn't contribute significantly. I was unsure if this exercise was valuable, so I polled my students after receiving this question. The one respondent indicated that he thought they may have been even more valuable than the essays, particularly the act of finding an artifact that represents specific learning. He admits that it sounded like busywork at first, but he came to appreciate the task. I will point out, though, that I could not find any correlation between the portfolio items and my observations about whether students were getting the big picture or experiencing personal growth, not like I could with the essays. Also, the portfolio entries were never intended to contribute to the final grade, but rather as a simple measure of accountability, although the course plan doesn't make this clear.

The final exam is just a pair of reflective essays chosen from a suite of options that I provide. Here's what their actual final exam looked like:

What is the final exam?

Most of our work this semester was bound up in the making of Spirits at Prairie Creek Park. Our team has built a shared understanding that is necessarily bound up in the context of our collaboration: the people, the problems, the places, the donuts, etc. The main point of the final exam is to help lift our thoughts out of the particulars of this context, to improve our ability to draw upon our understanding as we move on to other endeavors.

Choose two categories from those given below and give an essay response to one of the prompts therein. Keep in mind the definition of "essay" given on the course description; you are also welcome to reference my essay on essays. I encourage you to use your portfolio entries and past essays as evidence to support your arguments. If you think of a different prompt, or even a new category, let me know, and we can negotiate how to move forward.

Academic Miscellany

Consider one or more of these essential questions:

• How do teams coordinate activity to address interdisciplinary research and development problems?
• What is the role of theory in research and development projects like ours?

Note that you don’t need to answer the question per se: the point of an essential question is to guide inquiry because it doesn’t have a closed-form answer.

Expectations and Reality

• What was your biggest surprise of the semester? Delve into it in an essay, considering: why was it a surprise (that is, what knowledge did you have or not have coming a priori)? When did you realize you were surprised? What did you learn from the experience? Compare and contrast the final product to how you initially thought the game would turn out. What accounts for the differences?
• If you could improve upon the game in one way (including game design, asset production, and technology platform), what would you improve, and why?

Reflective Practice

• What was your biggest mistake of the semester? How did it come about, what did you do about it, and what do you think you learned from it? You might structure your response as a postmortem, for example, in the format described by Fitzpatrick and Collins-Sussman.
• Reflect on what you have learned this semester that is related to your involvement in the game studio, and choose one outcome that is most important to you. Write a reflection about the context of this outcome: Who was involved? In what places did it happen? Over what period of time? Was the experience mediated by technology writ large---software, designed spaces, furniture? What were the sights, the sounds, the smells, the tastes, and the tactile experiences involved? How did these factors contribute to this outcome?

Legacy

• Write an orientation document for future game studio students, something that could be given to them at the start of the semester to help them succeed in their work.

Discipline

• Produce an artifact in accordance with your academic focus that represents something significant you have learned this semester. Provide a brief author's statement to guide the perspective of someone outside of your discipline.

Some of the responses were beautifully composed and thoughtful, enough that it was worthwhile. I do believe that the essays I have read, and the conclusions I draw from them, align well with my observations of what students are doing. Once again, however, we run into an interesting research question that I don't have the resources to approach. Also, I am sad to report that in two years of using this final exam format, no one has taken that final option to produce a disciplinary artifact that represents their learning: some day I'll see a student dance a final exam.

3. How did you get the “official plaque”? Do all immersive classes get this, or only certain ones? 

I don't know. I haven't gotten one in years despite continued immersive projects. This was the first time in many years that I requested an official portrait, though, so maybe there's causation there.

4. What are some of the topics/activities you do during the first few weeks of the semester to get things started, and to get the students to understand that, in large part, they are “driving the bus” for this course?

I had used some preparatory activities in the past, but I found them mostly to be a waste of productive time. Now my philosophy is that the best way to get students acclimated to the work is to do the work. I usually spend the first week of the studio course doing some prep and background, though. I usually introduce a few board and card games that I believe exemplify game-based learning, and I teach them fundamentals of the MDA Framework as a rough introduction to understanding the differences between playing and making. There's an intentional playful bouncing back and forth between games and analysis that foreshadows the semester's work. We go very quickly into production mode though, where I have already broken down the user stories into sample tasks. I try to make the first sprint an easy win. For Spirits at Prairie Creek, we had a lot of unknowns we were trying to wrestle down in the first sprint, and maybe that was part of what caused some confusion moving forward. By contrast, for Traveler's Notebook: Monster Tales, I had already laid out the core gameplay and we could move forward with a minimum viable product. Of course, all of this leans on the fact that I am doing production courses; if a course is not based on production, then your mileage will vary.

One thing I neglected to mention in my previous post is that two students, in their final essays, wrote orientation guides for next year's students. Both of them addressed the need to recognize, up front, that they will have communication problems. Again, this is one of those things that I've internalized so much that perhaps I did not prepare the students well enough to expect it. Both of these students wanted future teams to know that communication problems were normal, but also something to be quickly recognized and addressed rather than hoping they would go away. I am still not sure if I should print those out and hand them out next year or incorporate those ideas into a more robust methodology-orientation document. When one of the problems is that I'm not sure students are reading the methodology at all, it makes it hard to know where best to focus my efforts.

I hope that helps!