Wednesday, May 4, 2016

A Project Retrospective and a Final Exam for the 2016 Spring Studio

I announced the release of Traveler's Notebook: Monster Tales a few days ago. This was the semester project of Studio 368, a multidisciplinary undergraduate team that I mentored during the Spring semester. After each of our four iterations, we held iteration retrospective meetings, during which they contributed answers to these four questions:

  • What did we do well that if we don't write down, we might forget?
  • What did you learn?
  • What should we do differently?
  • What still puzzles us?

I am fond of this format, and I use it regularly with my student teams. (In fact, I've written an empirical evaluation of this approach, should you care to read it.)

Studio 368 finished production on April 27, and on our final meeting of May 2, I led the team in a project retrospective. First, we collaborated on drawing a timeline of significant events from the semester, including production schedules, guest visits, important decisions, external playtesting, and dissemination opportunities. Then, I challenged them to think about what they had learned--their answers to the second reflection question above--and to annotate these on the timeline with sticky notes. Unlike a normal retrospective meeting, where people work individually and then find clusters, this time I encouraged them to post items as soon as they thought of them, and to talk to each other about refining the articulations where appropriate. I circulated some colored dots, which could be used to affirm or agree with items others posted.

The result looked like this:

That's too small to read, but it gives you the overall shape. Below, I have transcribed each note, attempting to maintain the orthography while prudently fixing spelling errors. If an item had dots affixed to it, I mark this in parentheses.
  • Don't be afraid to step out of your bubble and try something NEW (2)
  • Narrative-driven games are really interesting and redefine regular games (2)
  • FOOD is important (1)
  • game design for academic research is DOUBLE hard
  • GAME DESIGN is HARD (5)
  • Slack is AWESOME (1)
  • Articulating how we record qualitative data is difficult
  • Getting qualitative data is important to understanding the project (& how it meets its goals)
  • Github and feature branching is amazing (1)
  • How to integrate my artwork into a working platform / game (1)
  • KIDS LOVE VIOLENCE (1)
  • The importance of physical space (2)
  • The usefulness of the React library
  • How to build a game efficiently in an interdisciplinary team (1)
  • How to communicate with a team consisting of members of various disciplines (1)
  • How a culture's "monster" represents a real fear or threat to that culture (5)
  • Monsters are windows into culture (1)
  • Everything we make will be remade. Care about it but don't get attached. (2)
  • Pair programming accelerates production while minimizing errors (2)
  • How to adapt to a new set of programming libraries in a timely manner (2)
  • Implementing creative limitations makes for richer encounters (character limit, cultural implementation, multiple encounter reactions, etc) (1)
  • How to condense my writing to successfully meet character limits (1)
  • Design log is law... Design Lawg
  • Establishing visual metaphors as game mechanics (3)
  • In game design, the best idea wins! (1)
  • A good paper prototype takes time, but adds great clarity to the project (3)
  • Lack of specific roles in Agile/Scrum
  • It's okay to call a sprint a failure (5)
  • Jumping in on what others have been working on is difficult, yet a great learning experience (that should be had)
  • Don't throw up (1)
  • Going to events with the team really helps us bond (3)
  • I'm interested in game academia (1)
  • What we are making is something refreshing but innovative (3)
  • Google analytics: how to implement it
  • Sound design enhances the gaming experience (2)
  • Kids will find the limits of whatever you're developing (2)
  • We learned a lot (1)
I think this is an excellent list, full of beautiful learning outcomes for an interdisciplinary undergraduate course. The students seemed to enjoy the activity as well, engaging in the process with both quiet thoughtfulness and friendly laughter. 

One of the intentions behind this activity was to get them ready for their final exam. I have tried several different approaches to giving final exams for this kind of immersive learning class. This semester, I went back to our essential questions (on the course description) as well as the learning outcomes from my project proposal. I wanted to crack these open and expose the various ways that students can start to tease apart the lessons from the context. I am including the final exam verbatim below, which includes both prompts and a justification for itself:

What is the final exam?


Most of our work this semester was bound up in the making of Traveler’s Notebook: Monster Tales. Our team has built a shared understanding that is necessarily bound up in the context of our collaboration: the people, the problems, the places, the donuts, etc. The main point of the final exam is to help lift our thoughts out of the particulars of this context, to improve our ability to draw upon our understanding as we move on to other endeavors.


Choose two categories from those given below and respond to one prompt in each. If you think of a different prompt, or even a new category, let me know, and we can negotiate how to move forward. If you spent at least an hour at the Immersive Learning Showcase talking to guests about our project, then you only need to choose a prompt from one category, although you are always welcome to write more.


Transformative Games
  • Consider one or more of these essential questions:
    • How do games encourage or discourage the development of literacy?
    • What is the role of player culture in transformative game design?
    • What is the role of theory in research and development projects like ours?
What do you know about it now that you didn’t before the semester started? (Note that you don’t need to answer the question per se: the point of an essential question is to guide inquiry because it doesn’t have a closed-form answer.)


Expectations and Reality
  • What was your biggest surprise of the semester? Delve into it in an essay, considering: why was it a surprise (that is, what knowledge did you have or not have coming a priori)? When did you realize you were surprised? What did you learn from the experience?
  • Compare and contrast the final product to how you initially thought the game would turn out. What accounts for the differences?
  • If you could improve upon the game in one way (including game design, asset production, and technology platform), what would you improve, and why?


Reflective Practice
  • What was your biggest mistake of the semester? How did it come about, what did you do about it, and what do you think you learned from it?
  • Reflect on what you have learned this semester that is related to your involvement in the game studio, and choose one outcome that is most important to you. Write a reflection about the context of this outcome: Who was involved? In what places did it happen? Over what period of time? Was the experience mediated by technology writ large---software, designed spaces, furniture? What were the sights, the sounds, the smells, the tastes, and the tactile experiences involved? How did these factors contribute to this outcome?


Legacy
  • Write an orientation document for future game studio students, something that could be given to them at the start of the semester to help them succeed in their work.


Discipline

  • Produce an artifact in accordance with your academic focus that represents something significant you have learned this semester.
One of my students responded to this by sharing, “That moment when @paul.gestwicki makes the most reasonable final exam I've ever taken :raised_hands::clap::pray:"” I am not sure that reasonable is the first adjective most people use to describe my approach to teaching, but I am glad this student thought so.

I may write up a longer project retrospective of my own now that the semester is winding down, but for now, I wanted to share the list and thoughts about the final exam. As always, feel free to share your thoughts and reactions in the comments section.

Saturday, April 30, 2016

The Launch of Traveler's Notebook: Monster Tales

I am pleased to announce the release of my Spring Studio team's game, Traveler's Notebook: Monster Tales. It is a two-player, narrative-rich game in which you encounter monsters from various world cultures. The game grew out of my consideration of learning through games like Tales of the Arabian Nights, which I wrote about back in January, and which led to the team's identification of their goals and early design work. I plan to write a more thorough retrospective of the semester once it's all behind us. For now, check out the game at travelersnotebookgame.com!


If you're in the vicinity of Ball State University, members of the team will be at the Immersive Learning Showcase on Monday, 4-6PM in Cardinal Hall at the Student Center. They would be glad to give you a demo and talk about their experience.

Wednesday, April 20, 2016

Metacognitive learning through a design thinking framework

I have written a few times about the design thinking model that I use, lightly adapted from George Kembel's:


Brian McNely and I wrote a paper about how we observed a team falling into a failure mode of looping like this:


That experience inspired me to start using this model as an exercise in CS222. Around the beginning of the third three-week iteration on the final project, I take a day where I begin by introducing this model. Then, I ask the teams to annotate their paths through the phases, starting with their project pitch, and annotating major milestones such as the ends of iterations. This is a great metacognitive exercise that gets students thinking about how they have been proceeding. Inevitably, I get a student who reports that they did it "wrong" by starting in somewhere other than empathy or by jumping between phases, but then I explain that we're not using this in a prescriptive way, but as a tool to help us think about our processes.

In this semester's CS222 class, we did this exercise last week Friday, which was the first week of the final iteration. This is the only photograph I have of the event, but rest assured that this is representative of the kinds of diagrams the students generate.


I usually tell the story about the study that McNely and I did, pointing out that sometimes we software development types get stuck in "ideate-build-test" loops without stepping back and remembering for whom we are building this thing. The arcs in students diagrams tend to have a similar shape, often skipping over the empathy step entirely (which is understandable since it's not really connected to our course learning outcomes).

On Friday, though, I saw something I had never seen before. The They recognized that they had made this transition early in their project:last team to present their models showed that they had a strong arc like this:


That is, they observed that they identified the real problems within a specific demographic, but they never actually decided which of those problems they would solve, and how they would do so. Instead, they went right from identifying the problems to programming. Having looked at their code in two formal evaluations and some information evaluations, I think this is astute: it perfectly describes the failure mode that they fell into. The team is pulling itself together in this last iteration, and I hope that this exercise is formative to that process.

As I was talking to my class about this model, I had a moment of inspiration where I realized there may be disciplinary differences in how one gets trapped in subcycles. This is only a hypothesis at this point, but I think I've seen this before in my multidisciplinary classes:


I see programmers fall into the trap where they think of something, build it, think of something else, build it, and so on, without ever meaningfully evaluating it or really returning to the user's needs. The humanists--such as English and History majors--tend to think about the problems that exist and think of solutions, then think of other problems, then think of solutions, and so on, without actually building anything to validate their ideas. I wonder if teaching this model early would help students to frame their processes in a useful way and help them catch themselves unproductive spiraling.

Wednesday, March 9, 2016

Leaving story maps behind

I wrote on the first of this year about how I was excited to try using story maps as a management aid for my Spring Studio team, and in the middle of February I followed up with some of my findings from one iteration's use. The team just finished their second iteration, and after talking about it with them during our end-of-iteration review and retrospective meeting, we are going to drop the story maps and go to Scrum.

The problem with the story map—as we reified it—is that it presumes that the reader already has knowledge of the fundamental processes that are required to complete a story. For example, we had an activity on the story map called "Narrate my friend's story" with a story underneath it called "Read conclusion and result." (StoryConclusion, and Result are all key terms from our game design and are defined in a design log.) When I look at that story, I can imagine the steps required to complete it: sketch the UI, make sure I have actual conclusions and results, make sure it fits with the overall game UX flow, rough it in code with placeholder assets, start replacing placeholders with real assets, test the integration. The problem is that I am not the one doing these steps: rather, it is a team of people who never did this before. What happened in the first two iterations with the story map is that they would look at "Read conclusion results" and, in the discussion, I would explain that these steps were required. These steps weren't tracked explicitly, though, and so we ran immediately into cognitive overload, and the team moved forward with the illusion that they knew how to proceed. Note that this is basically the same phenomenon as in a traditional class, where you can tell the class something and they will nod but not take notes, believing that they understand; of course these students then fail to follow the proper steps when it comes time to execute the task independently.

I am hopeful that switching to Scrum will make it easier for the team to track its progress on smaller tasks. It will mean more attention devoted to planning meetings, as we break down stories into individually-measurable tasks. One reason I was hoping for a leaner approach using story maps is that this is only a three credit-hour course, unlike the six credit-hour studios of the past few years, and so this will eat even more time away from production. However, if we're not producing the right thing, then it doesn't matter how quickly we do it.

Saturday, March 5, 2016

Rewriting departmental P&T documents to follow Scholarship Assessed

At the end of February, I gave a presentation at Ball State's Engaging Community series on the topic of publishing community-engaged scholarship. I was glad to be invited, although I don't generally describe myself as a "community-engaged scholar." I see community engagement as a necessary component of the authentic game studio experience: if my teams were not engaged with the community, we would not be doing authentic work. In any case, I took the opportunity to talk about the variety of research questions my colleagues and I have investigated, and the corresponding variety of venues where this work has been published. I have posted the slides, if you want to take a look, though as usual they may not make sense without the stories surrounding them.

Before me in the session were two respected colleagues, Chadwick Menning and Mellisa Holzman, who organize the Elemental sexual assault protection program. One of their main talking points was how to contextualize community-engaged scholarship for the promotion and tenure process. Given that a program such as Elemental is not conventional scholarship, some within the university perceived it as being solely a teaching experience (because it arose from a VBC seminar) or as service (because it involves helping people, I guess). Mellisa's sage advice was to contextualize and justify the work within your vita—a tactical maneuver that is facilitated by conceiving of the vita as a personal expression rather than a bureaucratic formula.

Inspired by their story, I began my talk with some extemporaneous remarks about my own P&T adventures. I told the audience that I didn't really have any trouble defending my work as scholarship because I had rewritten my department's P&T document to make it easier. Back in 2008, I chaired my department's P&T committee, and I had just recently read both Scholarship Assessed and Scholarship Reconsidered. With a favorable committee and authority from the department, I was the primary author of the changes that moved us from a laundry list of recognized activities toward a definition of what we mean by "scholarship". Here is the relevant section:
All works of scholarship will be judged according to the following standards (adapted from Glassick, et al., Scholarship Assessed: Evaluation of the Professoriate, John Wiley & Sons and the Carnegie Foundation for the Advancement of Teaching, 1997, p36). It is the responsibility of the faculty member to clearly demonstrate in writing that the standards listed below are satisfied by the body of work presented to document his or her bid for promotion or tenure.
  1. Clear goals. The scholar clearly articulates the basic purposes of the project, defines objectives that are realistic and achievable, and identifies the most important questions in the field.
  2. Adequate preparation. The scholar shows an understanding of existing scholarship in the field, brings the necessary skills to the project, and brings together the resources necessary to move the project forward.
  3. Appropriate methods. The scholar uses methods appropriate to the stated goals, applies effectively the methods selected, and appropriately modifies procedures in response to changing circumstances.
  4. Significant results. The scholar achieves the stated goals. The results of the project add consequentially to the field and open additional areas for further exploration.
  5. Effective presentation. The scholar uses a suitable style and effective organization to present the results of the project, uses appropriate forums for communicating these results to their intended audiences, and presents his or her message with clarity and integrity.
  6. Reflective critique. The scholar critically evaluates her or his own work, brings an appropriate breadth of evidence to her or his critique, and uses this evaluation to improve the quality of future work.
I should note that this section is followed almost immediately by a more conventional section describing what categories of evidence are recognized. However, whereas this was previously an exhaustive list, it is now a list of items that simply don't require much extra contextualization.
The following categories of evidence of scholarship are recognized by the department. However, support materials not fitting clearly into any of these categories may still be submitted. All evidence of scholarship should be documented according to the standards in [the section above].
  • Publishing refereed articles in journals or conference proceedings;
  • Publishing book articles, research monographs, or textbooks;
  • Writing reviews for national journals;
  • Presenting papers at professional meetings;
  • Conducting seminars, institutes, or workshops on computer science topics;
  • Evidence of success in developing, experimenting with, and implementing new teaching procedures and techniques;
  • Receiving grants or fellowships;
  • Adoption outside the department of software or electronically distributed materials that you have developed;
  • Serving as a professional consultant on matters of computer science to organizations or individuals outside the department.
There was no trouble with my department's approving these changes, and these policies have been in place ever since. When I have been considered for tenure, promotion, graduate faculty status, or merit pay, I simply provide a paragraph or two that establishes any unconventional work as scholarship given the six criteria.

This change did not instigate a visible cultural shift within my department: my colleagues are generally engaged in conventional disciplinary research. However, as far as I can tell, no one has pushed back against these changes either. Although I may be something of a black sheep in terms of how I conduct my research program, I don't sense anyone questioning the scholarly nature of it. I am sure it helps that, in addition to my more progressive work, I regularly produce traditional forms of scholarly output as well.

Although I am happy with this departmental policy, I also believe it does not go far enough in embracing Boyer's philosophy. Our document is patterned after a college document that uses a traditional three-category taxonomy of professorial activity: teaching, research, and service. That is, it misses the point that Boyer made, that everything a professor does should be scholarly—that the way of the scholar is scholarship. Worse yet, the college document reuses the word "scholarship" in place of the more traditional "research," which only further confuses the matter. Boyer's four scholarships simply don't fit into a model that separates scholarship from teaching and service. While I would like to see my department's document align with Boyer's model more explicitly, it's not a battle I choose to fight. Regardless, it was an epiphany for me when, as a young scholar, when I realized I could stop thinking about teaching, research, and service, and instead think about doing all things as a scholar.

(Erin Moore helped organize the university's session on Publishing Community-Engaged Scholarship, and she told me that she has received a few requests for more information about my department's P&T policies. I hope this post helps provide a bit more context than I could in some off-the-cuff remarks. As always, comments are welcome.)

Monday, February 15, 2016

What else is on the wall?

In my last post, I described some of the fun stuff on the wall in the studio. There was one piece I didn't describe, because I did not know its story until I asked the team today:
I had assumed it was some programmer art that was put up as a counterbalance to the excellent drawings by our resident art major. In fact, this monster was drawn by one of the kids who playtested our prototype last week Wednesday. What an excellent piece of community-connected inspiration! 

Friday, February 12, 2016

What's on the wall?

My post from earlier today was, pretty clearly, my attempt to jot down thoughts about this semester's studio project before they got away from me. I found myself with an unexpected half-hour or so, which was just right for writing.

But I know why you all really come here: the pictures! So, what's on the wall in the studio today?


On the East wall, we have a great collection of posters and sketches. I encouraged students to bring in some wall decorations early in the semester, and one jumped right on that, bringing the Teemo and Smite posters, and later, the calendar. Once we got our theme, someone else brought in, "Keep Calm and Find Monsters." Today, we got the addition of some thematic Darkest Dungeon images and several original drawings for in-game monsters.


On the West wall is the result of our post-planning UI design workshop. A few of us were standing with markers, a few sitting besides, and everyone was listening in. Out of these emerged a few core ideas that we're going to run with, including: using a 4:3 aspect ratio (give or take); using the left quarter of the screen to show persistent status such as scores and player names, and to give access to options such as restarting the game; showing a world map in the main play area, with a "card" metaphor for encounters.