Revising Courses, Part II: Game Design

Following up on my previous post about revising my game programming course, today's post is about a revision to my game design course. This course is an honors colloquium, a special topics course only open to honors students and with an enrollment cap of fifteen. Every honors student has to take six credit-hours worth of colloquia, and the topics depend on who is available to teach them in any given semester.

This colloquium in part of a two-semester immersive learning project undertaken in collaboration with the Indianapolis Children's Museum, and it is being funded by internal funds through the Provost's office. Teaching in the Honors College is not a normal part of my load, and a major portion of the grant is "assigned time," allowing me to teach this colloquium. I mention this because the behind-the-scenes machinations are likely opaque to those outside academia: if I didn't have the grant, I would be teaching a Computer Science course instead of the colloquium. I am grateful to the Provost and his committee for approving my proposal, which allows me to teach this course that aligns so well with my research interests.

I taught a similar colloquium last Fall, and it had the same fundamental objectives: engage students in the academic study of games and learning, and, in collaboration with a community partner, have them produce prototypes of original games.There were two significant differences: I was team-teaching with my colleague Ronald Morris, and the community partner was the Indiana State Museum. It was my first attempt at achievement-based (i.e. badge-based) grading, and I wrote a lengthy reflection about the experience. For the redesign, I decided to focus on a few specific pain points from last time:

1. The students put off achievements until the end of the semester.
2. Some students made zero or nominal changes between game design iterations.
3. Not all the students were engaged in reflective practice: they were not adequately learning from the process.
4. In-class prototype evaluation time was rarely meaningfully used due to the points already mentioned.

I have met a few new colleagues through the conference circuit in the last year, and I am grateful for their willingness to share tips and tricks. In particular, the following changes reflect some specific ideas I have picked up from Scott Nicholson at Syracuse University and Lucas Blair at Little Bird Games.

Perhaps the most important revision to the course is the introduction of reflective essays. As in my game programming course, I was inspired by the participatory assessment model to grade reflections rather than artifacts. The students will present their weekly artifacts to the class, where artifacts might include summaries of essays and articles, posters, one-page designs, or prototypes. These artifacts will be subject to peer and expert formative evaluation, following the studio "crit" model. However, it will be students' reflections on these artifacts that are actually graded. As in my game programming course, I have decided to frame these reflections around essential questions and grade them based on (a) how they characterize an essential question, (b) the implications to practice, and (c) potential criticisms of the characterization. The research I have read predicts that this combination of achievement criteria and reflections should encourage students to produce high-quality artifacts without sacrificing intrinsic motivation.

Last time, the students had to choose a topic and iterate on it fairly early. The students with the best designs at the end of the semester were, for the most part, those who had to throw away major elements of their design or change themes entirely. This leads toward the desire to do more rapid iteration on ideas, not just prototypes; however, I still want each student to create a significant prototype by the end of the semester. To address this, I have divided the semester into three parts. In the first part, we will survey major themes of the course, such as games, fun, learning, museums, and children. The second part of the semester will be rapid creation of design sketches based on specific themes at the Children's Museum, about one sketch each week. The students will then choose one of these to prototype for their end-of-semester deliverable. I hope that this approach improves all the students' prototypes: even though they have less time to work on their prototypes, that time will be more focused and based on having had more reflective practice earlier.

Going along with the three-part division of the semester, I have organized the achievements into groups, some of which are tied to one of these parts. There is also an "unrestricted" category that can be earned at any time. I have introduced a throttle of one achievement submission per week plus one revision per two weeks. The students' grade is tied to the number of achievements they earn, and so this should help the students pace themselves while keeping me from having to evaluate an inordinate number of submissions at the end of the semester.

Following the most popular design principle for assessing learning with digital badges, I have introduced a "leveled" system of achievements. Certain achievements have gold stars attached to them, designating them as requiring special effort. These stars are tied in with number of achievements and reflection points in order to determine a student's grade. Note that two out of the three of these are directly in the student's control, and the one that isn't—reflection points—permits revision. Hence, students can essentially pick their grade based on their level of legitimate participation in the class. 

The full list of achievements is available online, and you can view it on its own page or embedded into the course description. Each badge is defined by a name, a blurb, criteria, and an image. The blurb and image are new this year, and I think they represent a major improvement. I used OpenBadges.me to design the badge images, making significant use of icons from The Noun Project. In case you're curious or want to sketch up your own, the border is the Ball State red taken from our logo (#ed1a4d) and the starred achievements use light yellow background (#ffff66).

Note that the core class activities in the second and third parts of the class are associated with achievements that lead up to stars. For example, a student who shows design revisions every week for the five weeks of prototyping will earn two gold stars, which are half of those required to earn an A. Indeed, I intend for the standard path through the course to consist of some student-directed inquiry in the first part, then two stars from one-page designs, then two stars from prototyping—and I will make this clear to the students at our first meeting! However, the system gives the students agency to choose a different path: if someone wants to focus on games criticism, or reading and reporting on game design texts, these are still rewarded and earn course credit.

One other change this year is based directly on student feedback from Fall. Last time, I had an achievement that was earned by playing several games that exhibited specific mechanics that I had identified. My intention was that I could guide students to experience genres, mechanics, and themes that I found interesting, but it ended up making the achievement hard to earn and delayed rewards for legitimate course activity. Note that there was no formal "leveled" achievements as I have this year with starred achievements, so this one achievement took much more time for the same reward as any others. This year, I have given the students much more freedom to choose a the games they will study and critique. They can choose analog, digital, or hybrid games, including sports and gambling games. I still provide some scaffolding through the games I chose to put on course reserves, but students who want to go in a different direction are free to do so.

The one weak spot in the course design, as of this writing, is the identification of essential questions. I have come up with two so far:

• What is the relationship between games, fun, and learning?
• How do you design an educational game for children?

I thought about introducing a third about design generally, such as "What is design?", but it seems that this is embedded into the second question. I worry that adding such an EQ would diminish the impact of the design-related one I already have. Finally, I will point out that the first essential question has explicitly guided my work over the last several years, and it was the explicit topic of my seminar at the Virginia Ball Center for Creative Inquiry: it is such a big question that others pale in comparison.

After having spent about three weeks this Summer revising my Fall courses, I find myself looking forward to the start the semester. It is good to have the time to rest, reflect, and revise. Of course, all this work has been without compensation, but, as the Spirit of Christmas once said, the rewards of virtue are infinitely more attractive.

Revising Courses, Part I: Game Programming

I spent the lion's share of the last two weeks revising my three courses for the Fall semester. They are the same courses as last time, although some of the themes have changed. After a trepidatious beginning, I am now quite pleased with the results. In today's post, I will describe the revision to my game programming course, an elective for Computer Science undergraduate and graduate students. The actual change to the course may appear small, but it represents a significant amount of research and learning on my part.

I have been structuring this course as a project-intensive, team-oriented experience. For example, last Fall the students implemented The Underground Railroad in the Ohio River Valley. I have also used this course to experiment with various methods of grading. I wanted the grading to be as authentic to the work as possible: students are evaluated on their participation and commitment only, not on quizzes or exams. For example, instead of midterm exams, I held formal face-to-face evaluations with each team member, modeled after industrial practice.

These methods work well, but reflecting on these experiences, I identified two potential problems. First, these methods fail in the case that a student refuses to participate or keep commitments: in particular, these methods produce little that could be considered evidence in the case of an appeal. Realistically, sometimes I get a bad apple, and so I want a grading system that allows me to give the grade I feel is earned. Note that while I admit to having given grades that are higher than I thought were earned, the assessment failure may be twofold: some students may require more concrete evidence of their own progress in order to improve or maintain performance, especially if such students lack intrinsic motivation.

The other potential problem stems from my wanting the students to engage in reflective practice, not just authentic practice. I wonder if some of my high-achieving team members have gotten through these production-oriented courses without having deeply considered what they learned. My model for encouraging reflective practice is based on industrial practice—agile retrospectives in particular—and is documented in my 2013 SIGCSE paper. This model, called periodic retrospective assessment, requires a team to reflect on its successes and failures intermittently during the semester, and at the end of the semester, to reflect on what it has learned. This sociocultural approach to assessment is appealing, and again, it seems to work in many cases, although it affords scant individual feedback.

While at this summer's GLS conference, I attended a talk about game-inspired assessment techniques given by Daniel Hickey. His model is called participatory assessment, and a particular aspect of it—which you can read about on his blog—is that it encourages evaluating reflections rather than artifacts. During his talk, he made a bold claim that resonated with me: writing is the 21st century skill. After having worked with Brian McNely for the last few years, I have come to understand “writing” in a more deep and nuanced way. (See, for example, our SIGDOC paper that takes an activity theoretic approach to understanding the writing practices involved in an agile game development team.)

Putting these pieces together, I decided to keep the fundamental structure of my Fall game programming course: students will work in one or more teams, organized around principles of agile software development, to create original games in multiple iterations. We will continue to use periodic retrospective assessment in order to improve our team practice and consider what we learned as a community. Now, I have also added individual writing assignments, to be completed at the end of each iteration. I want these reflections to be guided toward fruitful ends, and so I have brought in another pedagogic element that has intrigued me for the last several months: essential questions.

I first encountered essential questions (EQs) on Grant Wiggins' blog, and I blogged about this experience in the context of my advanced programming course. The primary purpose of EQs is to frame a learning experience. EQs have no trite or simple answers, and they are not learning outcomes, but they inform the identification and assessment of learning outcomes. With a bit of crowdsourcing, I came up with the following essential questions for my game programming course:

• How does the nature of game design impact the practices of game programming?
• How does game software manage assets and resources effectively?
• How do you coordinate interpersonal and intrapersonal activity within a game development team?

In reading about participatory assessment and the badges-for-learning movement, I came across Karen Jeffrey's HASTAC blog post. What she called “Really Big Ideas” seem isomorphic to EQs, and so I adapted her ideas in defining a rubric for evaluating reflections. I will be looking for reflections that provide the following:

• A characterization, with supporting evidence, of one or more essential questions of the course.
• The consequences of this characterization on individual and/or collective practice.
• The potential critiques of this characterization.

Deciding how to guide student attention is one of the most challenging parts of course design, and I recognize that by introducing these essays, I am reducing the number of hours I can expect students to spend on the development tasks at hand. However, these essays will afford conversation and intervention regarding reflective practice. They respect the authenticity of student work since, if done right, they should yield increased understanding and productivity from the students. This reasoning is similar to that given my proponents of team retrospective meetings as part of an agile practice: by reflecting on what we are doing, we can learn how to do it better. I have been encouraging my students to write reflectively, especially since starting my own blog; these reflective essays codify the practice and reward student participation.

The official course description for Fall's game programming course can be found at http://www.cs.bsu.edu/homepages/pvg/courses/cs315Fa13. I am happy to receive feedback on the course design, particularly the articulation of the essential questions, since they will be central to the students' learning experience.

Next time, I will write about the redesign of my advanced programming and game design courses, both of which involve turning to badges to incentivize and reward student activity.

Helpful Java Libraries: Notes from my IndyJUG presentation

Introduction

Yesterday, I gave a presentation at the meeting of the Indianapolis Java Users' Group. The presentation, "Serious Game Development in Java," gave some background about how I transitioned from game hobbyist to serious game development researcher, and I talked about my two successful Java-based serious games: Morgan's Raid and Equations Squared.

A portion of the attendees at the IndyJUG meeting

Usually when I talk about these projects, I am describing the immersive learning environment, my experience working with multidisciplinary undergraduate teams, or design and evaluation processes. Given that this crowd was primarily professional Java programmers, I decided to take a different tack, and I talked about the software architecture and what I learned by building these games. In particular, I talked about some of the libraries we incorporated into the game. Several attendees asked me if I could share more information about the libraries and why I chose them, so I decided to write this follow-up.

Morgan's Raid

Morgan's Raid was written using Slick2D, which had been my go-to library for Java game development. However, the original developer and maintainer, Kevin Glass, has moved on from the project, and it seems to be struggling now. Kevin did a good job keeping the libraries in sync with the native libraries of lwjgl, and with him off the project, I would not recommend starting new projects in Slick2D. Kevin has moved on to libGDX, which looks like an interesting project, although I prefer PlayN, as described in the next section.

Here is a brief description of the other libraries we used in Morgan's Raid. For brevity, I'm not including their transitive dependencies, but note that managing transitive dependencies on this project is precisely why I was blown away by Maven, as described in the next section.

• Apache Commons CLI: robust handling of command-line arguments, which we used to easily modify runtime behavior, e.g. fullscreen vs. windowed mode
• Apache Commons Configuration: robust handling of application configuration, such as animation speed, default volume, etc.
• EasyMock: mocking library for test-driven development
• Guava: broad and robust library that simplifies Java development, making particular use of the Lists, Maps, and Objects classes.
• Joda-Time: all the time calculations are handled with this fantastic library, and if you're doing any time-based logic, you should be using it too.
• SnakeYAML: parser for YAML, which we used to describe the cinematic scenes in the game
• CruiseControl: though not a library, we used this for continuous integration

I have several posts on this blog about the design and development of Morgan's Raid. If you're new, here are some of the most descriptive:

• The Components and Systems of Morgan's Raid
• Morgan's Raid postmortem

Equations Squared

When I began work on this project, I knew I wanted to develop an HTML5+Javascript solution with the least possible amount of pain. I evaluated several possibilities and settled on PlayN. This amazing library allows cross-compilation of the same codebase to desktop Java, HTML5+Javascript (via GWT), Android, and iOS (and Flash, kind of, but its support has not been great).

PlayN relies upon Maven to manage dependencies and project configuration. It took me some time to make sense out of how it was working, but now it's hard to imagine going back to manual dependency management. In Morgan's Raid, for example, when I wanted to add a new library, I had to manually download the binaries, and all the binaries of its transitive dependencies, put them into my project's lib folder, configure the build path, configure native libraries if necessary, and then hope that all the different libraries would work together. To upgrade any library to a new release, which happened to some of our core libraries during development, I had to repeat this process by hand. By contrast, to add, say, Mockito to Equations Squared, I just added this to my pom file:

<dependency>
  <groupId>org.mockito</groupId>
  <artifactId>mockito-all</artifactId>
  <version>1.9.0</version>
  <scope>test</scope>
</dependency>

That last bit, the scope, is really fascinating: it says that the project should use Mockito only when running unit tests. Scopes aren't needed for most of the libraries I use, but this example shows how simple a process it is to specify them. Also, need to update to a new release? Just update that version tag and Maven takes care of the rest.

Speaking of Mockito, it has become my favorite mock object library for Java. The API design is elegant and allows for readable code with minimal boilerplate. Here's a sample unit test—the same one I showed in my IndyJUG presentation—that demonstrates Mockito. By way of explanation, this code builds a token list containing "-" and "1", parses it into the expression "-1", then creates a visitor object and hands it to the expression. The expected behavior is that the visitor visits two nodes in the parse tree: the unary negation and the value 1. Note that mock and verify are static calls to the Mockito library.

@Test
public void testVisitorHitsNegationAndInteger() {
  tokens = ImmutableList.of(
    SymbolToken.MINUS, 
    IntegerToken.create(1));
  Expression e = parseTokens();
  Expression.Visitor visitor = mock(Expression.Visitor.class);
  e.accept(visitor);
  verify(visitor).visit(UnaryOperation.NEGATE);
  verify(visitor).visit(1);
}

Equations Squared uses the pythagoras, React, and TriplePlay libraries from ThreeRings. Pythagoras is a collection of geometry utilities that is well described on its Web site. React brings functional reactive idioms and the slots/signals idiom to Java, and that merits a small example. My GameView class exposes a signal with a method like this:

public SignalView onGameOver() {...}

Any agent in the system that needs to know when the game end condition is met can connect a slot that is notified when the signal is emitted, for example:

game.onGameOver().connect(new UnitSlot() {
  @Override
  public void onEmit() {
    displayListOfBadgesAndDemerits();
  }
});

Signals can have type parameters as well, though here I am using the simplest form. React also provides convenient value objects that emit signals when values change:

Value<Integer> v = Value.create();
...
v.connect(new ValueView.Listener() {
    public void onChange(Integer newValue, Integer oldValue) {
        // Handle change here.
    }
});

As you can see, what I'm doing is using React to provide convenient, quick, efficient reification of the observer design pattern. No extra boilerplate required here, no fat interfaces and adapter classes: just hook up slots and signals and get going.

Where PlayN provides a low-level API for game development, TriplePlay provides many of the niceties one needs to get games up and running, such as handling screen transitions, layer animations, and UI widgets. One of the reasons I love TriplePlay (and PlayN) is that the designer takes care to support fluent programming idioms. This is a direction I have been taking much of my own development as well. Consider this example from Equations Squared that handles popup notifications:

tweenTranslation(popup.layer())
    .from(325, 320)
    .to(325, 300)
    .in(0.9f)
    .easeOut()
    .then()
    .tweenAlpha(popup.layer())
    .to(0)
    .in(0.3f)
    .easeOut()
    .then()
    .action(deleteLayerAction);

The code reads exactly as one would explain the animation sequence. When source code is as short and expressive as it needs to be to convey an idea, that's a good program. Working with some of the fluent styling API in TriplePlay takes a little getting used to, especially if one comes from a push-button background as in Swing. However, I really felt like my own ability to express myself fluently in Java increased after learning to use TriplePlay.

For more on Equations Squared, check out The Story of Equations Squared.

Acknowledgements

I want to thank Michael Dowden for inviting me to present, and to all the IndyJUG community for their warm welcome. The event was graciously hosted by E-gineering, and I could tell by their amazing facilities that they are a company who takes their work seriously and respects their employees. Check out this centrally-located kitchen!

The E-gineering Kitchen

I also want to recognize the valuable contributions of the Apache Foundation, Google, and ThreeRings for their support of open source software development, as well as the significant contributions of Kevin Glass to Slick2D and Michael Bayne to the whole PlayN ecosystem.

Unboxing Incan Gold (Udc 4284)

I attended a gaming event at the local library a few weeks ago and was introduced to Incan Gold. My eldest son and I really enjoyed it, and I wondered if it would be accessible to my three-year-old. Remembering that my university library had a copy, I looked it up in the online card catalog. The entry describes it, "Educational resources non-print. Ask at desk." I wrote down the code on the page, "UDC 4284", and asked for it at the desk.

They must store non-print educational resources in boxes like this. Seems like a good idea to use uniform boxes with identifying codes, even though the game itself is in a box. I'm a collector of board games, and I like my boxes to be in relatively good condition too. I opened up the box...

Sure enough, Incan Gold. The labels on the box are probably helpful in case the game box gets separated from its containing box. Putting a university sticker on any library resource is probably a good idea, too. I took the game box out of the larger container and popped it open.

There's the rulebook. Little sticker on there indicates that it's from educational resources with code Udc 4284. I suppose that might get separated from the rest of the materials, and putting the code on there makes sure it ends up in the right place. I already know how to play, though, so let's move that rulebook out of the way...

Hey, take a look at the tent cards... and the temple cards! Looks like they wrote the Udc 4284 code on these too. Here's a close-up of the tent:

Is it only written on the top of each stack? Riffling through the cards reveals that not to be the case: the code is written on each and every card.

These are the Player cards. The code is written on the back of them. On the backs? That's marking the cards! I suppose it would be hard to write a code on the dark-colored front, but still, isn't it bad form to write on the backs of cards?

Same thing with all the Quest cards: code written on the backs, fronts pristine.

Fortunately, it's all one handwriting and the same marker! I would call these visually indistinguishable, for casual use anyway.

Ironically, the Temple cards have the codes on the front. If you're familiar with the game, this is the only kind of card where it wouldn't matter if it were on the front or the back.

Whew, can you believe someone went through and wrote "EdRes Udc 4284" on each and every card of the game? Sounds tedious. Well, let's move on and play the game. Better get out the gems.

Wow, even their own zipper baggie has "UDC 4284" written on it.

Wait, what's that on the gems... could it be?!

Every single gem has "4284" written on them! Black marker on the green and yellow gems, silver marker on the black ones.

Let's hope they never have to change their indexing scheme and relabel all the components!

Aside: My three-year old loves it and has just enough patience for a complete game. His strategy is questionable, but because it's a game that balances risk and reward, there has been times that he is safely in his tent with one or two gems while the rest of us return to base camp with nothing! The game allows for some good mathematical exercises for my elder son as well, including mental addition, change-making, and division with remainders.

GLS 2013 Conference Report, Day 3

[Day 1][Day 2]

Constance Steinkuehler gave a fantastic keynote presentation this morning. She gave a reflection primarily on her year at the Office of Science and Technology Policy in Washington DC. She focused her comments in four domains: federal agencies, philanthropic agencies, the games industry, and academic consortia. The talk was recorded for Webcast, although the link doesn't work as of this writing.

One of her major points was that we need more businesses coming out of serious games, particularly to deal with the pernicious problems of maintenance, marketing, and community. This echoes what Schoettler said Wednesday and Flanagan described doing on Thursday: seeing commercialism as a means to an end. I find this compelling as I consider my own corpus of work and current interests. However, I also fear the negotiations with university IP people. I chatted with a colleague who works in a start-up, and he described how university lawyers killed a private-public partnership over a $15,000 grant. That's peanuts for any serious project, and from his story, they wasted much more than the overhead cost of the grant in their conversations around it. The school he was working with was larger—and more successful with external funding—than my own. However, Steinkuehler also made a strong argument during Q&A that to get anything done requires people and resources, but that people are the hardest part: if you want to see something done, you have to be the one to invest the effort to make it happen. By that token, I suppose I should bite the bullet and not shy away from the more ambitious projects. I think she used the term "Own that ulcer for two years," which got a laugh, but I feel I need to seriously consider the value of my own time and happiness as well, and that of my family.

She made a strong call as well for the GLS community to be playing, critiquing, and spreading the word about each others' games. This is a nice idea, but I wonder if the community tries to keep on its kids gloves a bit too much. The last two games I've played in this vein were overproduced garbage. That is, the gameplay did not match the learning objectives at all, and the narrative could very convincingly be argued to teach the opposite of their goals.

Steinkuehler described her frustration in trying to collaborate with AAA publishers on learning games, coming to the conclusion that they just couldn't justify the lesser revenue to their shareholders, and that the megapublishers are necessarily risk averse. By contrast, she made the valuable and justifiable claim that we—the community that designs and develops learning games—are indie developers, and we need to embrace that. In particular, she encouraged everyone to go to GDC and walk through the Indie Arcade. Easy sell for me: that trip's going into my next grant proposal.

The four parting points of the keynote were: collaborate; we need more businesses; stop using the "chocolate-covered broccoli" metaphor (because of what one has to believe in order for this metaphor to be meaningful); and that happiness is a global priority (and so not to back away from "fun").

After the keynote, I ended up in a session that was a bit outside my interest area, but I was curious as to what happened in it. The most entertaining talk was one by Edd Schneider and Tony Betrus who reported on a series of experiments on task completion in GTA. The part I found most interesting was how they took a series of easy, medium, and hard tasks and compared the results between having the tasks ordered by a teacher (starting with easy, then into medium, then into hard) versus allowing the player to choose which ones to do in what order. They found that the teacher's choice resulted in the players completing more tasks, but the player's choice resulted in the them completing more hard tasks. Schneider and Betrus argued that this was a win for both cognitivists and constructivists, highlighting that they produced better results—while acknowledging that constructivists probably care less about learners completing small irrelevant tasks.

After this, I wandered over to the Education Arcade, my first trip there of the conference. I walked through to get a feel for what was there, and I overheard a great question from Roger Travis, who was giving feedback to a designer (I think): "What's the verb of the learning objective?" This is the kind of question I've heard posed regarding course syllabi particularly when I was serving on the College Curriculum Committee and as departmental Undergraduate Program Coordinator. I think it's a useful technique for sharpening the discussion of learning outcomes. Yet, I had never thought to apply it to serious games, which surprised me. I had just presented yesterday on the learning objectives for Morgan's Raid, as articulated by the original design class, and they were pretty raw; they wouldn't pass muster on the College Curriculum Committee, that's for sure. I'll need to keep this in mind when I work with this coming Fall's game design course to see if we can make some hay out of this.

Walking through the Education Arcade, I saw a poster about Monsterismus, a game to teach programming. Well, that's up my alley. Turns out it's by Matthew Berland, who was a discussant in several sessions I attended the previous two days, and whose comments at the time made him seem like a kindred spirit. Berland, Don Davis, and I, along with a chap whose name I did not jot down, ended up talking at length about games to teach programming, and I used the opportunity to bounce some of my latest thoughts off of them, including breaking the C-style programming mold (such as by using stream-based or prototype-based languages) and incorporating programming as player action in board games. I tried writing a lengthy blog post on these ideas at the end of last semester, but I had a very hard time inventing a vocabulary with which I was happy; fortunately, these guys seemed to grok where I was coming from, and I look forward to continued conversations with them as well as seeing where their work takes them. (Incidentally, Berland and Davis are also involved in IPRO, a game for iOS that is played by programming soccer robots, for those of you in the Mac space. I guess you can find it on your AppStore, or something.)

After lunch was a fireside chat with Jim Gee, but it was a bit anticlimactic since he mostly talked about The Anti-Education Era, which I just read. He is clearly something of a superhero to many in this community, and he had some good stories. Gee pointed out that schooling should not be geared toward jobs since 3/5 of the jobs are in the service sector and require only rudimentary training; instead, education has to be about something bigger than merely certification for employment. In a similar vein, he mentioned that education is a complex system, and that it is known that control studies cannot work in complex systems because you cannot fix the input variables. Yet, policymakers don't seem to understand this. I would have had some questions for Gee, in particular from my criticism of aspects of the book, but they were only accepting questions via Twitter, which is not a thing I do. I don't mind not having asked my question, but it was strange to me that they limited questions to this one technology.

Several awards were scheduled to be given next, and I was contemplating skipping this part. I tend not to like award ceremonies, but GLS did it right. They brought up a slide with a bunch of names on it and asked all those people to come to the front. Then, they ran through all the categories, handed out the awards, cheered, and were done in five minutes. Attention everybody: if you need to give awards at a conference, do it this way.

In summary, I found this to be an excellent conference. The mix of attendees was refreshing, including professionals (including non-profits such as museums), academics, and teachers. I was able to get lot of new references to read about, which is always good. I definitely hope to return again in the near future, and given that it's in driving distance, perhaps I can convince some of my students to come along as well.

It was good to see some friends at this conference too, people I have met at other conferences and online. When I started working in serious games about three years ago, I felt very disconnected from any community of practice. Now, I feel like there are people whom I can contact for help and inspiration, and indeed, several of the people I've met through these conferences have been generous in sharing their stories and advice.

For those of you who read this far, I will share my collection of Conference Tips from my trip to GLS. Note that most of these are based on mistakes committed (often unwittingly) by graduate students in attendance, but not all.

• If you come to the microphone to give any kind of formal talk, and you have not been introduced, introduce yourself.
• Don't say you're interested in making a game to "target women." They're half the population. That's not a target.
• If your volunteers have shirts, don't put, "Have questions? I can help!" on the back of the T-shirt. This means the person who can help is already walking away from you. Put this on the front.
• Never put light text on dark slides: it fails with a modicum of ambient light. There's certainly no excuse for this if you're from the host institution and know the rooms that will be used.
• Don't put thing on the slides that people cannot read, and then tell them it's not important to be able to read it. Design your slides differently.
• When giving a presentation, don't use a friend as a remote control. Get an actual remote control, or manipulate the slides yourself. It's less distracting than your trying to tell the other person when to move forward or backward.
• Don't use 3D bar charts for any purpose. They're just bad visualizations. Exceptions can be made for irony.
• Don't use pie charts either, 2D or 3D. They're ineffective and misleading. Exceptions can be made for pie charts describing their own similarity to Pac-Man.
• Be aware of the British and American hand signs that involve sticking fingers up at people, and to be safe, don't use any of them. That is, if you need to count, do it with your palm to the audience.

Ice cream on the terrance? Yes, please.
See you next time, Madison.

GLS 2013 Conference Report, Day 2

This morning's keynote was from Mary Flanagan about her work at TiltFactor. I have been a vocal fan of her lab's work since meeting some of them at Meaningful Play. The keynote did not disappoint—except the part where she used non-horizontally-aligned 3D bar charts to display some data. Ack. In any case, bias was a theme of the presentation, and she referenced the Why So Few? report, which was not previously familiar to me. Flanagan shared an interesting study that showed how Apples to Apples and Awkward Moment, even though they share the same game mechanisms, produce significantly different psychological impact on players. In another study, her lab showed that females' performance in Blokus reduced dramatically depending only on whether it was introduced as a game, a strategy game, or a spatial game. Included in her presentation was a strong recommendation to read Timothy Wilson's Redirect, so I've put it on my summer reading list.

As the previous morning's keynote had done, Flanagan also addressed the issue of sustainable models. This makes me reflect on the choices I give to my own student teams. For example, last Spring's game development studio consciously and explicitly chose to design a game for a very specialized context: educational outreach programs at the Indiana State Museum. This is a very narrow niche! I wonder if perhaps I should be sharing more of the advice I'm hearing from recognized leaders in serious games, and if as a result we should be targeting wider markets. Of course, then university IP rules and commercialism raise their collective head, and that's just not a beast I've felt up to taming.

I attended the morning session on assessment, and the first paper was presented by Jodi Asbell-Clark, director of the Educational Gaming Environments group at TERC. She described work evaluating Impulse, a physics education game. Notably, the game is designed for implicit learning, something I am interested in. She described the game as "situating game mechanics in scientific behaviors." They heavily instrumented the game to produce rich player activity data and combined this with both interviews and video recordings of players. This is yet more encouragement for me to push my teams to include meaningful logging data with all of our educational projects. An interesting anecdote from the presentation was that kids who liked science tended to also like science-like puzzle games and to also see their relevance to their lives, whereas kids who reported as not liking science did neither. I need to make time to read the full paper, as the presentation was full of great ideas that I would like to explore in my own work.

The next presentation was a study connected to Projenitor X. The big takeaway for me here was a reference to Kevin Dunbar's work on the critical and inspirational role of failure in scientific discovery. It's certainly something for me to look into after the conference.

Dan Hickey's presentation was my primary reason for attending the assessment session, since yesterday he promised to talk about why we should only be assessing reflections, not the other artifacts students are making. He presented a new assessment model based upon Engle and Conant's 2002 Cognition and Instruction article on productive disciplinary engagement, which I have not yet read but look forward to doing. He also strongly recommended Torrance's 2007 Assessment in Education piece—similarly not yet read but bookmarked.

During his talk, Hickey admitted to being a convert from individual differences psychometrics to sociocultural psychology. After the three talks, the discussant framed all three from an explicitly cognitivist perspective, Andrea diSessa's in particular. I'm still learning bits of both traditions, but from what I know, I suspected that this framing was not true to Hickey's core theories; so, I posed the question of why Hickey chose the sociocultural route. I have to admit, I had a hard time following the answer, but I think that's because it was based in a lot of the references that he had previously shared. I think this is a rich area for me to study, to help me to better understand assessment as well as sociocultural theories. Then, I'll ask Hickey again, and perhaps be better prepared to understand the answer!

Later in the day, Jim Bower shared some stories from his career, the most relevant of which to me was his articulation of the partnership model rather than the client model for educational software. This is something that we occasionally struggle with in the immersive learning model at Ball State, and the message sent by the university does not always match either the reality or the intention. Bower made a point that it is challenging to work with subject-matter experts who assume kids should love their content as much as they do, and when asked to elaborate on this, he described how educating partners as well as the end users was part of his goal.

I have a note in my notebook, "I should make a tower defense game." I have never made one. It would probably be a fun project. Maybe next GGJ or Ludum Dare.

A presenter referenced the term "intrinsic integration," which I have not heard before. It seems to reference designing games in which the player activity aligns with what you want them to learn. I talk about this a lot in my papers and teaching, but I never thought it needed a name. I suppose I should do a bit of legwork and figure out what communities are using this term so I can be sure to use it to help frame my theories of effective game design.

I went on a tour of Filament Games in the afternoon, mostly because I wanted to see how they used space and managed their talent. I verified that we had permission to take photos, which I assume means I can share them here. Not the best documentary evidence, I admit, but enough for me to share a few comments below.

Filament organizes into worker-units called "pods" that consist of two programmers and one UX designer. These pods then are connected with a producer, game designer, audio designer, etc., to tackle development tasks. The space in their studio leveraged this structure, with pods tightly collocated and separated from each other by dividers. Illustrators sat together and game designers sat together, but the space was clearly open enough that face-to-face communication was easy to achieve. Indeed, as we visited and ate lunch, I saw such informal meetings occur.

The tour was led by Maurice Cheeks, who mentioned that they follow agile practices, and so I was surprised when I did not see conventional artifacts such as Kanban boards or Scrum boards. I was able to talk with Gregg Sanderson, a project director, about how they track time and effort. They make significant use of Jira supplemented with Google Docs. He was kind enough to show some live data from Jira about stories, sprints, and estimates, which was great to see. Two particular pieces that stood out to me was that it looked like they had people assigned to fine-grained stories, not tasks, and that each of the disciplinary specialists (programmers, designers, etc.) provided their own estimates to complete their parts. After seeing it, I think I will likely stick with my familiar sticky-note approach in the short term, but depending on what kind of team I can recruit for next Spring, it's good to know what options are out there. (Did I mention I have a cool new project next academic year? More on that another time.)

Later in the afternoon, I heard someone make the claim that a game in which a player can click on animated animals for hints counts as helping the player to learn social-emotional skills. I find this hard to swallow, but I did not feel like challenging them, since I was getting myself psyched up for the next session.

My coffee, about 4oz left for the presentation.

I was the last speaker in my session. The first described an analysis of some of the iCivics games (coincidentally developed by Filament) and the second provided a theoretical model for how game design and civic engagement are intrinsically connected. My talk was a bit different, not dealing with civics or civics education, but rather with what real people actually learn by playing Morgan's Raid.

The Friendly Audience at GLS 2013

I'm happy to share the paper with anyone who's interested, but the short version is that this was Lyle Franklin's undergraduate honors thesis collaboratively transformed into a conference paper, with some additional input from my colleague Ronald Morris. He conducted a semistructured interview and thinkaloud gameplay with fifth graders in order to answer the broad question, "What does a student learn when playing Morgan's Raid?" We discovered three themes in the data. First, the students did learn about some of the important facts of the raid: who Morgan was, what he did, that it was part of the Civil War. Second, students demonstrated increased empathy after playing the game, both for the people of Indiana in 1863 as well as for Morgan and his raiders. Third, the student's youth-culture interpretations of parts of the game—particularly the words "chaos" and "reputation"—led them toward historically counterfactual conclusions. Fascinating stuff!

This was the end of the business day, so I enjoyed a beer and a nice dinner on the Terrace before returning to see how my family's day was in lovely Madison, Wisconsin. But that's the topic for another post.

[Day 1][Day 3]

GLS 2013 Conference Report, Day 1

I'm in beautiful Madison, Wisconsin, for the 9th Games+Learning+Society conference. My wife and I decided to make it a family trip, so we arrived Monday and took in some of the sights yesterday. We particularly enjoyed waking around the capitol building and the Henry Villas Zoo.

This is my first time at GLS, and I wasn't completely sure what to expect. I had heard that the conference would have a healthy mix of educators and researchers. My plans to attend last year's did not pan out. This year, I am presenting a paper entitled "Empirical Research on the Effectiveness of Morgan's Raid," which was coauthored with Lyle Franklin and Ronald Morris. It is based on Lyle's work on his undergraduate honors thesis, and I'm proud to be presenting it tomorrow.

There are two seasons on university campuses: winter and construction

The morning started with a few comments from Steve Schoettler, who is most famously a co-founder of Zynga and is now involved in the start-up Junyo. He started by sharing some stories from his background. My favorite was the story of Zynga's first game, a poker game that became popular enough for there to be a black market for chips. Schoettler and the rest of the team recognized this as an opportunity, a statement from the players that there was a valuable feature to be had if only they could create it—and, of course, monetize it. The talk was generally interesting, though he meandered into making broad claims about education reform that did not sit well with the audience. In particular, he had a direct assault against teachers' unions that drew the ire of many in the crowd. I suspect he will be remembered more for this than for anything else, which is a shame, but also a good cautionary tale to know your audience.

I want to highlight two calls for action that Schoettler included in his keynote. First, he called for a concerted quantification of learning. Given the broad and successful use of play analytics used at Zynga, it follows that if one could similarly quantify learning, one could apply similar data mining techniques. He did not mention the role that badges-for-lifetime-learning play here, and I remember learning at MeaningfulPlay that one of the goals of an open badge framework is to permit academic study of human learning activities. Later, however, he made a call to answer the question of what students actually learn. This strikes me as a complementary but contrasting point to the first, and it dovetails into what I want to bring up in my talk tomorrow. Quantitative methods can only ever answer the question "how much?" In fact, all the statistical techniques I know can only answer "probably how much?" By contrast, "What did they actually learn" is necessarily a qualitative question: each person learns based on his or her unique situation and history. I was hoping that he might recognize and comment on this difference, but by this point, the crowd was too busy cheering for unions and teachers. (And, in truth, it was probably not the right venue for this kind of discussion.)

I attended research-oriented talks all day, and I enjoyed it. The general quality of work was good, in my opinion. In the first session I attended, Douglas Clark presented a meta-analysis of a large body of learning games. I like to see this kind of work, and I was hoping to catch him and ask about how many of the studies address validity and reliability of their instruments, but I didn't have the chance. I was disappointed when an attendee brought up some qualitative studies that he referred to them, offhandedly, as not methodologically rigorous enough to fit into his meta-analysis. It's a shade of this same problem that's been bothering me, of a blind trust in statistics for statistics' sake, as if the quest for statistical significance is really the heart of the problem. Of course, if you're reading my blog, you've probably already read "The Earth is Round (p<0.05)", but I find it doesn't hurt to re-read it and occasionally re-recommend it.

An intriguing part of Clark's talk was an observation that most of the papers they studied either (a) spent a lot of time explaining and justifying their methodology, with almost no discussion of the actual game being used, or (b) described the game design in detail, and only lightly addressed research methodology. He encouraged attendees to consider which of the two groups they tended towards, and to consider the other audience as well when writing papers. This seems like sound advice.

Another speaker in the same session was Michael Lavine who discussed the Games and Learning Publishing Council. Looks like they're not quite ready from prime time yet, but I look forward to following the group.

The sessions are structured so that there are three speakers, and their three talks are given in sequence. Then, the moderator ("discussant") makes some general comments—which seem to often run much longer than necessary—before the audience is invited to ask questions of any of the presenters. I do not care for this model, since it seems that the attempt to put all the talks under one tent rarely works, and usually, the best presentation simply gets all the question. I prefer the approach taken at most of the conferences I frequent, which is 15 minutes to talk, five minutes of Q&A, and then out. One interesting tidbit from the Q&A this morning, though, was from a teacher who suggested that YouTube videos would be the best way for her to consume lesson plan and curriculum ideas for a game. This was interesting and surprising—I am not sure I would have thought of it on my own—and so I wanted to make a note of it here.

In the next session, Scott Nicholson gave a great talk about his playful courses, which is the term he prefers over "gamified." A few good ideas here that I need to think about as I revise my courses for Fall: don't use the word "optional" to describe assignments; multilevel badges (bronze, silver, gold) with unlimited attempts at improvement; and individualized learning contracts were useful.

I was glad to hear a talk by Dan Hickey later in that session. I had previously only known him as the author of one of my favorite education blogs, so it was good to see him in person. (I have not yet walked up and introduced myself as a fan of his blog, but that's on the agenda for tomorrow.) He was reporting on work related to Quest Atlantis, but I think it was his comments on the side that I found most interesting. Here's the best: "All this blah blah about '21st-century skills' is just that: blah blah. The 21st-century skill is writing." He also called for an end to studies that ask whether giving students autonomy increases their motivation: we know it does, so we don't need to keep checking in different contexts. He referenced another scholar, whose work is not familiar to me, in a claim that competition in education is not bad per se, but rather, it's the lack of feedback in competitive environments that hinders learning. His study showed that with good feedback—that is, feedback that is meaningful and that is used—competition can enhance learning. The key thing, it seems, is the feedback. I will have to read the paper to get a better understanding of the supporting research, which seems up my alley.

In a presentation, someone referred to Lord of the Rings Online as a "generic fantasy MMORPG." What does that mean? It's based on the property on which practically all other fantasy settings have their roots. Minor thing, but odd to me that someone who takes games seriously would belittle a game like this as "generic."

The provided lunch was a tasty Indian buffet, an unexpected treat. Dinner included a range of hors d'oeuvres, some pasta stations, very nice desserts, and wine and beer, all around a poster session. I talked to a few poster presenters, and I noticed a high concentration of non-student-presenters. The most interesting conversation I had was around a poster describing a restructuring of Bloom's Taxonomy for game-based learning, a collaboration between Second Avenue and University of Rochester. Sadly, my photos came out all screwy, otherwise I would share them as well as the names of the researchers. We had a good discussion, and I learned that Bloom's Taxonomy is taught quite rigidly as "true" in conventional teacher education. They use teacher's familiarity with this model in order to help them contemplate how game-based learning shakes up the status quo. Our discussion made me realize that my interpretation of Bloom's Taxonomy may not match with many others—particularly those trained in the teacher education tradition. This is good to know as I continue to frame my own ideas.

The conversations, posters, and free booze were all good,  but it was about this time that I got an email from my wife that she was feeling poorly, so I decided to call it a night. Turns out, I got back to the hotel right as a storm was breaking, so the timing was good. Now my wife is resting, the boys are in bed for the night, the blog post is written while ideas are still fresh, and it's time for me to turn in. My talk tomorrow is the last one in the 4:00PM session, so I suspect it will be a coffee-achiever day.

[Day 2][Day 3]