Trying Labor Logs for a Game Design Course Final Project

I have been teaching introductory game design for years, and my students have always had to do some kind of multi-week final project. Each week, I ask them to give a status report on their progress. Most semesters, this has to include some form of playtesting, and in some semesters, I have also allowed alternative work for when the students want a break. Throughout the whole time, I reminded the students that they were being graded on process and not product. No one can guarantee that creative endeavor such as original game design will result in a good product, but a student can show that they have learned and followed an appropriate process for mitigating risks.

One of the perennial problems with this approach was that it was pretty easy for students to get equal credit for doing vastly different quality of work. Often, it would be clear from the presentation that nothing substantial was done in the intervening week, but in the absence of objective criteria, I rarely pressed this. The few cases where I did give only partial credit for the week's progress were so overt that no student ever pushed back on it. As a systems designer, though, I never liked this hole in my course plan. Call it an exploit, if you will.

Several months ago, I became aware of Asao Inoue's work on labor-based grading. His stated goal is to improve equity and inclusion in the writing classroom, and his approach was to use adapt concepts of contract-based grading where the only metric for success is labor—that is, meaningful effort on task. I read only a portion of his book, which made it clear that he and I have different goals. However, I did enjoy reading about the implementation of his philosophy: students track their work in a convenient Google Sheets spreadsheet. The sheet does some automatic computation for them and provides affordances for reflection.

I have been working on the final project specification for my game design class, and I just published a draft for student review this morning. It includes a reference to a sample labor log based on Inoue's template. The columns I used are as follows:

• Date
• Start Time
• Duration (minutes)
• Primary Activity: one of Design, Playtesting, Documentation, Research, and Collaboration
• Location
• Engagement: one of Very High, High, Medium, Low, and Very Low
• Mood
• Notes

I also have a computed column—Week—which uses the course calendar to determine the week of the project (1–5) corresponding to the date. 

There is a sense in which the creditable labor is just the duration, but I like the idea of including some of the extra pieces, such as Primary Activity, Location, Engagement, and Mood. I haven't come up with a final exam plan for the class yet, but I will likely have a question that asks students explicitly to reflect on the patterns they see in their logs.

The spreadsheet uses a pivot table to tally the accumulated duration for each week and present it in a bar chart. If I had more time, I would like to annotate the bar chart with the grade cutoffs that I have written up in the project plan. That said, I think some visualization is better than none.

I am eager to see if a requirement to track labor on the project has any impact on the observed quality of students' weekly progress. Of course, the shift from in-person presentations to discussion board posts is probably a more significant variable here, but I'll keep an open mind. You can expect me to share more thoughts about the labor logs here on the blog toward the end of the semester.

A year of achievement-oriented assessment in CS222

I conducted a formal research study as part of my experiments with achievement-oriented assessment in CS222. Although the semester is over, I have not yet opened my consent forms to determine whose data I can use in the formal study. First, I wanted to take some time to write out my informal thoughts about the past academic year.

For context, here are a few links:

• Revising Courses, Part III: Advanced Programming
• Tweaking Achievements and Reflections
• Fall and Spring course descriptions

My intention was to use the same format in the Fall and Spring, to get more consistent data for the study, but there was one piece that I felt I had to change between semesters: the tight binding of achievements and reflections. In the Fall, students' weekly writing assignments had to tie directly to their achievements. Late in the Fall, I realized  that this was an unnecessary restriction on the topic of their writing, and that what I really wanted was for them to be reflecting on any course experience, whether it was the achievements or not. The Spring reflection essays were more varied and much less formulaic. In fact, some delved into personal stories that were quite moving and inspiring. Of course, I cannot say more about that publicly, except to point out that it was the kind of moment that makes me proud to be a positive influence in a person's life.

There were still some hiccups with the reflective writing. There were three parts to each essay in the Spring—one less than Fall—but still students submitted without hitting each part. I can probably mitigate some of this by spending more time with Blackboard, copying the requirements from the course description to each assignment rather than referencing the course description. The programmer in me wants to simply reference the authoritative document, but experience has shown that students fail to check the reference.

I am also concerned about students' balance of attention between course activities. The achievements are mostly programming-related and are designed to integrate with the final projects. The best students saw this and leveraged it, but it feels like the capacity to recognize this dropped exponentially with students' effort or attention. Reflections, similarly, were supposed to tie all these experiences together, and the best students got this; many other students wrote trite, predictable, platitudinous, or probably-false reflections just as a way to earn course credit. I believe that anyone can learn from reflective writing, and I suspect many of my students did, but I also suspect that many more did not learn significantly from this activity, since they treated it as a box to be checked. This makes me wonder if the whole system of reflections should, itself, be an achievement, rather than required. More on this in a moment.

The achievement system led to students engaging in a wider variety of learning activities than I had seen before. Furthermore, much of this was self-directed: I gave criteria whereby achievements were earned, and students were responsible for finding the resources to earn them. Of course, I myself was a resource, but one could almost draw a line between high and low grades as to who recognized this. This is a common problem, not related to achievements I think, where "strong" students will talk to professors, ask questions, seek guidance, and "weak" students suffer from culturally-imposed fears that prevent this (fear of showing weakness, fear of authority figures, etc.).

All of the achievements were posted to a shared folder on Google Docs where students could see all that each other have posted. All of my evaluations were posted as comments on these shared documents. It is not clear to me the extent to which students read each others' posts and learned from them, but I know that it did happen at least sometimes. I am uncertain whether students read each others' posts for content or for form, but it seems to be a bit of each: from what I gather, ambitious students would scan others' posts and see where I left comments, and then try to avoid these pitfalls. As above, however, we see the weaker students making the same mistakes that their peers made just a few lines above them.

Several achievements dealt with what I called "production code," meaning that it had to be the students' final project code or parts of Open Source software. This definition, which seemed clear to me, caused students trouble in both semesters. Regularly, students would submit achievements that used code that did not fit this definition, sometimes the same students week after week. The confusion between current project code and old project code seems like a lack of attention to detail; however, some students confused public open source code with snippets on StackOverflow or tutorial sites. I need to clean up this definition in the future, making it clear that the point is to look at code that is part of a system, not didactic or decontextualized.

This leads to another unexpected oddity: in the final round of reflections and in the final exam, students used terms like "production code" as if they were universally meaningful, as opposed to only being meaningful in our class. I did not make it clear that this was a definition for us, and so I worry that students may go into interviews or other discussions and mistakenly express this as vocabulary. This was much worse with the achievements, in fact, many of which I tried to name in clever ways. For example, one achievement deals with Java's toString, equals, and hashCode methods, and I called the achievement, "Master of the Big Three." I didn't know what else to call it when I was writing it up, and that seemed like fun, so I called it that. However, then I overheard students talking about "The Big Three" as if that meant something in an absolute sense. Similarly, I had several achievements dealing with design patterns, and since I see patterns as a way-of-programming, I called these "Pattern Disciple" achievements. Then I heard students talking about "pattern discipline"—they didn't even get the word right, but they were still using these corny terms as if they were accepted jargon. I am not sure how to fix this aside from coming up with much drier names for the achievements, but the punster in me will likely find this impossible.

To get an 'A' in the class, students needed to earn "meta-achievements," which were coherent sets of related achievements. For example, the "Engineer" achievement required using UML and design patterns. (In retrospect, it may have made sense to call these "quests" instead, to better show the metaphor of an 'A' being at the end of the journey.) The members of one of the Spring teams did earn this achievement, although because of how I articulated the achievements, they had to incorporate the Decorator pattern where it really had no place. When I saw this, I was disheartened: what's the point of a pattern if it doesn't actually solve a problem in the domain, and would the students recognize the difference? When I expressed this doubt to the team, one of them actually defended my policies, saying that he was happy to have an excuse to study the pattern and use it, regardless of whether it "fit" the project. At least this guy saw the balance of pragmatic and academic here, and the result was positive for us both; however, I think I can circumvent this in the future by articulating quests in a way that ensures the activity is still motivated by the project context. Most students have difficulty finding the kind of intrinsic motivation that this one had!

Right at the end of the semester, I had an enlightening conversation with a student regarding the entire metaphor of achievements. I asked why he had not been more active in pursuing achievements when they were clearly tied to the course grade. His answer may have been an excuse, but it's interesting nonetheless. As a gamer, he conceptualized "achievement" as they are often treated in game design: as ancillary rewards, secondary to the primary gameplay. The discourse around the semester focused on the nine-week final project, and I can certainly see how students would see that as "the point of the course." From a game design perspective, these weren't really "achievements," then, as used in contemporary game design: they were essential, as missions or quests, waypoints or progress markers. Another student put it another way, saying that they weren't achievements at all, they were just a set of optional assignments. This is valuable feedback, and I see two ways out of this situation: re-name the achievements to reflect their essentially-mandatory nature; or actually make them optional, perhaps as gatekeepers to tiers of grades ('B' and 'A', for instance).

Achievement submissions were organized by topic, so all the "Master of the Big Three" achievements were on one page, for example. This was convenient for students to see what their peers had been working on, and what kind of feedback I had given to them. The resulting course document structure was like a wiki, then, with content organized by topic. I wonder how different the learner's experience would be if we took a portfolio approach, organizing the content by contributor. This, I think, would make it easier for individuals to see and think about what they did all semester. On the other hand, it would work against the goal of helping students see themselves as part of a community, and I suspect it would be less likely that students would look at each others' work, since one would not know whether a particular student earned a particular achievement or not. Also, it might lead to many mid-level students honing their sites on the portfolios of a small number of strong students; this would be problematic since, from my reading of the literature, students learn better from people at similar skill levels. This leads me to think that the wiki-structure approach is superior to the portfolio-structure approach, and now that it's on my blog, maybe I will remember this point as I consider what ideas to incorporate into my summer course revisions.

When I first imagined the achievement system, I envisioned the contribution of high-quality content on which I could provide meaningful, thoughtful responses. In fact, in my original design, I would not evaluate the achievement submissions at all, since each one had crisply-defined requirements, but I added a professor-review before the start of the Fall semester. In practice, there were many submissions that were not up to snuff. On one hand, this is predictable: if students will often submit assignment submissions without actually fulfilling their requirements, why wouldn't they do the same with achievement submissions? On the other hand, why would they do this? The achievements were not graded on some archaic percentage scale. There were only three grades: incorrect, acceptable, and correct-with-distinction (coded as 1, 2, and 3). The best I can figure is that students uploaded incorrect submissions because schooling culture encourages "submit and hope for points" as the dominant idiom, rather than "consider whether or not this is correct." (This is just one of many soapbox issues for me, where I point my finger squarely at... well, pretty much all the rest of formal education, which is difficulty because I have pretty small fingers. I could go on about how the dominant schooling culture discourages reflection, accountability, creativity, and deep learning while encouraging risk-averseness, but this parenthetical phrase is long enough.) I wonder, perhaps, if the numeric coding itself brought to mind students' conventional experience with points. Indeed, one student told me that he really wanted to get those '3's, even though, from a grading perspective, they were equal to '2': the only difference was one of pride. If I keep this aspect of achievements, I will try turning these into symbols, perhaps thumbs-down, thumbs-up, and ... I don't know, maybe Chuck Norris.

As a preliminary conclusion, I call the achievement-based grading system a success for this course. Students were able to focus on their nine-week projects and engage in meaningful content-oriented and reflective activity, with freedom to choose activities that aligned with their interests. At a minimum, I want to redesign some achievements to align more clearly and obviously with the projects, but I will hold off on major redesigns until I take some time to read my study data.

Tweaking achievements and reflections

Before the start of the Fall 2013 semester, I wrote about the changes I was incorporating into CS222, the Advanced Programming class. With that semester behind us, I can say that it went well, although not without the need for tweaks. For the Spring, I am keeping the fundamental structure. Here is a summary and justification for the tweaks.

I am scaling down the number of achievements required for each grade level. The original plan in the Fall was two achievements per week. However, students were not as adept as I had hoped at aligning achievements and projects. The intention was that students would earn achievements by doing their projects, but the majority of the class tried to do achievements separately from the projects. In the Fall, we ended up making a change midstream, reducing the number required during the nine-week project to only one per week. For the Spring, I'm pulling back a little bit more, expecting two achievements per week during the first three weeks, then only one per week during the two-week "warm-up" project and the nine-week major project.

Another change addresses the intended achievement-project synergy: restricting the code that can be used to complete achievements. When working toward achievements, Fall's students were permitted to use any code to which they had a license. In practice, I had students submitting trivial modifications to terribly-conceived CS1 and CS2 projects all the way to the end of the semester. In Spring, students can use any such code during the first third of the semester, but once they start their nine-week projects, they must use either that code or public open source code.

I have also changed the how and when reflections are required. In Fall's section, students wrote one reflection per achievement. I am sure that the best students—those on the road to being reflective practitioners—found this fruitful, albeit time-consuming. The students with worse writing skills spent inordinate time on these reflections, time that should have been spent understanding the material instead. Their action was justified, perhaps, by the fact that reflections were a significant part of the course grade. Also, I allowed resubmission of the reflections, but I think my attempt to encourage mastery learning instead fostered the"submit and hope" approach that I find so endemic and disdainful. In the Spring, the students will instead be asked to write one reflection per week, and this reflection may focus on any course-related experience: achievements, in-class discussions, project team discoveries, and so on.

The structure of the reflections will change as well. In the Fall, students were asked to do three things in their reflections: characterize an essential question, describe implications to practice, and consider alternative perspectives. Many students struggled with these all semester. Although I believe that tackling these problems is good for students, there is a limited about of time students can devote to this class. Hence, I have reduced and rearticulated the requirements: in the Spring, students will be asked to address one of the essential questions ("characterize" caused undue problems) and describe implications to practice.

Despite the addition of achievements and structured essays, perhaps the most uncertain of my Fall experiments was in not grading the final projects. As mentioned above, the course design was supposed to foster students' intrinsic motivation to work on the projects, which would provide context for the achievements. My ambitious designs failed to account for the pragmatic, goal-directed nature of the undergraduate! Students realized that the projects were not graded, and many students found this unmotivating: they wanted their work to be recognized directly in their grades. I will be returning to my conventional approach, grading both the projects and their ancillary artifacts such as presentations and a final report.

A final significant change that I expect to bring vast improvements is getting off of Blackboard. We used Blackboard's wiki in the Fall as a place for students to post and share their achievement artifacts. The wiki is ghastly, and because it was so difficult to give targeted feedback, we ended up with a very poor signal to noise ratio. Often, a student would post something that was wrong, and I would give formative evaluation to that student, encouraging revision; the student would not revise, and within a few days, I would receive several similar submissions from students who assumed the original was correct. What a mess. In the Spring, we're going to use Google Drive instead. I am uneasy about requiring students to register for a Google Account in order to participate, but most have them anyway, and the technology stack is so vastly superior that I'm going to try it. I will set up a shared folder that will contain one document per achievement, and students will post their artifacts within those documents. My feedback can take the form of comments, which are visible to all readers. I will have a bit of manual work to translate these data into Blackboard's gradebook, but that's worth it to help close the gap between my feedback and peer learning.

I finished up the course description this morning and linked it to my Web page. The achievements themselves are basically the same as the ones I designed last Summer, with a few notable additions that reward technology (integration with relational databases and structured data parsers) as well as community-building (interviewing alumni and senior capstone teams). Each achievement has also been clarified as to what exactly the corresponding artifact is—some had been vague in the previous iteration.

The semester starts in just a few days, and I'm looking forward to meeting the new crop of CS222 students. I am not looking forward to teaching at 8AM and Monday's high of -6°F, but I am trying not to complain.

But seriously, 8AM and -6°F high temperature.

Revising Courses, Part II: Game Design

Following up on my previous post about revising my game programming course, today's post is about a revision to my game design course. This course is an honors colloquium, a special topics course only open to honors students and with an enrollment cap of fifteen. Every honors student has to take six credit-hours worth of colloquia, and the topics depend on who is available to teach them in any given semester.

This colloquium in part of a two-semester immersive learning project undertaken in collaboration with the Indianapolis Children's Museum, and it is being funded by internal funds through the Provost's office. Teaching in the Honors College is not a normal part of my load, and a major portion of the grant is "assigned time," allowing me to teach this colloquium. I mention this because the behind-the-scenes machinations are likely opaque to those outside academia: if I didn't have the grant, I would be teaching a Computer Science course instead of the colloquium. I am grateful to the Provost and his committee for approving my proposal, which allows me to teach this course that aligns so well with my research interests.

I taught a similar colloquium last Fall, and it had the same fundamental objectives: engage students in the academic study of games and learning, and, in collaboration with a community partner, have them produce prototypes of original games.There were two significant differences: I was team-teaching with my colleague Ronald Morris, and the community partner was the Indiana State Museum. It was my first attempt at achievement-based (i.e. badge-based) grading, and I wrote a lengthy reflection about the experience. For the redesign, I decided to focus on a few specific pain points from last time:

1. The students put off achievements until the end of the semester.
2. Some students made zero or nominal changes between game design iterations.
3. Not all the students were engaged in reflective practice: they were not adequately learning from the process.
4. In-class prototype evaluation time was rarely meaningfully used due to the points already mentioned.

I have met a few new colleagues through the conference circuit in the last year, and I am grateful for their willingness to share tips and tricks. In particular, the following changes reflect some specific ideas I have picked up from Scott Nicholson at Syracuse University and Lucas Blair at Little Bird Games.

Perhaps the most important revision to the course is the introduction of reflective essays. As in my game programming course, I was inspired by the participatory assessment model to grade reflections rather than artifacts. The students will present their weekly artifacts to the class, where artifacts might include summaries of essays and articles, posters, one-page designs, or prototypes. These artifacts will be subject to peer and expert formative evaluation, following the studio "crit" model. However, it will be students' reflections on these artifacts that are actually graded. As in my game programming course, I have decided to frame these reflections around essential questions and grade them based on (a) how they characterize an essential question, (b) the implications to practice, and (c) potential criticisms of the characterization. The research I have read predicts that this combination of achievement criteria and reflections should encourage students to produce high-quality artifacts without sacrificing intrinsic motivation.

Last time, the students had to choose a topic and iterate on it fairly early. The students with the best designs at the end of the semester were, for the most part, those who had to throw away major elements of their design or change themes entirely. This leads toward the desire to do more rapid iteration on ideas, not just prototypes; however, I still want each student to create a significant prototype by the end of the semester. To address this, I have divided the semester into three parts. In the first part, we will survey major themes of the course, such as games, fun, learning, museums, and children. The second part of the semester will be rapid creation of design sketches based on specific themes at the Children's Museum, about one sketch each week. The students will then choose one of these to prototype for their end-of-semester deliverable. I hope that this approach improves all the students' prototypes: even though they have less time to work on their prototypes, that time will be more focused and based on having had more reflective practice earlier.

Going along with the three-part division of the semester, I have organized the achievements into groups, some of which are tied to one of these parts. There is also an "unrestricted" category that can be earned at any time. I have introduced a throttle of one achievement submission per week plus one revision per two weeks. The students' grade is tied to the number of achievements they earn, and so this should help the students pace themselves while keeping me from having to evaluate an inordinate number of submissions at the end of the semester.

Following the most popular design principle for assessing learning with digital badges, I have introduced a "leveled" system of achievements. Certain achievements have gold stars attached to them, designating them as requiring special effort. These stars are tied in with number of achievements and reflection points in order to determine a student's grade. Note that two out of the three of these are directly in the student's control, and the one that isn't—reflection points—permits revision. Hence, students can essentially pick their grade based on their level of legitimate participation in the class. 

The full list of achievements is available online, and you can view it on its own page or embedded into the course description. Each badge is defined by a name, a blurb, criteria, and an image. The blurb and image are new this year, and I think they represent a major improvement. I used OpenBadges.me to design the badge images, making significant use of icons from The Noun Project. In case you're curious or want to sketch up your own, the border is the Ball State red taken from our logo (#ed1a4d) and the starred achievements use light yellow background (#ffff66).

Note that the core class activities in the second and third parts of the class are associated with achievements that lead up to stars. For example, a student who shows design revisions every week for the five weeks of prototyping will earn two gold stars, which are half of those required to earn an A. Indeed, I intend for the standard path through the course to consist of some student-directed inquiry in the first part, then two stars from one-page designs, then two stars from prototyping—and I will make this clear to the students at our first meeting! However, the system gives the students agency to choose a different path: if someone wants to focus on games criticism, or reading and reporting on game design texts, these are still rewarded and earn course credit.

One other change this year is based directly on student feedback from Fall. Last time, I had an achievement that was earned by playing several games that exhibited specific mechanics that I had identified. My intention was that I could guide students to experience genres, mechanics, and themes that I found interesting, but it ended up making the achievement hard to earn and delayed rewards for legitimate course activity. Note that there was no formal "leveled" achievements as I have this year with starred achievements, so this one achievement took much more time for the same reward as any others. This year, I have given the students much more freedom to choose a the games they will study and critique. They can choose analog, digital, or hybrid games, including sports and gambling games. I still provide some scaffolding through the games I chose to put on course reserves, but students who want to go in a different direction are free to do so.

The one weak spot in the course design, as of this writing, is the identification of essential questions. I have come up with two so far:

• What is the relationship between games, fun, and learning?
• How do you design an educational game for children?

I thought about introducing a third about design generally, such as "What is design?", but it seems that this is embedded into the second question. I worry that adding such an EQ would diminish the impact of the design-related one I already have. Finally, I will point out that the first essential question has explicitly guided my work over the last several years, and it was the explicit topic of my seminar at the Virginia Ball Center for Creative Inquiry: it is such a big question that others pale in comparison.

After having spent about three weeks this Summer revising my Fall courses, I find myself looking forward to the start the semester. It is good to have the time to rest, reflect, and revise. Of course, all this work has been without compensation, but, as the Spirit of Christmas once said, the rewards of virtue are infinitely more attractive.

Revising Courses, Part I: Game Programming

I spent the lion's share of the last two weeks revising my three courses for the Fall semester. They are the same courses as last time, although some of the themes have changed. After a trepidatious beginning, I am now quite pleased with the results. In today's post, I will describe the revision to my game programming course, an elective for Computer Science undergraduate and graduate students. The actual change to the course may appear small, but it represents a significant amount of research and learning on my part.

I have been structuring this course as a project-intensive, team-oriented experience. For example, last Fall the students implemented The Underground Railroad in the Ohio River Valley. I have also used this course to experiment with various methods of grading. I wanted the grading to be as authentic to the work as possible: students are evaluated on their participation and commitment only, not on quizzes or exams. For example, instead of midterm exams, I held formal face-to-face evaluations with each team member, modeled after industrial practice.

These methods work well, but reflecting on these experiences, I identified two potential problems. First, these methods fail in the case that a student refuses to participate or keep commitments: in particular, these methods produce little that could be considered evidence in the case of an appeal. Realistically, sometimes I get a bad apple, and so I want a grading system that allows me to give the grade I feel is earned. Note that while I admit to having given grades that are higher than I thought were earned, the assessment failure may be twofold: some students may require more concrete evidence of their own progress in order to improve or maintain performance, especially if such students lack intrinsic motivation.

The other potential problem stems from my wanting the students to engage in reflective practice, not just authentic practice. I wonder if some of my high-achieving team members have gotten through these production-oriented courses without having deeply considered what they learned. My model for encouraging reflective practice is based on industrial practice—agile retrospectives in particular—and is documented in my 2013 SIGCSE paper. This model, called periodic retrospective assessment, requires a team to reflect on its successes and failures intermittently during the semester, and at the end of the semester, to reflect on what it has learned. This sociocultural approach to assessment is appealing, and again, it seems to work in many cases, although it affords scant individual feedback.

While at this summer's GLS conference, I attended a talk about game-inspired assessment techniques given by Daniel Hickey. His model is called participatory assessment, and a particular aspect of it—which you can read about on his blog—is that it encourages evaluating reflections rather than artifacts. During his talk, he made a bold claim that resonated with me: writing is the 21st century skill. After having worked with Brian McNely for the last few years, I have come to understand “writing” in a more deep and nuanced way. (See, for example, our SIGDOC paper that takes an activity theoretic approach to understanding the writing practices involved in an agile game development team.)

Putting these pieces together, I decided to keep the fundamental structure of my Fall game programming course: students will work in one or more teams, organized around principles of agile software development, to create original games in multiple iterations. We will continue to use periodic retrospective assessment in order to improve our team practice and consider what we learned as a community. Now, I have also added individual writing assignments, to be completed at the end of each iteration. I want these reflections to be guided toward fruitful ends, and so I have brought in another pedagogic element that has intrigued me for the last several months: essential questions.

I first encountered essential questions (EQs) on Grant Wiggins' blog, and I blogged about this experience in the context of my advanced programming course. The primary purpose of EQs is to frame a learning experience. EQs have no trite or simple answers, and they are not learning outcomes, but they inform the identification and assessment of learning outcomes. With a bit of crowdsourcing, I came up with the following essential questions for my game programming course:

• How does the nature of game design impact the practices of game programming?
• How does game software manage assets and resources effectively?
• How do you coordinate interpersonal and intrapersonal activity within a game development team?

In reading about participatory assessment and the badges-for-learning movement, I came across Karen Jeffrey's HASTAC blog post. What she called “Really Big Ideas” seem isomorphic to EQs, and so I adapted her ideas in defining a rubric for evaluating reflections. I will be looking for reflections that provide the following:

• A characterization, with supporting evidence, of one or more essential questions of the course.
• The consequences of this characterization on individual and/or collective practice.
• The potential critiques of this characterization.

Deciding how to guide student attention is one of the most challenging parts of course design, and I recognize that by introducing these essays, I am reducing the number of hours I can expect students to spend on the development tasks at hand. However, these essays will afford conversation and intervention regarding reflective practice. They respect the authenticity of student work since, if done right, they should yield increased understanding and productivity from the students. This reasoning is similar to that given my proponents of team retrospective meetings as part of an agile practice: by reflecting on what we are doing, we can learn how to do it better. I have been encouraging my students to write reflectively, especially since starting my own blog; these reflective essays codify the practice and reward student participation.

The official course description for Fall's game programming course can be found at http://www.cs.bsu.edu/homepages/pvg/courses/cs315Fa13. I am happy to receive feedback on the course design, particularly the articulation of the essential questions, since they will be central to the students' learning experience.

Next time, I will write about the redesign of my advanced programming and game design courses, both of which involve turning to badges to incentivize and reward student activity.

Achievement-oriented grading for a game design class

I just posted a draft course description for the game design course I described in my previous post. After consulting with my co-instructor, we agreed to incorporate achievement-based grading. This is my first time designing this "gamified" style of summative assessment, but I'm pretty happy with the results. We articulated twelve achievements:

Seeker

Find an educational museum game aside from those shared by the instructors and write an analysis of it

Dabbler

Play a game outside of the canon and write an analysis of it.

Reflective Practitioner

Create an artifact (such as an essay or video) in the last two weeks of the semester to reflect upon the semester's experience, explicitly drawing upon game design literature

Nigh Completionist

Play and write analyses on 75%–99% of the games in the canon (or their approved alternatives).

Completionist

Play and write analyses of all the games in the canon (or their approved alternatives).

Detail-Oriented

Visit the Indiana State Museum as a guest and write about the experience.

Explorer

Visit another museum as a guest and write about the experience.

Quality Assurance

Participate in 75%–90% of the prototype evaluation sessions.

Quality Assurance Expert

Participate in over 90% of the prototype evaluation sessions.

Reviewer

Make significant comments on five other students' posted game analyses or essays

Expert Reviewer

Make significant comments on ten other students' posted game analyses or essays

Socializer

Attend a game-related community event and write about it

The current draft states that earning ten or more achievements merits an A, nine is an A-, eight is a B, and so on.

There are three parts of the course that we did not frame as achievements: a short student research presentation, five iterations of prototyping, and a rulebook requirement for student-designed games. These three elements are necessary for the course to work, so instead of framing them as achievements, we incorporated letter-grade demerits for failing to accomplish them.

In our discussion of the achievement-based grading, we noted that it's very similar to contract grading: a student can essentially choose their level of commitment to the course based on the grade they want to earn. All of the student-created artifacts (including writings, designs, and presentations) are subject to both peer and expert formative evaluation—a reflection of the studio-based learning I use in my Computer Science courses.

As always, feel free to share your thoughts in the comments.

EDIT: See this post for a reflection on the efficacy of this approach.

La Fin du CS222: An Assessment in Three Acts

In December 2010, I wrote about an experimental approach for assessing a course's effectiveness, using convergent and divergent exercises and mind maps as a replacement for traditional written exams. Since then I have used this approach a few times, with only minor alterations.

For last Fall's CS222: Advanced Programming final, I was inspired by a recent purchase to present the assessment to my students as La Fin du CS222: An Assessment in Three Acts.

Act I, Scene I
Each student constructed a mind map on the topic of programming. They had done the same exercise on the very first day of the semester.

Act I, Scene II
In this scene, I distributed to the students their original mind maps back to them. The main source of conflict in this scene was that it was the final assessment of how well I had learned their names.

Act I, Scene III
Each student wrote an essay comparing and contrasting the two mind maps.

The essays were generally well-written and insightful, especially those where there was not much change. One student commented, "Now I know that this random assignment we did at the beginning of the year was not so much for you but for us." This is great evidence of metacognition, that the student has learned to think about the value of reflection to himself rather than to some arbitrary authority.


Act II, Scene I
I hung my giant, flip-chart-sized sticky notes and told the students that we were going to list, in twenty minutes, all of the things we learned this semester that were somehow connected to the course. I briefly explained that this was a divergent thinking exercise, meaning that anything could be listed and there would be no debate or criticism at this stage. We ended up with 119 items, which may be a new record.

Act II, Scene II
I asked the students whether there were any items on the list that should be consolidated. Many suggestions were made, though not all were chosen by the majority. After this process, we were left with 113 items.

Act II, Scene III
Each student was given four stickers, and they were asked to place these by the four items that they found most significant. It was made explicitly clear that significance was subjective.

Act II, Scene IV
We identified at the top elements by vote, which are given in the table below. I have provided clarifying links for the curious.

Item	Votes
Object-oriented design	19
Coding conventions	14
Agile Manifesto	14
Effective Java	11
Test-Driven Development	7
Stack Overflow	7
User Stories	6
Mercurial	6

Act II, Scene V
The students were challenged to write an essay about what they actually did to learn these items that the community had chosen as the most significant.

I believe that it is important to help students understand the connection between what they personally do and what they have learned. Last year, the first time I did an exercise like this, we did another round of divergence and convergence to determine what actions helped learning. I was not very happy with the results because they became too generic: I was hoping for more specifics than "the project," for example. Unfortunately, most of the essays were written in generalities rather than specifics. In part, I think this is human nature: if we write in specifics, we make statements for which we are accountable and that may be shown to be false. Writing in established generalities, such as "I learned a lot by working on a team," is certainly easier than identifying specific times, places, and situations with the team that led to learning about a particular topic. I realize now that, if I want the students to write about specifics, I need to provide more explicit guidance to this end. I also need to narrow the focus: taking the top ~10% of items is too many: even though students were told they could pick from among the eight listed above, many wrote vague statements about all eight or a majority of them, as if they would get points for coverage rather than quality.

Some of the essays did contain interesting insights. My favorite comment was on the hassles of merging in Mercurial. We had previously discussed in class how tool-supported merging could be frustrating, but that it was a necessary evil—and actually not evil at all compared to the manual, ad hoc alteratives. This student identified how the frustrations of merging made him focus on learning encapsulation, because he realized that the better encapsulated and more modular his team's system was, the fewer times they would have to merge. Excellent!


Act III, Scene I
Cinnamon Rolls, courtesy of my lovely wife.

CS315 Final Assessment: Games about Beneficence

For the last eight weeks of the Fall semester, the eleven students in my CS315 Game Programming class had been working as one team, following Scrum to complete a version of Morgan's Raid implemented entirely in Unity. The team worked very well together, as I have written about previously. At the end of the semester, they had produced a reasonable facsimile of the Morgan's Raid release, given our constraints.

Regardless of their collective accomplishments, I still have to assign individual grades to each student, and I take this responsibility seriously. We used self- and peer-assessments during our four two-week sprints, and these helped me approximate students' individual contributions to the project. However, I still felt that I was missing hard, individualized evidence, and so I presented my students with a novel form of final assessment that we called Mini Ludum Dare.

Ludum Dare is the Rapid Game Development Community, most well known for hosting regular 48-hour game development competitions. Each competition begins with the announcement of a theme, and all games must incorporate that theme. For our Mini Ludum Dare, I told the students to spend no more than four hours on digital prototypes, my rationale being that I wanted to see what they could get done in about a half-day's work. This seems like an appropriate assessment, that at the end of a game programming class, one can individually produce a proof-of-concept for an idea. Because I was grading strictly on technical issues, I also provided a "default" game design that students could use if they were not feeling inspired by the theme. They were allowed to complete the prototype in Unity or Slick, these being the two primary technologies of instruction for the semester.

The theme was "Beneficence." This can be interpreted in a few ways, the most obvious connection being to the statue who also graces Ball State University's logo.

During our university-assigned final exam slot, the students each had a few minutes to show off their creations. It was inspiring to see how creative our Computer Science majors could be! The Java-based solutions are a bit tricky to get online, but with some coaxing (and peer pressure from the community on the department's Facebook page), the Unity-based games are online. Here are some links:

• Jumpin' Beneficence by Jason Buie
• ZombBenny by Thomas Mast
• Beneficence: The Hunt for Knowledge by Phil Wingham
• Beneficence the Killa by Craig Bavender
• Whack-a-Benny by Matt Waters

A pretty amazing selection of game prototypes built in only a few hours each. Nice work, gentlemen!

Teaching Assessment Culture to Undergraduates

My CS222 students are in the middle of their six-week projects, in which they just finished the first of two required iterations. There are three teams, with 4–5 members per team. Last time I taught the course (which was also the first time the course was taught, by me or anyone else), I continued to give rigorous assignments throughout the six week project. This time, I decided to try turning the accountability structure on its head and making the students think about how they would achieve the learning objectives of the project. This inversion of control was designed to foster reflective practice and metacognition.

At the beginning of the project, the students were told that they would have to display competency in the following concepts and technologies through their six-week projects:

• Requirements analysis
• Unit testing
• Acceptance testing
• UML: sequence diagrams, class diagrams
• Design patterns
• Architecture: model-view separation with MVC or Holub-style model-view-presenter
• Logging
• OOP: encapsulation, polymorphism
• Iterative and incremental development
• Distributed version control with Mercurial
• User-centered design
• Programming conventions
• Application of expert tips (e.g. Effective Java, Pragmatic Programmer)
• Estimation of time to complete tasks, including feedback for improvement
• Effective team communication
• Technical presentation
• Reflective practice

Several of these had been discussed in the preceding eight weeks of the semester, but many had not.

Each team had to create a "learning objectives document" and share it with me via Google Docs. The document was required to express how the students intended to show competency in each of these areas. They were instructed that this would be a living document: throughout the semester, the teams should come back to the document and consider how they would demonstrate competency.

For the first three-week iteration, the documents were rarely updated by the teams. I was using in-class meetings to introduce topics from the list that the students had not yet seen, scaffolding these with some light readings while the students focused on getting their first milestone delivered. Immediately following the milestone deliverable, the students were given the assignment to update their learning objectives document, ensuring that all items have clear and explicit plans for how the team intended to demonstrate competence.

As I looked over their submissions, at first I was disappointed that there was so little of substance. For example, under the topic of "effective team communication," most of the teams said something like, "we get together at regular times to work as a team." In my comments on their work, I suggested that they had missed the target: working together is a requirement, but learning to work together effectively is a learning objective. To show the latter requires evidence.

I believe it was that particular case—though certainly not in isolation—that made me realize a connection between what I was asking the students to do and what I myself have to do in terms of assessment. I will give some background in order to explain the case.

It was about three years ago, while serving as chair of my department's Promotion and Tenure Committee and therefore also representing the department to the College of Sciences and Humanities Promotion and Tenure Committee, that I read Scholarship Reconsidered and Scholarship Assessed. I loved these books, finding in them clear articulations of my intuition. I have used the framework from Scholarship Assessed within much of my work with undergraduates, following the principle that if the point of higher education is to be exposed to scholars and scholarship, then having students become scholars themselves is the best way for them to understand it. These books also made me very cold on the devious connotations of "research," as used throughout the institution of higher education.

Since last year, I have been departmental assessment coordinator. My university, like most others, is attempting a cultural shift towards more regular, formal, and fruitful assessment. In fact, the Future of Education Task Force suggested that we become obsessed with assessment and continuous self-improvement. This is a good thing, and I wholeheartedly agree with the mission. The downside is that I have had to attend umpteen workshops and meetings that re-introduce the fundamentals of assessment, presented to dubious or antagonistic faculty (who, I can only presume, have good intentions but see no incentive to change decades of practice). I have also served on the College of Sciences and Humanities Curriculum Committee, which means I have read tens of assessment plans for new and revised courses. Suffice it to say that I've been bumping into assessment pretty much everywhere I go.

Now, when I look at the task I gave my students, it was clearly an assessment task—yet, as it was in the guise of teaching rather than service or research, I had not consciously connected it to my past experience. Looking at how the students tried to approach the task, I see the same kinds of confusion that is common among faculty and staff who have not adopted a mindset of incremental and iterative development, a perspective that views problem-solving as a form of model building that requires positive feedback loops.

This insight made it easier for me to evaluate their work as novice assessors. I do not think that I adequately addressed how assessment works—that is, how initiating and fostering a feedback loop of self-improvement will pay off in pragmatic terms. The students' learning objective documents did not adequately provide evidence, but without seeing the bigger picture, they had no reason to believe that evidence was worth gathering. This might be best exemplified in a student's response to a note I left in the document asking for more evidence: the student modified the corresponding part of the document and then asked, "Does this satisfy your requirements?" My Socratic response was, "Does it help you demonstrate whether you have met the learning objective?"

While there have been some stumbling blocks in this approach, I am happy with the overall shape of it. Part of my current research is an investigation into the role of writing in STEM classrooms, and this CS222 happens to be part of the experiment. I will be teaching the class again in the Fall, and I am now considering how such a learning objectives document might be introduced earlier in the semester. This should help students get the idea that they are responsible for their learning, putting them on the path of reflective practice as early as possible.