What We Learned in CS222, Fall 2013 edition

For several semesters, I have been using the final exam to collect information about what students value having learned in Advanced Programming. Students work together to come up with a list of items they learned during the semester, and then they vote on which items were most important to them; full details on this approach can be found in my January 2012 blog post and my 2013 SIGCSE paper. I first used this format in Fall 2010, and I shared results from Spring 2013 in my post from May 2013. This table summarizes the top items from four different semesters:

Most Popular "What We Learned" Results by Semester

Fall 2010	Fall 2011	Fall 2012	Spring 2013
Team programming	Object-oriented design	Clean Code	DRY
Test-Driven Development	Coding conventions	Teamwork	Clean Code
Use of software libraries	Agile Manifesto	SRP	Build one to throw away
Refactoring	Effective Java	Class design	Refactoring
UML	Test-Driven Development		Naming Conventions
Design patterns	Stack Overflow		Research before acting

It might be worth noting that the first two columns used Joshua Bloch's Effective Java as a primary text, and the latter two used Robert C. Martin's Clean Code.

This past Fall was my first experiment in coupling achievement-oriented assessment with essential questions, maintaining my usual emphasis on reflective writing; I wrote a bit about that experiment just the other day. Looking at the results from Fall, here are the most popular "What We Learned" statements:

Item	Votes
Clean Code takes practice	13
Pair programming	10
It's OK to fail or start over	6
How to reflect on work	5
User stories	5

It is interesting to me that instead of simply saying they learned "Clean Code" as in previous semesters, these students agreed that Clean Code takes practice. It may be just anecdote, but this was the first semester that I allowed resubmission of work—part of my attempt to encourage mastery learning. Although I frequently felt like my feedback was not being heeded during the semester, perhaps this indicates that the resubmission policy helped students learn something quite valuable: not just the idea of Clean Code, but that putting these ideas into practice is difficult.

The middle row—it's OK to fail or start over—may be related to the resubmission policy. However, I suspect it is more directly related to the final projects. There was more than one team to whom I gave the same advice after two or three iterations, only to have them still not follow it. Some teams made significant platform changes between iterations, changes that forced them to abandon large parts of their implementation. I think that this is an important step in the students' maturation: they likely never had to throw away working code before, nor did they have to deal with a changing definition of "working."

I am pleased to see "How to reflect on work" was one of the top items, since the students spent a great deal of time during the semester writing achievement reflections. Although I have changed the balance of achievements and reflections for the Spring, as described in my previous post, I hope that this Spring's group will have a similarly positive experience regarding reflective practice.

Tweaking achievements and reflections

Before the start of the Fall 2013 semester, I wrote about the changes I was incorporating into CS222, the Advanced Programming class. With that semester behind us, I can say that it went well, although not without the need for tweaks. For the Spring, I am keeping the fundamental structure. Here is a summary and justification for the tweaks.

I am scaling down the number of achievements required for each grade level. The original plan in the Fall was two achievements per week. However, students were not as adept as I had hoped at aligning achievements and projects. The intention was that students would earn achievements by doing their projects, but the majority of the class tried to do achievements separately from the projects. In the Fall, we ended up making a change midstream, reducing the number required during the nine-week project to only one per week. For the Spring, I'm pulling back a little bit more, expecting two achievements per week during the first three weeks, then only one per week during the two-week "warm-up" project and the nine-week major project.

Another change addresses the intended achievement-project synergy: restricting the code that can be used to complete achievements. When working toward achievements, Fall's students were permitted to use any code to which they had a license. In practice, I had students submitting trivial modifications to terribly-conceived CS1 and CS2 projects all the way to the end of the semester. In Spring, students can use any such code during the first third of the semester, but once they start their nine-week projects, they must use either that code or public open source code.

I have also changed the how and when reflections are required. In Fall's section, students wrote one reflection per achievement. I am sure that the best students—those on the road to being reflective practitioners—found this fruitful, albeit time-consuming. The students with worse writing skills spent inordinate time on these reflections, time that should have been spent understanding the material instead. Their action was justified, perhaps, by the fact that reflections were a significant part of the course grade. Also, I allowed resubmission of the reflections, but I think my attempt to encourage mastery learning instead fostered the"submit and hope" approach that I find so endemic and disdainful. In the Spring, the students will instead be asked to write one reflection per week, and this reflection may focus on any course-related experience: achievements, in-class discussions, project team discoveries, and so on.

The structure of the reflections will change as well. In the Fall, students were asked to do three things in their reflections: characterize an essential question, describe implications to practice, and consider alternative perspectives. Many students struggled with these all semester. Although I believe that tackling these problems is good for students, there is a limited about of time students can devote to this class. Hence, I have reduced and rearticulated the requirements: in the Spring, students will be asked to address one of the essential questions ("characterize" caused undue problems) and describe implications to practice.

Despite the addition of achievements and structured essays, perhaps the most uncertain of my Fall experiments was in not grading the final projects. As mentioned above, the course design was supposed to foster students' intrinsic motivation to work on the projects, which would provide context for the achievements. My ambitious designs failed to account for the pragmatic, goal-directed nature of the undergraduate! Students realized that the projects were not graded, and many students found this unmotivating: they wanted their work to be recognized directly in their grades. I will be returning to my conventional approach, grading both the projects and their ancillary artifacts such as presentations and a final report.

A final significant change that I expect to bring vast improvements is getting off of Blackboard. We used Blackboard's wiki in the Fall as a place for students to post and share their achievement artifacts. The wiki is ghastly, and because it was so difficult to give targeted feedback, we ended up with a very poor signal to noise ratio. Often, a student would post something that was wrong, and I would give formative evaluation to that student, encouraging revision; the student would not revise, and within a few days, I would receive several similar submissions from students who assumed the original was correct. What a mess. In the Spring, we're going to use Google Drive instead. I am uneasy about requiring students to register for a Google Account in order to participate, but most have them anyway, and the technology stack is so vastly superior that I'm going to try it. I will set up a shared folder that will contain one document per achievement, and students will post their artifacts within those documents. My feedback can take the form of comments, which are visible to all readers. I will have a bit of manual work to translate these data into Blackboard's gradebook, but that's worth it to help close the gap between my feedback and peer learning.

I finished up the course description this morning and linked it to my Web page. The achievements themselves are basically the same as the ones I designed last Summer, with a few notable additions that reward technology (integration with relational databases and structured data parsers) as well as community-building (interviewing alumni and senior capstone teams). Each achievement has also been clarified as to what exactly the corresponding artifact is—some had been vague in the previous iteration.

The semester starts in just a few days, and I'm looking forward to meeting the new crop of CS222 students. I am not looking forward to teaching at 8AM and Monday's high of -6°F, but I am trying not to complain.

But seriously, 8AM and -6°F high temperature.

A semester in five burndown charts

It's the end of the semester here, and I am looking forward to writing a bit about the semester once its behind us. In the meantime, here's a visual vignette about my Game Programming course. We had a great team of students working on original video games with a community partner. The semester was divided into five Sprints, following the principles of Scrum. We used burndown charts to track progress throughout the Sprint, plotting actual hours remaining (green) against steady progress (blue).

I find that these five images tell an interesting story of a team's progress. The students had never worked together before, and most of them had never been part of an agile team. They were learning not just technical skills but also professional and social ones. The charts illustrate this learning during the semester, as they improve with their abilities to estimate, to plan, to deliver technical results, and to respond to changes.

That bump in Sprint 2 is interesting. The team recognized that many problems in the first sprint came from a combination of procrastination and poor estimation. They started in on their tasks right away in Sprint 2, and when things didn't go as well as hoped, they blew up the estimated hours remaining on those tasks. It turns out that the tasks weren't as bad as they feared, and as they worked through their problems, the estimates dropped back down to more accurate levels.

The erratic path in Sprint 5 illustrates that the team underestimated the effort required to polish the games for public release. Interestingly, the burndown chart was still useful even though the estimates were quite wrong. The team could look at the chart, which said they were dozens of hours ahead of steady, while knowing that there was still significant work to be done. This discordance forced them to re-evaluate their tasks and estimates, to reconsider the sprint's plan, and eventually, to successfully deliver on every user story.

A Concept Dependency Graph Analysis of Dominion

Just five years ago, Rio Grande Games published Donald Vaccarino's deck-building card game, Dominion, and it took the designer game world by storm. In short order, it seemed that every publisher was riffing on this new mechanic, and now, deck-building is an established game mechanism in any designer's toolbox. I include Dominion on my list of recommended games in my game design course because of its popularity and its influence in game design: this game provides the metric by which other deck-building games are judged.

Three recent situations inspired me to look at Dominion more critically—particularly, how one learns, teaches, and conceptualizes the game.

First, one of my game design students wrote a critical analysis of Dominion, and a significant portion of her analysis was based on the opinion that the rulebook was unclear. Specifically, she and her opponent were unable to determine when they were to discard their cards and draw new ones. I was surprised at this, in part because I had a student last year who upheld the same rulebook as a pillar of clarity. It may be worth noting that last year's student self-identifies as a "gamer," whereas this year's student had never heard of Dominion prior to taking my course. In any case, this was not the result I expected.

Second, I was recently with a group of friends who were teaching Dominion to a new player. They began by explaining actions and the chaining of actions and the strategic importance of this tactic, but before they had even taken any cards out of the box. This struck me as a particularly awkward way to introduce the game, since concepts such as "you're building your own deck" were not yet discussed, and it clearly was not resonating with the new player. In fact, his questions revealed that despite several minutes of accurate rule explanation, he had not built a coherent mental model of the game. This suggests to me that it wasn't a problem of accuracy, but of scaffolding—that is, giving the right information at the right time for the learner to advance.

Third, I was retelling the stories above to my lovely wife, and I explained that I thought it was important to focus on core gameplay when teaching a game. I justified that you cannot make sense out of peripheral game design elements without understanding the core, and therefore you should teach games from the core outward. She pointed out to me that sometimes she gets confused when I explain games this way because I don't give enough "big picture." Touché.

This got me thinking about the relationship between core gameplay and learning, and more specifically, how the dependencies among game concepts might reveal effective means for teaching the game. While many designers talk about "core gameplay," there is not consensus on where one draws the line around what is "core" and what is not. I decided to try mapping out the concepts required to play a game of Dominion as well as the dependencies among them, and this is what I came up with:

Concept dependency graph for Dominion.

This is a concept dependency graph, where the nodes represent gameplay concepts and the edges represent dependencies. That is, given A→B, one needs to understand concept B in order to understand concept A. I will refer to terminal nodes with no dependencies as atomic concepts. I am only considering the original Dominion game—no expansions—and only those kingdom cards recommended for new players. I also omitted concepts that were not relevant to gameplay, such as the use of randomizers or the determination of starting player.

The graph was drawn using dot, part of the graphviz project. Dot generates layered graphs (that is, each node is in a discrete layer) and follows automatic graph drawing heuristics such as minimizing crossings, minimizing edge length, and maintaining aspect ratio. A quick look at the graph shows a potential "core" of the game emerging from the atomic concept "Each player has their own cards."

Concept dependency graph, detail of potential "core"

If someone had never played a deck-building game before, I think this sets up a compelling description: it's a game in which you have your own deck of cards from which you draw a hand, then you play actions and buy new cards from stacks on the table, and you cycle your discard pile back into your deck when it is exhausted. There's no mention here of what those actions actually are, or the "cleanup" phase, or even victory condition, but I think it's fair to argue that these aren't part of the core gameplay. Keep in mind that "core gameplay" is not rigorously defined; one could just as well argue that the victory condition should be in the core, and be justified. That is, I'm not saying that this is right, but that it satisfices.

The concept dependency graph has three atomic concepts: each player has their own cards; there are stacks of cards in the center of the table; and there is a trash pile. Of these, the last is almost certainly not part of the core gameplay: it's an design element that permits a small number of kingdom cards to permanently remove cards from a game. Like the numerous Dominion expansion, it is a design element that permits the introduction of more action variations. It is interesting that the computer-optimized drawing of this graph put the trash pile atomic concept on the periphery of the drawing, where as the fact that there are stacks of cards ends up "wrapped" within other dependencies.

Choosing only the ten basic kingdom cards allowed me to ignore Curse cards in my concept dependency graph. However, Moat is included in this set, and it required adding several concepts and dependencies. Without Moat, it doesn't matter that Militia is an Attack card, and so that concept could be elided. Moat is a Reaction, and so that requires an explanation. Moat is also the only card in the set that has two different actions on it, from which the player must choose one; in all other cases, the player does everything on the card when it is played. This provides some justification for not including Moat as a beginner card, despite its gameplay utility in foiling Militia. It would be an interesting study to compare beginners' ability to learn the game with and without Moat to see if there was any significant difference.

It is noteworthy that the "ABC" concepts—that your turn comprises Action, Buy, and Cleanup—are relatively deep in the graph. You cannot really make sense of ABC unless you know what Action, Buy, and Cleanup mean. However, my experience shows that it still provides a memorable peg on which to hang your hat when teaching the game. That is, I propose that the mnemonic is there because these are not atomic concepts. Similarly, the atomic concepts manifest in directly observable configurations: without knowing anything about how the game is played, one can see that players have their own cards (based on players' handling them and placing them in easy reach), you can see stacks on the table, and there is a single "Trash" card among these stacks.

I knew I wanted a hierarchical drawing of the graph because I was looking explicitly for layers of concepts. I was curious, however, as to whether a force-directed drawing would yield any further insight. The result of running the graph through neato is given below. There's no new insight to be had from this visualization, at least not as relates to finding a potential core gameplay.

Force-directed drawing of the concept dependency graph

The concept dependency graph provided an interesting tool for analyzing Dominion, particularly with respect to the search for core gameplay. I am curious as to whether this tool can be used on other existing games to similarly fruitful ends or whether this is a coincidence. What I have shared here is the third major version of my concept dependency graph for Dominion: the previous versions did not induce any visual "core", but they also contained what I considered to be mistakes in the concept and dependency articulation.

This project has made me wonder whether other graph formalisms might reveal new insights into game design. A propositional network, for example, permits separate reasoning about cards and the having of cards. I did a bit of work with SNePS in graduate school, and it might be a useful tool to deploy here. I know that Machinations presents a formal language for representing game design elements, but I know relatively little about it; it may provide some insight into the learnability of games and core gameplay as well. I am curious to hear from anyone who has tried either tool toward these ends.

If you find any other uses for this formalism in studying game design and learning, or if you know of any related work in the literature, please leave a note in the comments. Thanks for reading!

Two-Week Project Showcase

The focus of my sophomore-level Advanced Programming course (CS222) is a large project at the end of the semester. In the past, this has been a six-week project, delivered in two three-week iterations. It is a project of the students' own invention: they have to pitch a project to satisfy a set of constraints I provide. This semester, I decided to expand the project to nine weeks with three iterations. This goes hand-in-glove with the revised, achievement-oriented assessment model that I am using this semester.

Most students have never worked in a programming team prior to CS222, much less defined their own projects. To warm them up, I give a two-week programming project before starting the big project. The two-week project is done in pairs, and I provide the requirements analysis as user stories. This provides a model for them to follow when they pitch their own projects using user stories.

This semester, I gave a new two-week project that was inspired by my 2013 Global Game Jam project. The students were challenged to write an application that takes a word from the user and then hits Wikipedia's API to determine the last modified date of the corresponding page. The curriculum provides no prior exposure to Web programming or networking, and so I provided a very short sample program that demonstrated how to open a URL in Java and read the stream. This project touches on some very important ideas, including Web services and structured data.

In the past, I have evaluated the two-week project in a rather conventional way: I provide a grading rubric at the start of the project, then I check out students' projects from the department's Mercurial server and give each pair a grade. I wanted to do it differently this semester, in part because of the achievement-oriented assessment. The two-week project provides a vehicle for students to earn achievements and write reflections: I'm not evaluating the project itself but rather how students use it to explore course concepts as articulated through the achievements and essential questions.

I decided to devote a class day at the end of the two-week project to a showcase. Each pair had to have a machine running a demo alongside a summary poster. We rearranged the classroom, moving all the tables to the perimeter and clustering the extras in the center. In order to foster peer interaction, I printed some sheets whereby students could vote on which team had the best UI, the best poster design, the cleanest code, and the most robust input handling.

The students enjoyed this format. There was an energy in the room as the students explored each others' solutions, talking about implementation strategies and UI decisions. A few students had trouble getting their projects working at all, and I heard one student say how disappointed he was, because it left his team unable to fully participate in the activity. This represents positive peer pressure and project orientation, which can be starkly contrasted against instructor pressure and grading-orientation.

I had recommended two strategies in class: using Joda Time to handle time formatting and using Java's DOM parser to deal with Wikipedia's output. I was surprised to see that almost every team used Joda Time (and used it to earn the Third Party Librarian achievement) but only one team used DOM. Every other team read the output stream as a single string and then searched it using Java's String class. This provided an excellent opportunity to teach about input validation. My sample Wikipedia URL queried the "soup" page for its last modified time, and the result looks like this:

<?xml version="1.0"?>
<api>
  <query-continue>
    <revisions rvcontinue="574699285" />
  </query-continue>
  <warnings>
    <revisions xml:space="preserve">Action 'rollback' is not allowed for the current user</revisions>
  </warnings>
  <query>
    <normalized>
      <n from="soup" to="Soup" />
    </normalized>
    <pages>
      <page pageid="19651298" ns="0" title="Soup">
    <revisions>
      <rev timestamp="2013-09-27T05:20:10Z" />
    </revisions>
      </page>
    </pages>
  </query>
</api>

Keep in mind that this is coming in as one continuous stream without linebreaks. Aside from the one group that did an appropriate and simple DOM lookup, students used String#indexOf(String) to search for "timestamp=", and then did manual string parsing using that reference point. This approach works for most cases, but it opens the application up to an attack that I'll explain in the next paragraph, giving the reflective reader a moment to consider it.

If you ask the application for the last modified info of the Wikipedia page "timestamp=", you get a phenomenon similar to an SQL injection attack: the indexOf operation picks up an unintended location, and the manual string manipulations fail. I had seen this when meeting with a pair the previous week who were working on their Advice Seeker achievement. They had thought their solution to be rock solid, and they were appropriately excited when I showed them how to crash it. They became my covert hitmen during the showcase, crashing solution after solution by finding holes in string parsing logic. So, while few students took the opportunity to learn XML parsing in the two-week project, maybe they learned something even better: the embarrassment of doing it the lazy way and seeing your system go down in public!

When I explained to the students what were doing at the showcase, I had expected teams to show up at their stations when I came by. However, it seems they were so excited to see each others' work that they didn't think about this. Since not every team had a person at their station when I came around, I was not able to give my expert evaluation to each group, nor was I able to model for the students how to give critical feedback. On the other hand, the students got a lot of peer feedback and I was able to meld into the group, becoming just one of many people interested in seeing demonstrations and code. I am not yet sure if I would do this part differently next time or not.

One aspect that is still unclear to me is the extent to which students were working for intrinsic motivation versus extrinsic reward. I was approached after class by a student from one of the teams whose solution was not working. During the hour, they had talked to other students and realized what they did wrong, which is an activity I certainly want to foster. The student asked, clearly in a state of agitation, if his group could fix their application even though it was due to be completed the previous night. I confirmed that this would be fine, and the student went away expressing joy and thanksgiving. I suspect his perspective was that he had just been given an opportunity to save his grade. What I really gave him was an opportunity to make his project work and feel good about getting it done, even if a bit late. I don't think he realized that there's no entry in the course grading policy for the two-week project, that the whole thing was just a fun context for us all to play and learn together. I hope that when he figures this out, he sees this as a reflective learning opportunity and not simply smoke and mirrors.

In conclusion, I am very happy with the showcase format. It was definitely worth using a class meeting for this event. I think this two-week project was particularly well-suited to the showcase format since it's fairly small, permits multiple solutions, helps students build better understandings of the modern computing ecosystem, and can have interesting failure states. Perhaps next time around I need to add an achievement related to XML parsing, since this seemed to promote students' use of Joda Time quite well.

(I have some nice pictures of the showcase that I took while standing on top of the teaching station, but I feel like I can't post them here without my students' permission. Sorry.)

Steering a student toward better reflections

We're in the third week of the semester already, and I have been happy with the changes I made over the Summer. I get the sense that my students like the achievement-oriented grading so far, or at least, they have not staged a rebellion. Many of the students found it difficult to understand the relationship between achievements and reflections. A few students didn't understand this two-step process at all—that you do an achievement first, then write a reflection. However, most of the problems came from difficulty understanding my reflection requirements, as stated in the course description:

A good reflection should address the following:

• Describe how your artifact and experience characterize one or more of the essential questions of the course.
• Describe the implications of this characterization on your own personal practice or, when relevant, your team's collective practice.
• Identify and address potential criticisms of this characterization.

When I talk about these requirements informally, I describe them as: characterize an EQ; implications to practice; and critique your characterization. I did not give the students an example to start the semester since past experience shows that, given an example, all solutions will look like that example. However, looking back on it, I think a simple example would have helped them to see that the reflection is not just the achievement restated, but a shift in focus and discourse.

To this end, I share the following story about a student who came to office hours for help with his achievements. I have the student's permission to share this story, though we'll keep him nameless. He was frustrated that he had not received any points for his reflections, and looking over his submissions, he had simply submitted his achievement artifacts again in the reflections slot on Blackboard. We started by looking at the first achievement he had earned—Studious—which requires reading Bill Rapaport's "How to Study" Guide and then creating a study plan.

First, I pointed out that the reflection has to happen after earning the achievement, and that the first step was to describe how earning the achievement helped him to characterize one of our essential questions of the course. I asked the student to talk through his experience of earning the achievement. He told me that he had been assigned to read Rapaport's guide last year: his instructor had worked with me before and enjoyed Rapaport's guide so much, that he assigned it in a prerequisite class to my class! The student took an informal posture and tone, and told me that since he read it before, he didn't expect to get anything new out of it, but since it was required, he re-read it anyway. As he thought, he didn't learn anything new. At this point, he corrected his posture and his voice turned "cold academic" as he started in with, "Well, according to the author, time management is an important skill for students, and being a student is like being a professional, and so he would say that we should be practicing time management too..."

I cut him off and pointed out how he was telling me how Rapaport might characterize an essential question, not how he would. I encouraged him to take a big step back and think—that is, reflect—on what he had extemporaneously told me about doing the reading even though he had done it a few months before, how he entered it with some negative expectations, and how these expectations were met. So, I asked, how does that experience help you characterize an essential question?

He paused and thought, looking over the list of essential questions. "Well," he tentatively offered, "I guess it means that a professional will sometimes try things he thinks will help, but they won't."

"Yes," I encouraged him. "So, what does that mean for you? That is, what are the implications to practice."

He quickly returned, "If I'm trying something and it doesn't work, I shouldn't get frustrated."

Aha! I congratulated him on the clear articulation of a discovered truth, grounded in his experience with the achievement. We reviewed what we had so far, and I pointed out that he could express it in two to four concise sentences; these reflections were not to be judged on length but on clarity of insight. We moved to the third step, which I have seen to be the most difficult for the students. How might an intelligent layperson critique your characterization? In this case, how might someone argue against this characterization—that sometimes professionals do what they think is best, and it doesn't help—and how would you address such criticism?

This was harder for him to address than the last point. He suggested that, since he's not a professional, he could be wrong. I pointed out that this was not a very interesting observation, and he could use that as a criticism any time he was addressing the essential question of professionalism. He seemed to have trouble pulling something together, so I improvised. One might say that the professional does learn something from this experience: he learns from the failure and tries something new. And if that's the best criticism we have—that what looks like failure is in fact progress—then we are happy to accept it and move on.

The student thought about that, then he asked, "But couldn't we use that as a criticism every time, too?" I laughed and told him that I would probably notice if he did, but he should feel free to use it for the first one anyway.

This was a positive meeting all around. It was good for me to sit one-on-one with a student who simply didn't understand my instructions: talking with a student about his struggles with metacognition grants more perspective than typing responses mechanically into Blackboard. He clearly benefited in the short term because he built a mental framework for approaching reflections, and I hope that the benefits extend beyond the course: the whole point of these reflective activities is to help students become better learners.

Revising Courses, Part III: Advanced Programming

Inserting reflective essays into my game programming course was straightforward, and the augmented achievement-based grading system of my game design course was an incremental improvement. Now, I will share the story of the biggest renovation in my Summer of Course Revision: complete renovation of my Advanced Programming course.

CS222: Advanced Programming has been around for about four years now, and I have taught it more often than any other. Several posts on my blog reflect on this course, including this one from the first semester's experience and this one reflecting on this past academic year. For the past several offerings, I have used the same fundamental course structure: studio-based learning, using daily or weekly writing assignments for the first several weeks, gradually turning toward a focus on a pair-programmed two-week project and then a small-team six-week project. I have been basically happy with the structure, but reflecting on my experiences, I identified the following pain points:

• The course was sequenced along a particular path of learning that did not resonate with all the students.
• There was not enough mastery learning: students would often do poorly on an assignment and clearly not revisit it, since weeks later, they still didn't understand the concept. This caused especial pain when these assignments covered technology that was core to my pedagogy such as distributed version control.
• It was easy to slip into a mode where I would talk for the whole class meeting. The studio orientation—through which students show artifacts that represent their learning, and these artifacts are subject to formal and informal peer and expert critique—was not guaranteed in my course structure: it relied on daily ingenuity rather than codified form. (The time and space constraints imposed by the university negatively affect studio orientation as well, but these are beyond my control.)
• It was not clear that all the team members were engaged in reflective practice when working on their projects. That is, I felt that there were people falling in the cracks, not learning adequately from the team projects, sometimes dues to lack of reflection and sometimes due to lack of practice.
• Students enter the course with high variance in knowledge, skill, experience, and motivation.

In addressing these pain points, I wanted to make sure I kept all the good parts of the course. Students work in teams on projects of their own design, using industrial-strength tools and techniques. Teams have to incorporate some kind of management techniques and give multiple milestone presentations, both of which remind students about the "soft skills" of Computer Science. There is a good overall flavor of agile development, pragmatism, object orientation with patterns, and the importance of reflective practice and lifetime learning.

If you've read my last two posts, you probably see where this is going! I decided to replace the daily and weekly assignments with a system of course achievements (a.k.a. badges), and students will write reflections that relate the achievements to essential questions of the course. The complete course description is available online, or you can choose to view just the achievements.

I have developed the following set of essential questions:

• What does it mean to be a "professional" in software development?
• How do you know when a feature is done?
• How do small teams of developers coordinate activity?
• How does a Computer Science professional use ubiquitous and chaotic information to be a lifetime learner?

As in my game design course revision, students' grades will be based on number of achievements earned and written reflections on those experiences. There are also four "meta-achievements" that can be earned by completing sets of regular achievements. These are leveled achievements and reflect four different potential paths of mastery: a Clean Coder has documented evidence of applying ideas from each of the first twelve chapters of the book; a White-Collar has demonstrated savvy at project management and presentation; an Engineer is moving toward an understanding of software architecture and patterns; and a User-Centered Designer has designed, evaluated, and iterated on user-facing systems. The introduction of these meta-achievements was partially inspired by an alumnus who, when I told him about this redesign, said that most courses were like a Call of Duty game, but that this was more like Deus Ex, and that it would benefit from having more quests.

I want to help students succeed in this kind of learning environment, and so the first month is designed to help them understand what it means for them to have more ownership over course activity. In the first week, I will provide a review of important CS1 and CS2 topics, focusing on the mechanics of object-oriented programming in Java. Knowing that student experience and comfort with this material is highly variable, I can use this week to focus on how to navigate the course structure. In fact, I will strongly recommend that their first achievement is Studious, which requires them to read William Rapaport's excellent How to Study guide and write a study plan for the semester.

We will still use Clean Code as a shared focus, but I have changed how I am scaffolding students' experience in integrating its expert tips. Previously, I had identified specific tips or chapters for students to read and then asked them to apply these tips to their projects—past or current. There was little evidence that this "stuck" with the students, as later coursework would violate the concepts supposedly learned. This was, in part, due to lack of mastery learning, where students would accept a low grade and move on without having learned the material. Furthermore, because we paced Clean Code readings and assignments through the semester, I would frequently encounter examples in class and in critiques that afforded application of a tip that the students had not yet studied. In the revised course, we will read the book relatively quickly, but then keep returning to it in informal expert critiques, adopting an iterative approach that is more apropos to how one learns and remembers these tips: a bit at a time, and a bit more next time.

In my game design course, students are required to present to the class to earn their achievements, but the enrollment in that course is half that of this one. I toyed with the idea of having my CS222 students post their work on the walls, a portion of the class each day, but I feared that this would have too many negative consequences. Instead, I am requiring students to post their artifacts to a wiki, and my intention is to review the wiki changes between class meetings to identify notable entries. This way, I can bring up the wiki on the projector and model an appropriate critical process. I can also insert mini-lectures as necessary to clarify misunderstandings. I am eager (or perhaps anxious) to see how the density of misunderstandings in this redesigned course, which encourages mastery learning through reflection, compares to that of my current status quo, which encourages correctness up front. We'll be using BlackBoard's wiki only because it's easy for all the students to find; it was frustrating to me during my experimentation that it has no wikitext editor. If the wiki turns out not to work for us, we can always move to an in-person poster-style presentation.

Since I will be doing more just-in-time teaching—reacting to students' insights and confusions—I have decided to expand the conventional six-week, two-milestone project to three milestones over about eight weeks. The students like this part of class the most, and I like the idea of adding another milestone, since this gives them another opportunity to learn from mistakes in tools, design, organizational structures, and presentation. For both this large project and the small project, I will provide common requirements that everyone must follow, such as using distributed version control. Previously, these things were simply worth some points on the project; however, if I want to focus on assessing their reflections while also requiring some shared technological experience, it's fairly easily done with a hard-and-fast requirement. With multiple milestones, there will be an opportunity to ensure each team is following the requirements. For example, if I see a team not using DVCS, then I can remind them.

I am eager to see how students take these changes. I expect to be able to blog about some of the day-to-day experiences, and I anticipate writing some academic papers on aggregate student performance in all these modified classes.

Here are the critical links again:

• CS222 Course Description
• CS222 Achievement List