Reflecting on CS222, Spring 2025 Edition

This semester's CS222 class was unlike any other. My department is once again involved in a 1+2+1 program with a Chinese university, meaning that Chinese students complete their first undergraduate year in China, their next two years in the USA, and then their final year back in China, culminating in dual degrees. It was a vibrant program many years ago, and I am glad to see that it has come back. In the Spring semester, I had all of the eligible 1+2+1 students in my section, which meant that half of my enrollment consisted of domestic students and half consisted of Chinese nationals. I used to have some in my classes, but I've never had half of the enrollment in a class be international students before. It meant that on most days, there were more native Chinese-speakers in the classroom than English-speakers. 

One of the things I realized this semester is how often I speak in idioms, using phrases whose literal translations don't have an obvious meaning. For example, I told one of the teams that they had gotten "in a pickle." Then I laughed and wrote that expression on the board, gesturing to it and pointing out how ridiculous it was. I explained that it meant that the team had gotten themselves in trouble, possibly of their own creation. 

Unfortunately, moments like this were not as powerful as I would have hoped. Turning to my class, the domestic students were smiling as they contrasted the literal and figurative meanings of the phrase, but almost all of my Chinese students were as they always are: eyes glued to their monitors or smartphones. A guest speaker's presentation allowed me to sit in the back of the room and verify my suspicion about what was going on: most of the students had their phones propped up on their laptops and were running voice translation software. They were not listening to the speaker in any significant sense but reading in Chinese what it thought the speaker was saying. Occasionally, a student would switch to the laptop to visit a site or verify a term, but mostly they were reading real-time translated transcriptions of what was spoken. Not all of them did this, but most did. Some listened and took notes. One watched Chinese television. In this way, they are not unlike domestic undergraduates.

As usual, we completed a major project lasting about 8 weeks, and it was split into three iterations. I gave lengthy feedback to each team's submission, and sometimes teams even read and responded to it. An unusual frustration from this semester was that none of the teams really nailed the final iteration. I expect 20% or fewer to get the process right the first time, and then maybe 40% on the second iteration, and I usually get 60% or better addressing the fundamentals by the third iteration. I didn't have that this time: each project had something fundamentally wrong that I had already pointed out to them in a previous iteration. One conclusion from this is that I may need to rearrange or remove some of the elective content from the class in favor of more supervised in-class practice. Another consideration, though, is that I hadn't accounted for the extra labor done by international teams. I strongly encouraged—but did not require—international project teams, and most of the teams were. This means that in addition to tackling the significant challenges of the course, which include trying to change habits and conceptual models around programming, these teams were also dealing with language and cultural differences. That work is real work but it was outside the grading scheme. When I laid out their final grades yesterday, I decided to add a flat boost to the international teams to compensate them for undertaking the challenge. The resulting grades looked more appropriate to me for how well I knew the relatively small class. If I were to teach another class with such a high proportion of international students, I would need to revisit the question of whether an up-front incentive might be fruitful.

Course achievements were once again counted as regular assignments rather than their own category, as I did last semester. I do think this works better since it lowers the cognitive load of the course. My department currently has unprecedented growth in student organizations, and I need to bring back some of the achievements related to those. 

Counting achievements as a regular assignment has helped students navigate that part of the class, but I have yet to crack the problem that many students who ought to resubmit work don't do it. The idea is that students resubmit work until they have shown an understanding of it, and I am not sure what impedes students from doing this. It's possible that it's too liberal for Canvas to handle. Canvas won't tell them that something is due because they can resubmit any past assignment once per week. I suppose I could make a separate weekly resubmission assignment, which may solve the problem that some students fail to understand what "one per week" means. It feels like catering to their LMS addiction to me, but maybe it's what they need to help them through the challenging content.

I'm still stuck on the question of whether the content of the course would work well as a portfolio. Take a principle like naming: I could require students to submit in a portfolio an evaluation of something they wrote before 222 (which I do as an assignment) along with a sample of work that demonstrates their following the principles. Such a portfolio would be a powerful testament to what they have learned, but I've struggled to figure out how to pace such a thing. When I first started using achievements in CS222, all the assignments were from a grab bag of achievements, and in many ways, it was a portfolio-style assessment. As I put more emphasis on the project, I had to put that aside. I mention it here in part because, as I understand it, the 2026-2027 academic year will see a transition from a 15-week to a 14-week semester here. That will require me to blow up the course and rebuild it since there's no way I could just trim a few things out and have the rhythm still match.

The last note I'll mention here is something from my planning document, where I keep short notes about how things went and what I want to change the following year. This year, I had several teams undertake projects whose nature did not fit the requirements of the final project. The reasons undoubtedly come from a combination of background, culture, language, and generational differences. To cut to the chase, I realized I need to make it even more explicit that the final project needs to do something computationally interesting: being a clever interactive app is not enough. Some of the projects students wanted to undertake would have been great in a course on web and mobile apps, but they were not good contexts for exploring TDD for the first time.

I will be on sabbatical in the Fall, which means I'll be stepping away from this and all my other regular courses for a semester. It will be a good chance to catch my breath, and I won't have to carve out a week over the summer to rebuild the course.

A student discovers the vacuousness of midsemester grades

Midsemester grades are epistemologically invalid. Yes, you can compute something but it doesn't mean anything. At midsemester, every student's grade ought to be an 'F' since they have not finished the course. Aside from that, any faculty worth their paycheck should be giving student enough feedback that they know how they are doing, what to do to improve, and how to get more help or practice. At Ball State, a rule was passed a few years ago that we have to give "midsemester grades" in all classes. Par for the course, nobody actually defined what "midsemester grades" mean.

At midsemester, I threw my students' grades into a spreadsheet blender and dutifully keyed the results into the terrible Web database interface. Because I use a form of mastery grading—where students can resubmit work until it receives full credit—it is not uncommon for students to have an F or a D at midsemester and end up with a B or an A.  Furthermore, the majority weight of a student's grade comes from work in the second half of the semester. I made an apologetic announcement on Canvas about having to file midsemester grades, explaining exactly what I did, that it should contain no surprises, and suggesting how students interpret their status. (It turns out, explaining to someone how to achieve their goals is the actual solution to the problem that the university is trying to solve through midsemester grades. The process is human and personal, not quantitative and automatic. No one should be surprised by this.) A few students expressed concern about their midsemester grades, but not many, which I take as an indication that perhaps a majority read the Canvas announcement. A professor can dream, anyway, since the alternative is despair.

But none of that is my reason for writing today. The previous two paragraphs provide the backdrop my tale. Pardon the metadiscourse.

As I explained in early January, I introduced a new achievement into CS222 this semester. Tool-Maker is earned by "satisfying your obsessions," and more particularly by doing the following.

Write a program to compute a student's grade in CS222 based on the policies described on this course site. You may use any programming language or environment that you like, but you must follow the practices of Clean Code.

I know of one student who is pursuing this achievement, and he has been working on it on and off for a little while, sending me occasional questions to clarify policies described on the course site. Today, he emailed me such a question. He asked whether the tool should compute one's grade as if there are no future assignments or whether it should compute one's grade and counting future work as zero.

Exactly!

He stumbled directly into the midsemester grade problem. He did not mention if his question was based on memories of my Canvas announcement, and so I suspect he simply ran across this as he tried to know whether his tool was correct or not. I told him that either way was fine with me. Practically speaking, I don't care, because the two goals of the assignment are to understand class policies and to get practice with Clean Code. My scholar sense tingles, however: once he has completed the work, I will have to ask him why he chose the path he did. One of them may simply be easier than the other, and I couldn't blame him for taking that one. However, he may make his choice based on perceived usefulness, and which model he finds useful might be a tasty bit of insight.

What We Learned in CS222, Spring 2017 Edition

It is time, dear reader, for the End of Spring 2017 Blog Posts!

I want to start with my traditional "What We Learned in CS222" post, wherein I share an overview of what my students believed they learned in CS222, as self-reported on their final exam. This semester, they came up with 104 items in half an hour, which struck me as kind of low, but it was also a talkative group: several had a rather long wind-up to stating the item that would eventually be written down. Each student was given five votes to identify the items that were most important to them. After voting, we gathered items with the same number of votes into bins. These were the top five—the items with four or more votes:

Item	Votes
Clean Code Naming Rules	9
Test-Driven Development Process	8
Single Responsibility Principle	5
User Stories	4
Test often	4

Interestingly, the first four on this list were the first, second, seventh, and eleventh items listed, while the fifth was the very last—the 104th.

Regular readers may recall that "Clean Code" usually comes up early and is highly voted upon. I was surprised when the first contribution from the students was not just the generic "Clean Code", but specifically the naming rules. It's true that students continued to struggle with naming rules throughout the semester, but I do not think that this was much different from any other semester.

I'm not sure I have other new reflections on CS222 that I haven't shared before, except that there were a few times during the semester that I realized too late I should have had achievements for out-of-class activities that I value. The first was the CS120 Art Show, which is run every semester by my friend and colleague David Largent. We use a media computing theme in CS120, and the Art Show is an opportunity for students to select their favorite work from each section and have it showcased in the Atrium on campus. Celebrity judges then give awards based on the visual design as well as the source code design. Most of my CS222 students went through this course, and it would be valuable to have them go talk to the first-year majors and minors about their experiences. Since it's valuable, I could easily make it an achievement.

Similarly, reading students' essays at the end of their third-and-final project iteration, a few mentioned getting feedback from Code Smash!, which is the closest thing we have to a CS club. Most that referenced it talked about getting feedback from upperclassmen, while one wrote with pride about how a particular aspect of his final project let him help an upperclassman. It got me thinking that participation in Code Smash is valuable and should be rewarded, and as I considered this, I realized the somewhat special place that CS222 students find themselves in: they know much more than the freshmen about programming, but they know much less than the seniors about the various subdisciplines of Computer Science. I can imagine adding some kind of achievement to reward either "reaching down" to help those lower in the curriculum, "reaching up" to get help from someone with more experience, or maybe a single achievement that takes the two experiences and invites reflection on the contrast. I am uncertain whether this would be the same as the Art Show idea or if it has a different spin, but it's something for me to consider when I do my annual course redesign.

A few days ago I wrote about the attendance problem in CS222, in the context of our roundtable evaluation. This has been simmering on the back burner, and I wonder if the achievement system provides an answer here as well. The truth is, I don't want students to come to class "because they have to." That's the wrong attitude, and it fosters disengagement and bitterness. Frankly, those students need to mature before they can succeed anyway. However, I can use achievements as an incentive, and I can perhaps close the loop between the in-class activities and the final projects or essential questions. I am thinking of an achievement that asks a student to reflect on three in-class activities they participated in, encouraging them to draw the lines between the specific actions of the activity and the goals of their project teams.

Next Fall, we're finally moving CS222 back to a Tuesday-Thursday 90-minute schedule, the way that it was the first 2–3 years it was offered. I have been asking for this revision for years, since 50 minutes is just not enough time to introduce an idea, engage teams in meaningful work, and have them share and reflect on their experiences. This means I will have to do some rather significant revisions to my course plan during my annual course redesign. I used to give daily assignments that were due Tuesday and Thursday, which caused some whinging from students but also kept everybody on strict schedule. I stopped doing that with the switch to MWF because the grading load was too high for me to keep up with. I am leaning toward bringing those daily assignments back as a replacement for the more broadly-defined weekly assignments I have been using for a few years. I expect I will post again over the summer as I sort out those details.

Thanks for reading!

CS222 Fall 2014: What we learned, and where we're going

The final exam for my CS222 class once again featured the construction of a list of what the students learned (or more properly, what they think they learned). They compiled a list of 111 items, and each student was asked to pick their top seven. Once all the votes were tallied, these was most consensus around these four items:

Item	Votes
Test-Driven Development	16
JUnit testing	12
Make realistic goals	12
You can learn a lot from failures	10

Transcribing the list is like a semester-in-review. There are some very powerful ideas that the students brought up, many of which only had one or two votes. Examples include "Nobody owns code in a group project," "Dirty code is easy to write and hard to maintain," "Be wary of second-order ignorance," and "Donuts solve most transgressions." I could tell stories about each one of these—and I know that if I don't write down the stories, the details will be lost to the sands of time. However, I also know that this is going to be a long post, so I will have to leave out some of these details. It's worth noting that some of the items in the list also embody lingering confusion. I just write down what the students say during this exercise, only asking for clarifications. Still, my teacher-sense was tingling when students offered items like "Instantiating variables," which shows a misunderstanding of terminology, semantics, or both.

The students were asked to write about what experiences led them to learn one of these items. I believe everybody chose to write about learning from failure, which turned out to be a good theme for the semester, as I talk about more below. One of the students, in his closing essay, pointed out that "make realistic goals" and "learn from failure" are the kind of thing you'd expect to hear from a weekend leadership workshop at a hotel, which I thought was an astute observation and brought a smile to my face. He himself acknowledged that these items are easy to say and hard to follow, and I found it uplifting to read about how so many students had transformative experiences around personal and team failures during the semester. My hope is that they integrate these lessons into better practices as they move forward in college and then in their lives as alumni.

Taking a step back from the end of the semester, let me set the context for this particular class. About two years ago, the Foundations Curriculum Committee approved a change to our introductory Computer Science class. We decided to adopt the combination of Media Computing, Pair Programming, and Peer Instruction that proved so successful at San Diego. The CS222 course that I have been working on for a few years comes two semesters after this course, and so this Fall I had my first cohort of students from this change. The change of context went along with a change of language, so whereas previously my students had two introductory semesters in Java, this group had a semester of Python and a semester of Java.

I was a bit surprised that these students did not appear to be any better or any worse prepared for CS222 with respect to maturity of programming. As always, they could generally hit the keys until programs kind of worked. There were some who still struggled with basic control structures, almost uniform misuse of technical terminology, and practically no understanding of either object-oriented programming or its syntactic elements. I suppose this means that our first round of data point to the change being a success, since I did not notice any real change in preparation, yet more students are sticking with the major. (Then again, it could just be the economy.)

I think I had slightly overestimated their understanding of Java in the early part of the semester. My intention was to give some warm-up exercises, but I think these were neither formal enough nor scaffolded enough. There were several days where I posed a challenge and said, "Try this for next time!" but—with shades of when I tried making the final project not worth course credit—because it wasn't collected and graded, I do not think many people really tried. For the Spring, I have integrated three formal assignments, one per week before the two-week project. Because these assignments are intended to form students' for the coming weeks of activity, I have decided to adopt a mastery learning approach here: students have to do the assignments until they are correct. (As I write this, I realize that there may be a hole here: right now, I have it set up so that the assignments must be correct to get higher than a 'D' in the course, but this means students might put them off. I think I will have to revise that before Spring, to actually stop them from moving on in the course somehow until the assignments are done, or put a strict time limit on how long they have to submit a reasonable revision.)

The two-week project in the Fall was a good experience, although I think the students didn't realize it until for a few weeks afterward. I gave them a good technical challenge, involving processing data from NPR's RSS feeds, and Clean Code with TDD was required. Of course, many students did not start the project when they should have, but more importantly, I don't think that anybody actually followed the requirements. Several students had projects that ostensibly worked, and they were proud of these, but they were horribly written. I was honest in the grading, which I think many of the students had never experienced before either. Many of them panicked, but then I pointed out to them that the project had no contribution to their final grade at all. This allowed us to talk honestly about the difference between requirement and suggestion, and it forced them to rethink their own processes. In their end-of-semester essays, many students came back to this as one of the most important experiences of the course—a real eye-opener. I am fairly certain this contributed to "learn from failure" being one of the top items of consensus in the final exam.

I realized too late in the semester that I had a problem with the achievement-based grading system. I had designed several interesting quests, which a student had to complete in order to unlock A-level grades. One of them, "Clean Coder," was designed to be the easiest one: it required the completion of several achievements related to Clean Code, which was required for the project anyway. However, it looked like it was harder, because it had more steps than the others. The other achievements had fewer steps required because I knew that students would be doing Clean Code activities as well. Sadly, the students didn't think this through, and I did not convey it clearly enough, with the result that nobody pursued the Clean Code achievements. Not coincidentally, many teams struggled with fundamental Clean Code ideas in their projects.

I also encountered something new this semester, which could actually be novel although I suspect it was previously under my radar. As in previous semesters, I allowed collaborative achievement submissions, since much of the work is intended to be done by the team collectively. However, it came to my attention that a few teams were assigning an "achievements person" who became responsible for doing the achievement work while the rest did other things. This is quite a reasonable division of labor, but it's not at all what I intended.

Because of the quest difficulties and the unintended division of labor, I made several changes to the achievement-based assessment system for the Spring. All achievement claims will now be made by individuals, which helps me ensure that I know each student earns their keep. I also reduced the number required to unlock different grade levels. However, the total workload should be about the same, as I am bringing in end-of-iteration reflection essays. Several semesters ago, I required both achievements and reflection essays, but I found this to be too much work. I find myself wanting students to tie their experiences more to our essential questions, and so I'm pulling the end-of-iteration essay idea from my studio courses into CS222. I think it should be a good fit here, although I am dreading the return of the inevitable "This feels like a writing course!" evaluations from students who don't recognize the epistemic value of writing.

I have also completely removed the quest construct. Many of the items that were quests are now simply achievements. The quests are fun and provide a nice narrative on top of the achievements ("I am becoming an engineer!" "I care about user-centered design!"), but they also bound the students too early to a specific path: a team who decided on one quest could not feasibly change paths to take another, which is unfortunate since the whole thing was really designed to be inspirational, not constricting. In the Spring, then, there will be no other barrier between B-level and A-level grades aside from the number of achievements. Realistically, it's the project grade that most influences final grades anyway.

Before winter break is over, I plan to make a checklist for the final project iterations. I don't know if students actually will read it or use it, but maybe it will help those teams who are working diligently but suffering from second-order ignorance. Common items that teams forget before the submission include removing compiler warnings, tagging the repository, and checking out their project onto a clean machine to ensure that all of their dependencies are properly configured or documented.

I incorporated self- and peer-evaluations into each iteration in the Fall, and these provided a throttle on how many points an individual could earn from a collective team project. The design, obviously, is to catch freeloaders. I used a similar system in my game programming course, where I also asked students to reflect on it explicitly, and that group was pretty evenly split between people who liked it, those who didn't, and the neutral. Although I did not ask students to reflect on these evaluations explicitly in CS222, I was surprised that it just didn't come up in most students' writings, and where it did, it was very positively. There were some teams where I think they even helped students to air some difficulties early and get over them. I still need to find a better way to help students recognize that, given a rubric, all I want is an honest response. I did hear students talking about giving each other "good" evaluations, and I tried to convince them that it wasn't about "good" and "bad", but about honest feedback. Perhaps it's an intractable problem because these evaluations contribute to a grade, and so inevitably students pragmatically see "good" and "bad" as having priority over team cohesion.

The course evaluations for Fall were nice to read, as the students praised the areas of the course that I think went well, and they provided constructive criticism in places where things were rocky. At least two students described the achievement-based system as feeling like it was "still in beta," but both were quite forgiving of this as well. I think drawing on video game metaphors here helped me, as students recognize that "beta" incorporates their feedback for the betterment of future players. Despite generally high praise for my course, I cannot get out of my head the fact that more than one student wrote something like, "Don't believe the people who say you are a monster." Two used the word "monster" specifically. These were all couched in grateful and kind ways, from students encouraging me to continue to be honest in my critical feedback. I suppose I must have a mixed reputation. I know I shouldn't let it get to me, but I cannot help but wish to be a fly on the wall.

This was a really fun group of students, and I think we did a lot of good work together. I hope they carry these big ideas of software development and craftsmanship with them as they move on. In my last presentation to them, I reminded them that I spent the last fifteen weeks holding them accountable to high professional standards, and that in the coming semesters, they will be responsible for holding themselves and each other accountable. I hope they do.

Some references:

• Fall 2014 course plan
• Spring 2015 course plan

Revising courses, Summer 2014: Advanced Programming

I finished revisions on CS222, my advanced programming course. As I wrote about last summer, this was one of the courses that I radically altered last academic year. The 2013–2014 version used achievement-oriented assessment, a two-week warm-up project, a nine-week programming project, and reflective writing. I enjoyed teaching it in this format, and while I believe it was fair, I also think it may have been too scattershot. The best students did amazing work, but many students struggled with the balance of responsibilities from one class. With a bit of sadness, I have decided to drop the reflection essays. They were always a bit ancillary: although they tied to the themes of the class, they did not share the same kind of project-orientation as the rest of the work.

Dropping the requirement for separate reflections has allowed me to expand the writing requirements within the achievements themselves. I will be keeping the policy that allows students to resubmit achievements, which I had not allowed on the reflections due to the overwhelming amount of my time this would have required. Focusing my grading efforts on the achievements, and seeing them as the primary non-programming writing artifacts, I will be able to give more direct feedback and, hopefully, see students improve their writing more visibly.

Similar to my game design course revision, I have pulled the achievement system out of the first few weeks of the class. A gentler introduction to the themes of the class will be had from a more conventional approach, with assigned reading, in-class activities, and small practice assignments. The achievements system will kick in during the two-week project, which precedes the major, nine-week project.

Last year, I probably came the closest I ever will to my hypothetical "your grade is the lowest grade you earn all semester" grading scheme. It was a good teaching experience, and it formalized—in the course structure—the idea that a good computer scientist needs to be balanced. However, it did lead to some stress and heartache at the end of one of the semesters. I cannot say much about it publicly, but the anonymous version is that a team did a crackerjack job on a project but had not met the requirements for a passing grade in other areas. This experienced forced me to reflect on whether or not this grading scheme really reflected what I believed and valued. Of course, every semester is an experiment, and so I will try something different this year.

In the Fall, students will be able to earn up to a C grade by simply doing the final project reasonably well. If a student gets up to a C-level grade in the final project, that's all he or she needs to get that grade in the course. This sounds "average" to me: if you can work together with a group of near-strangers, understand the rules of Clean Code, and make a passable project, you can get through the course. (Majors need a C− or better to move on in our curriculum—a policy I disagree with, but that's for another day.) This will permit achievements to actually represent something beyond the usual, something "good". So, a student must complete ten achievements in order to get a B-level grade. If a student wishes to show that he or she is "excellent," an A-level grade can be earned by additionally completing a quest. I am not sure that's the right word, but it's better than "meta-achievement," which I used last year. Some quests are completed by finishing coherent sets of achievements, but I have used this to also open up new opportunities. One quest is completed by having the final project be a contribution to an open source project, and another is completed by working with a real client. I am hopeful that these will be enticing to the students and encourage them, from the beginning, to take the initiative and do something ambitious. Even if they take a risk and fail, they won't "fail" the course, since they can always fall back on a lower—but still passing—grade.

Last year, I did not include formal peer evaluations into the grades as I had in the past. It was still clear to me who was contributing to projects and who was not. However, I will have about  twice as many students in the Fall as last Spring, and an order of magnitude more projects to monitor, so I am bringing back peer evaluations. The peer evaluation form comes from a 2011 blog post, and I have described precisely how it fits into the grading scheme on the final project specification. In the past, I have sometimes forgotten to tell students ahead of time how each iteration is graded, but this time I have laid out all the details. I have occasionally not counted the first iteration at all, since they are sometimes pretty rough. This year, I will be using a quadratic series: first iteration has a weight of one, second iteration has a weight of four, and third iteration has a weight of nine. This should ensure that students still do an honest job on the first iteration while also reflecting that they should be getting better at working together as time goes on.

This Fall marks an important change in the population of my students. In Fall 2013, the foundations curriculum committee enacted a change to our introductory programming sequence. It had previously been taught in Java both semesters, using a conventional approach (read that with a bit of a sneer). Last academic year, our students started in Python using the Media Computing approach, following that with a more conventional course introducing data structures using Java. The Spring CS2 course was a bit rocky, as to be expected with any such shift in a curriculum. I would expect the students coming into CS222 in the Fall to be particularly weak with Java, although this will become less of a problem as the wrinkles in CS2 get ironed out. Time will tell what kind of remediation I will have to do. The truth is that most of these students have been programming for such a small amount of time, many harbor deep misunderstandings about essential programming ideas when they come into CS222. The good news is that by having them work in teams on a real project, they can learn from each other and be motivated to do better. Indeed, this is one of the main outcomes of CS222, so that then these students can move on from our foundations courses and be productive in the rest of the curriculum.

Here are some links for reference:

• The course description for Fall 2014 and the final project specification
• Last year's course descriptions for Fall and Spring
• Last summer's post about Advanced Programming course revisions
• My reflection on using achievements during 2013-2014

Revising courses, Summer 2014: Serious Game Design

I recently finished revisions to my Serious Game Design course. These come in response to the changes I made last Summer, when I designed a rather complex system of achievement-oriented assessment. I was happy with how the system allowed students variability in how they approached the themes of the course. However, I may have been a bit too tricky in the design. I had designed grade incentives for students to follow a particular path, and of course, these bright Honors College students determined that path and followed it. That involved unnecessary indirectness, however, and a few students called me out on it in course evaluations.

Seeing it in that light, I plan to divide next year's course into two units. The first five weeks will be structured, with students doing common readings and activities. Last year, I encouraged students to play a variety of games and reflect on them, but I think this was a "you can lead a horse to water" problem. Yes, the students played some games they wouldn't have otherwise played, but often just once (which is insufficient for critical analysis) or sometimes incorrectly (which leads to inappropriate analysis).  Hence, the activities of the first five weeks will involve more reading and discussion of common experiences—including games we will play together in class, assuredly by the rules—and less divergent experiences.

The second unit of the class will be the design workshop. Last year, we spent five weeks coming up with various game concepts, and the following five weeks were spent iteratively prototyping one of them. The variety of game concepts last year was excellent, and they did increase in quality over time; however, in the last five weeks, many students got burned out or stuck on prototyping. I think they kept working on the prototypes only for the grade and not because they thought it was the best use of their time. This Fall, the plan for the last ten weeks is to allow students to present either concepts or prototypes each week. I have put a threshold on the number so that, for students to earn "good" grades, they need to iterate on a prototype several times. However, I am hoping that opening up these ten weeks this way will let students dabble a bit more, rather than feeling trapped into one prototype in which they have lost confidence.

I have kept a pared down version of the achievements system for the ten week design workshop. For a student to get an 'A', the minimum number of design artifacts would be two game concept documents and four generations of a prototype. Presenting one each week, this leaves four weeks for other legitimate activity—that is, to earn achievements. Most of these are repeated from last Fall's list, but there are two new ones. Networker is earned by going to a formal meeting of game designers, of which there are a few in Indianapolis, and Organizer is earned for organizing a playtesting session for the whole class. I am pleased with this last one, since setting up playtesting is an important part of the process, but there's no reason an interested student couldn't do it rather than me. This gives a student a chance to make a big impact in an authentic way. There is also an achievement that can be earned if students want to just keep on developing prototypes, since this is surely authentic class activity. I thought about adding an achievement for bringing snacks to a team meeting, but as much as I love snacks, this is probably too easy an out.

For several years, I have included an option in my game design class for students to read and present on A Theory of Fun for Game Design, the book that sparked my current scholarly endeavors. I have held off on requiring the book, however, for fear of trying to force students to follow "my path." Yet, every year, one or two students read the book, were amazed by it, and recommended that I make everyone read it. This year, I have decided to require it, and we will read and discuss it in the first five weeks. It was meaningful to me when I read it ten years ago and to my VBC students back in 2012, and I hope that it will inspire this Fall's group as well. I also hope that it will help us develop some common vocabulary.

Once again, we will be working with my friends at The Children's Museum of Indianapolis. They will providing themes and feedback on the designs, and I know that this was an important aspect of the course for the students last year. Last year's Fall game design course was intentionally scattershot: students explored a wide range of themes and ideas. This year, we are going to be a bit more focused. Also, I am planning on designing a game in the Fall as well, to use as a running case study. I look forward to the creative challenge as an opportunity to model some of my favorite practices, as well as to show students that nobody gets it right the first time.

Here are some links for the interested:

• Fall 2014 Course Description
• Fall 2013 Course Description
• Last year's post on game design course revisions

A year of achievement-oriented assessment in CS222

I conducted a formal research study as part of my experiments with achievement-oriented assessment in CS222. Although the semester is over, I have not yet opened my consent forms to determine whose data I can use in the formal study. First, I wanted to take some time to write out my informal thoughts about the past academic year.

For context, here are a few links:

• Revising Courses, Part III: Advanced Programming
• Tweaking Achievements and Reflections
• Fall and Spring course descriptions

My intention was to use the same format in the Fall and Spring, to get more consistent data for the study, but there was one piece that I felt I had to change between semesters: the tight binding of achievements and reflections. In the Fall, students' weekly writing assignments had to tie directly to their achievements. Late in the Fall, I realized  that this was an unnecessary restriction on the topic of their writing, and that what I really wanted was for them to be reflecting on any course experience, whether it was the achievements or not. The Spring reflection essays were more varied and much less formulaic. In fact, some delved into personal stories that were quite moving and inspiring. Of course, I cannot say more about that publicly, except to point out that it was the kind of moment that makes me proud to be a positive influence in a person's life.

There were still some hiccups with the reflective writing. There were three parts to each essay in the Spring—one less than Fall—but still students submitted without hitting each part. I can probably mitigate some of this by spending more time with Blackboard, copying the requirements from the course description to each assignment rather than referencing the course description. The programmer in me wants to simply reference the authoritative document, but experience has shown that students fail to check the reference.

I am also concerned about students' balance of attention between course activities. The achievements are mostly programming-related and are designed to integrate with the final projects. The best students saw this and leveraged it, but it feels like the capacity to recognize this dropped exponentially with students' effort or attention. Reflections, similarly, were supposed to tie all these experiences together, and the best students got this; many other students wrote trite, predictable, platitudinous, or probably-false reflections just as a way to earn course credit. I believe that anyone can learn from reflective writing, and I suspect many of my students did, but I also suspect that many more did not learn significantly from this activity, since they treated it as a box to be checked. This makes me wonder if the whole system of reflections should, itself, be an achievement, rather than required. More on this in a moment.

The achievement system led to students engaging in a wider variety of learning activities than I had seen before. Furthermore, much of this was self-directed: I gave criteria whereby achievements were earned, and students were responsible for finding the resources to earn them. Of course, I myself was a resource, but one could almost draw a line between high and low grades as to who recognized this. This is a common problem, not related to achievements I think, where "strong" students will talk to professors, ask questions, seek guidance, and "weak" students suffer from culturally-imposed fears that prevent this (fear of showing weakness, fear of authority figures, etc.).

All of the achievements were posted to a shared folder on Google Docs where students could see all that each other have posted. All of my evaluations were posted as comments on these shared documents. It is not clear to me the extent to which students read each others' posts and learned from them, but I know that it did happen at least sometimes. I am uncertain whether students read each others' posts for content or for form, but it seems to be a bit of each: from what I gather, ambitious students would scan others' posts and see where I left comments, and then try to avoid these pitfalls. As above, however, we see the weaker students making the same mistakes that their peers made just a few lines above them.

Several achievements dealt with what I called "production code," meaning that it had to be the students' final project code or parts of Open Source software. This definition, which seemed clear to me, caused students trouble in both semesters. Regularly, students would submit achievements that used code that did not fit this definition, sometimes the same students week after week. The confusion between current project code and old project code seems like a lack of attention to detail; however, some students confused public open source code with snippets on StackOverflow or tutorial sites. I need to clean up this definition in the future, making it clear that the point is to look at code that is part of a system, not didactic or decontextualized.

This leads to another unexpected oddity: in the final round of reflections and in the final exam, students used terms like "production code" as if they were universally meaningful, as opposed to only being meaningful in our class. I did not make it clear that this was a definition for us, and so I worry that students may go into interviews or other discussions and mistakenly express this as vocabulary. This was much worse with the achievements, in fact, many of which I tried to name in clever ways. For example, one achievement deals with Java's toString, equals, and hashCode methods, and I called the achievement, "Master of the Big Three." I didn't know what else to call it when I was writing it up, and that seemed like fun, so I called it that. However, then I overheard students talking about "The Big Three" as if that meant something in an absolute sense. Similarly, I had several achievements dealing with design patterns, and since I see patterns as a way-of-programming, I called these "Pattern Disciple" achievements. Then I heard students talking about "pattern discipline"—they didn't even get the word right, but they were still using these corny terms as if they were accepted jargon. I am not sure how to fix this aside from coming up with much drier names for the achievements, but the punster in me will likely find this impossible.

To get an 'A' in the class, students needed to earn "meta-achievements," which were coherent sets of related achievements. For example, the "Engineer" achievement required using UML and design patterns. (In retrospect, it may have made sense to call these "quests" instead, to better show the metaphor of an 'A' being at the end of the journey.) The members of one of the Spring teams did earn this achievement, although because of how I articulated the achievements, they had to incorporate the Decorator pattern where it really had no place. When I saw this, I was disheartened: what's the point of a pattern if it doesn't actually solve a problem in the domain, and would the students recognize the difference? When I expressed this doubt to the team, one of them actually defended my policies, saying that he was happy to have an excuse to study the pattern and use it, regardless of whether it "fit" the project. At least this guy saw the balance of pragmatic and academic here, and the result was positive for us both; however, I think I can circumvent this in the future by articulating quests in a way that ensures the activity is still motivated by the project context. Most students have difficulty finding the kind of intrinsic motivation that this one had!

Right at the end of the semester, I had an enlightening conversation with a student regarding the entire metaphor of achievements. I asked why he had not been more active in pursuing achievements when they were clearly tied to the course grade. His answer may have been an excuse, but it's interesting nonetheless. As a gamer, he conceptualized "achievement" as they are often treated in game design: as ancillary rewards, secondary to the primary gameplay. The discourse around the semester focused on the nine-week final project, and I can certainly see how students would see that as "the point of the course." From a game design perspective, these weren't really "achievements," then, as used in contemporary game design: they were essential, as missions or quests, waypoints or progress markers. Another student put it another way, saying that they weren't achievements at all, they were just a set of optional assignments. This is valuable feedback, and I see two ways out of this situation: re-name the achievements to reflect their essentially-mandatory nature; or actually make them optional, perhaps as gatekeepers to tiers of grades ('B' and 'A', for instance).

Achievement submissions were organized by topic, so all the "Master of the Big Three" achievements were on one page, for example. This was convenient for students to see what their peers had been working on, and what kind of feedback I had given to them. The resulting course document structure was like a wiki, then, with content organized by topic. I wonder how different the learner's experience would be if we took a portfolio approach, organizing the content by contributor. This, I think, would make it easier for individuals to see and think about what they did all semester. On the other hand, it would work against the goal of helping students see themselves as part of a community, and I suspect it would be less likely that students would look at each others' work, since one would not know whether a particular student earned a particular achievement or not. Also, it might lead to many mid-level students honing their sites on the portfolios of a small number of strong students; this would be problematic since, from my reading of the literature, students learn better from people at similar skill levels. This leads me to think that the wiki-structure approach is superior to the portfolio-structure approach, and now that it's on my blog, maybe I will remember this point as I consider what ideas to incorporate into my summer course revisions.

When I first imagined the achievement system, I envisioned the contribution of high-quality content on which I could provide meaningful, thoughtful responses. In fact, in my original design, I would not evaluate the achievement submissions at all, since each one had crisply-defined requirements, but I added a professor-review before the start of the Fall semester. In practice, there were many submissions that were not up to snuff. On one hand, this is predictable: if students will often submit assignment submissions without actually fulfilling their requirements, why wouldn't they do the same with achievement submissions? On the other hand, why would they do this? The achievements were not graded on some archaic percentage scale. There were only three grades: incorrect, acceptable, and correct-with-distinction (coded as 1, 2, and 3). The best I can figure is that students uploaded incorrect submissions because schooling culture encourages "submit and hope for points" as the dominant idiom, rather than "consider whether or not this is correct." (This is just one of many soapbox issues for me, where I point my finger squarely at... well, pretty much all the rest of formal education, which is difficulty because I have pretty small fingers. I could go on about how the dominant schooling culture discourages reflection, accountability, creativity, and deep learning while encouraging risk-averseness, but this parenthetical phrase is long enough.) I wonder, perhaps, if the numeric coding itself brought to mind students' conventional experience with points. Indeed, one student told me that he really wanted to get those '3's, even though, from a grading perspective, they were equal to '2': the only difference was one of pride. If I keep this aspect of achievements, I will try turning these into symbols, perhaps thumbs-down, thumbs-up, and ... I don't know, maybe Chuck Norris.

As a preliminary conclusion, I call the achievement-based grading system a success for this course. Students were able to focus on their nine-week projects and engage in meaningful content-oriented and reflective activity, with freedom to choose activities that aligned with their interests. At a minimum, I want to redesign some achievements to align more clearly and obviously with the projects, but I will hold off on major redesigns until I take some time to read my study data.

Tweaking achievements and reflections

Before the start of the Fall 2013 semester, I wrote about the changes I was incorporating into CS222, the Advanced Programming class. With that semester behind us, I can say that it went well, although not without the need for tweaks. For the Spring, I am keeping the fundamental structure. Here is a summary and justification for the tweaks.

I am scaling down the number of achievements required for each grade level. The original plan in the Fall was two achievements per week. However, students were not as adept as I had hoped at aligning achievements and projects. The intention was that students would earn achievements by doing their projects, but the majority of the class tried to do achievements separately from the projects. In the Fall, we ended up making a change midstream, reducing the number required during the nine-week project to only one per week. For the Spring, I'm pulling back a little bit more, expecting two achievements per week during the first three weeks, then only one per week during the two-week "warm-up" project and the nine-week major project.

Another change addresses the intended achievement-project synergy: restricting the code that can be used to complete achievements. When working toward achievements, Fall's students were permitted to use any code to which they had a license. In practice, I had students submitting trivial modifications to terribly-conceived CS1 and CS2 projects all the way to the end of the semester. In Spring, students can use any such code during the first third of the semester, but once they start their nine-week projects, they must use either that code or public open source code.

I have also changed the how and when reflections are required. In Fall's section, students wrote one reflection per achievement. I am sure that the best students—those on the road to being reflective practitioners—found this fruitful, albeit time-consuming. The students with worse writing skills spent inordinate time on these reflections, time that should have been spent understanding the material instead. Their action was justified, perhaps, by the fact that reflections were a significant part of the course grade. Also, I allowed resubmission of the reflections, but I think my attempt to encourage mastery learning instead fostered the"submit and hope" approach that I find so endemic and disdainful. In the Spring, the students will instead be asked to write one reflection per week, and this reflection may focus on any course-related experience: achievements, in-class discussions, project team discoveries, and so on.

The structure of the reflections will change as well. In the Fall, students were asked to do three things in their reflections: characterize an essential question, describe implications to practice, and consider alternative perspectives. Many students struggled with these all semester. Although I believe that tackling these problems is good for students, there is a limited about of time students can devote to this class. Hence, I have reduced and rearticulated the requirements: in the Spring, students will be asked to address one of the essential questions ("characterize" caused undue problems) and describe implications to practice.

Despite the addition of achievements and structured essays, perhaps the most uncertain of my Fall experiments was in not grading the final projects. As mentioned above, the course design was supposed to foster students' intrinsic motivation to work on the projects, which would provide context for the achievements. My ambitious designs failed to account for the pragmatic, goal-directed nature of the undergraduate! Students realized that the projects were not graded, and many students found this unmotivating: they wanted their work to be recognized directly in their grades. I will be returning to my conventional approach, grading both the projects and their ancillary artifacts such as presentations and a final report.

A final significant change that I expect to bring vast improvements is getting off of Blackboard. We used Blackboard's wiki in the Fall as a place for students to post and share their achievement artifacts. The wiki is ghastly, and because it was so difficult to give targeted feedback, we ended up with a very poor signal to noise ratio. Often, a student would post something that was wrong, and I would give formative evaluation to that student, encouraging revision; the student would not revise, and within a few days, I would receive several similar submissions from students who assumed the original was correct. What a mess. In the Spring, we're going to use Google Drive instead. I am uneasy about requiring students to register for a Google Account in order to participate, but most have them anyway, and the technology stack is so vastly superior that I'm going to try it. I will set up a shared folder that will contain one document per achievement, and students will post their artifacts within those documents. My feedback can take the form of comments, which are visible to all readers. I will have a bit of manual work to translate these data into Blackboard's gradebook, but that's worth it to help close the gap between my feedback and peer learning.

I finished up the course description this morning and linked it to my Web page. The achievements themselves are basically the same as the ones I designed last Summer, with a few notable additions that reward technology (integration with relational databases and structured data parsers) as well as community-building (interviewing alumni and senior capstone teams). Each achievement has also been clarified as to what exactly the corresponding artifact is—some had been vague in the previous iteration.

The semester starts in just a few days, and I'm looking forward to meeting the new crop of CS222 students. I am not looking forward to teaching at 8AM and Monday's high of -6°F, but I am trying not to complain.

But seriously, 8AM and -6°F high temperature.

Two-Week Project Showcase

The focus of my sophomore-level Advanced Programming course (CS222) is a large project at the end of the semester. In the past, this has been a six-week project, delivered in two three-week iterations. It is a project of the students' own invention: they have to pitch a project to satisfy a set of constraints I provide. This semester, I decided to expand the project to nine weeks with three iterations. This goes hand-in-glove with the revised, achievement-oriented assessment model that I am using this semester.

Most students have never worked in a programming team prior to CS222, much less defined their own projects. To warm them up, I give a two-week programming project before starting the big project. The two-week project is done in pairs, and I provide the requirements analysis as user stories. This provides a model for them to follow when they pitch their own projects using user stories.

This semester, I gave a new two-week project that was inspired by my 2013 Global Game Jam project. The students were challenged to write an application that takes a word from the user and then hits Wikipedia's API to determine the last modified date of the corresponding page. The curriculum provides no prior exposure to Web programming or networking, and so I provided a very short sample program that demonstrated how to open a URL in Java and read the stream. This project touches on some very important ideas, including Web services and structured data.

In the past, I have evaluated the two-week project in a rather conventional way: I provide a grading rubric at the start of the project, then I check out students' projects from the department's Mercurial server and give each pair a grade. I wanted to do it differently this semester, in part because of the achievement-oriented assessment. The two-week project provides a vehicle for students to earn achievements and write reflections: I'm not evaluating the project itself but rather how students use it to explore course concepts as articulated through the achievements and essential questions.

I decided to devote a class day at the end of the two-week project to a showcase. Each pair had to have a machine running a demo alongside a summary poster. We rearranged the classroom, moving all the tables to the perimeter and clustering the extras in the center. In order to foster peer interaction, I printed some sheets whereby students could vote on which team had the best UI, the best poster design, the cleanest code, and the most robust input handling.

The students enjoyed this format. There was an energy in the room as the students explored each others' solutions, talking about implementation strategies and UI decisions. A few students had trouble getting their projects working at all, and I heard one student say how disappointed he was, because it left his team unable to fully participate in the activity. This represents positive peer pressure and project orientation, which can be starkly contrasted against instructor pressure and grading-orientation.

I had recommended two strategies in class: using Joda Time to handle time formatting and using Java's DOM parser to deal with Wikipedia's output. I was surprised to see that almost every team used Joda Time (and used it to earn the Third Party Librarian achievement) but only one team used DOM. Every other team read the output stream as a single string and then searched it using Java's String class. This provided an excellent opportunity to teach about input validation. My sample Wikipedia URL queried the "soup" page for its last modified time, and the result looks like this:

<?xml version="1.0"?>
<api>
  <query-continue>
    <revisions rvcontinue="574699285" />
  </query-continue>
  <warnings>
    <revisions xml:space="preserve">Action 'rollback' is not allowed for the current user</revisions>
  </warnings>
  <query>
    <normalized>
      <n from="soup" to="Soup" />
    </normalized>
    <pages>
      <page pageid="19651298" ns="0" title="Soup">
    <revisions>
      <rev timestamp="2013-09-27T05:20:10Z" />
    </revisions>
      </page>
    </pages>
  </query>
</api>

Keep in mind that this is coming in as one continuous stream without linebreaks. Aside from the one group that did an appropriate and simple DOM lookup, students used String#indexOf(String) to search for "timestamp=", and then did manual string parsing using that reference point. This approach works for most cases, but it opens the application up to an attack that I'll explain in the next paragraph, giving the reflective reader a moment to consider it.

If you ask the application for the last modified info of the Wikipedia page "timestamp=", you get a phenomenon similar to an SQL injection attack: the indexOf operation picks up an unintended location, and the manual string manipulations fail. I had seen this when meeting with a pair the previous week who were working on their Advice Seeker achievement. They had thought their solution to be rock solid, and they were appropriately excited when I showed them how to crash it. They became my covert hitmen during the showcase, crashing solution after solution by finding holes in string parsing logic. So, while few students took the opportunity to learn XML parsing in the two-week project, maybe they learned something even better: the embarrassment of doing it the lazy way and seeing your system go down in public!

When I explained to the students what were doing at the showcase, I had expected teams to show up at their stations when I came by. However, it seems they were so excited to see each others' work that they didn't think about this. Since not every team had a person at their station when I came around, I was not able to give my expert evaluation to each group, nor was I able to model for the students how to give critical feedback. On the other hand, the students got a lot of peer feedback and I was able to meld into the group, becoming just one of many people interested in seeing demonstrations and code. I am not yet sure if I would do this part differently next time or not.

One aspect that is still unclear to me is the extent to which students were working for intrinsic motivation versus extrinsic reward. I was approached after class by a student from one of the teams whose solution was not working. During the hour, they had talked to other students and realized what they did wrong, which is an activity I certainly want to foster. The student asked, clearly in a state of agitation, if his group could fix their application even though it was due to be completed the previous night. I confirmed that this would be fine, and the student went away expressing joy and thanksgiving. I suspect his perspective was that he had just been given an opportunity to save his grade. What I really gave him was an opportunity to make his project work and feel good about getting it done, even if a bit late. I don't think he realized that there's no entry in the course grading policy for the two-week project, that the whole thing was just a fun context for us all to play and learn together. I hope that when he figures this out, he sees this as a reflective learning opportunity and not simply smoke and mirrors.

In conclusion, I am very happy with the showcase format. It was definitely worth using a class meeting for this event. I think this two-week project was particularly well-suited to the showcase format since it's fairly small, permits multiple solutions, helps students build better understandings of the modern computing ecosystem, and can have interesting failure states. Perhaps next time around I need to add an achievement related to XML parsing, since this seemed to promote students' use of Joda Time quite well.

(I have some nice pictures of the showcase that I took while standing on top of the teaching station, but I feel like I can't post them here without my students' permission. Sorry.)

Steering a student toward better reflections

We're in the third week of the semester already, and I have been happy with the changes I made over the Summer. I get the sense that my students like the achievement-oriented grading so far, or at least, they have not staged a rebellion. Many of the students found it difficult to understand the relationship between achievements and reflections. A few students didn't understand this two-step process at all—that you do an achievement first, then write a reflection. However, most of the problems came from difficulty understanding my reflection requirements, as stated in the course description:

A good reflection should address the following:

• Describe how your artifact and experience characterize one or more of the essential questions of the course.
• Describe the implications of this characterization on your own personal practice or, when relevant, your team's collective practice.
• Identify and address potential criticisms of this characterization.

When I talk about these requirements informally, I describe them as: characterize an EQ; implications to practice; and critique your characterization. I did not give the students an example to start the semester since past experience shows that, given an example, all solutions will look like that example. However, looking back on it, I think a simple example would have helped them to see that the reflection is not just the achievement restated, but a shift in focus and discourse.

To this end, I share the following story about a student who came to office hours for help with his achievements. I have the student's permission to share this story, though we'll keep him nameless. He was frustrated that he had not received any points for his reflections, and looking over his submissions, he had simply submitted his achievement artifacts again in the reflections slot on Blackboard. We started by looking at the first achievement he had earned—Studious—which requires reading Bill Rapaport's "How to Study" Guide and then creating a study plan.

First, I pointed out that the reflection has to happen after earning the achievement, and that the first step was to describe how earning the achievement helped him to characterize one of our essential questions of the course. I asked the student to talk through his experience of earning the achievement. He told me that he had been assigned to read Rapaport's guide last year: his instructor had worked with me before and enjoyed Rapaport's guide so much, that he assigned it in a prerequisite class to my class! The student took an informal posture and tone, and told me that since he read it before, he didn't expect to get anything new out of it, but since it was required, he re-read it anyway. As he thought, he didn't learn anything new. At this point, he corrected his posture and his voice turned "cold academic" as he started in with, "Well, according to the author, time management is an important skill for students, and being a student is like being a professional, and so he would say that we should be practicing time management too..."

I cut him off and pointed out how he was telling me how Rapaport might characterize an essential question, not how he would. I encouraged him to take a big step back and think—that is, reflect—on what he had extemporaneously told me about doing the reading even though he had done it a few months before, how he entered it with some negative expectations, and how these expectations were met. So, I asked, how does that experience help you characterize an essential question?

He paused and thought, looking over the list of essential questions. "Well," he tentatively offered, "I guess it means that a professional will sometimes try things he thinks will help, but they won't."

"Yes," I encouraged him. "So, what does that mean for you? That is, what are the implications to practice."

He quickly returned, "If I'm trying something and it doesn't work, I shouldn't get frustrated."

Aha! I congratulated him on the clear articulation of a discovered truth, grounded in his experience with the achievement. We reviewed what we had so far, and I pointed out that he could express it in two to four concise sentences; these reflections were not to be judged on length but on clarity of insight. We moved to the third step, which I have seen to be the most difficult for the students. How might an intelligent layperson critique your characterization? In this case, how might someone argue against this characterization—that sometimes professionals do what they think is best, and it doesn't help—and how would you address such criticism?

This was harder for him to address than the last point. He suggested that, since he's not a professional, he could be wrong. I pointed out that this was not a very interesting observation, and he could use that as a criticism any time he was addressing the essential question of professionalism. He seemed to have trouble pulling something together, so I improvised. One might say that the professional does learn something from this experience: he learns from the failure and tries something new. And if that's the best criticism we have—that what looks like failure is in fact progress—then we are happy to accept it and move on.

The student thought about that, then he asked, "But couldn't we use that as a criticism every time, too?" I laughed and told him that I would probably notice if he did, but he should feel free to use it for the first one anyway.

This was a positive meeting all around. It was good for me to sit one-on-one with a student who simply didn't understand my instructions: talking with a student about his struggles with metacognition grants more perspective than typing responses mechanically into Blackboard. He clearly benefited in the short term because he built a mental framework for approaching reflections, and I hope that the benefits extend beyond the course: the whole point of these reflective activities is to help students become better learners.

Revising Courses, Part III: Advanced Programming

Inserting reflective essays into my game programming course was straightforward, and the augmented achievement-based grading system of my game design course was an incremental improvement. Now, I will share the story of the biggest renovation in my Summer of Course Revision: complete renovation of my Advanced Programming course.

CS222: Advanced Programming has been around for about four years now, and I have taught it more often than any other. Several posts on my blog reflect on this course, including this one from the first semester's experience and this one reflecting on this past academic year. For the past several offerings, I have used the same fundamental course structure: studio-based learning, using daily or weekly writing assignments for the first several weeks, gradually turning toward a focus on a pair-programmed two-week project and then a small-team six-week project. I have been basically happy with the structure, but reflecting on my experiences, I identified the following pain points:

• The course was sequenced along a particular path of learning that did not resonate with all the students.
• There was not enough mastery learning: students would often do poorly on an assignment and clearly not revisit it, since weeks later, they still didn't understand the concept. This caused especial pain when these assignments covered technology that was core to my pedagogy such as distributed version control.
• It was easy to slip into a mode where I would talk for the whole class meeting. The studio orientation—through which students show artifacts that represent their learning, and these artifacts are subject to formal and informal peer and expert critique—was not guaranteed in my course structure: it relied on daily ingenuity rather than codified form. (The time and space constraints imposed by the university negatively affect studio orientation as well, but these are beyond my control.)
• It was not clear that all the team members were engaged in reflective practice when working on their projects. That is, I felt that there were people falling in the cracks, not learning adequately from the team projects, sometimes dues to lack of reflection and sometimes due to lack of practice.
• Students enter the course with high variance in knowledge, skill, experience, and motivation.

In addressing these pain points, I wanted to make sure I kept all the good parts of the course. Students work in teams on projects of their own design, using industrial-strength tools and techniques. Teams have to incorporate some kind of management techniques and give multiple milestone presentations, both of which remind students about the "soft skills" of Computer Science. There is a good overall flavor of agile development, pragmatism, object orientation with patterns, and the importance of reflective practice and lifetime learning.

If you've read my last two posts, you probably see where this is going! I decided to replace the daily and weekly assignments with a system of course achievements (a.k.a. badges), and students will write reflections that relate the achievements to essential questions of the course. The complete course description is available online, or you can choose to view just the achievements.

I have developed the following set of essential questions:

• What does it mean to be a "professional" in software development?
• How do you know when a feature is done?
• How do small teams of developers coordinate activity?
• How does a Computer Science professional use ubiquitous and chaotic information to be a lifetime learner?

As in my game design course revision, students' grades will be based on number of achievements earned and written reflections on those experiences. There are also four "meta-achievements" that can be earned by completing sets of regular achievements. These are leveled achievements and reflect four different potential paths of mastery: a Clean Coder has documented evidence of applying ideas from each of the first twelve chapters of the book; a White-Collar has demonstrated savvy at project management and presentation; an Engineer is moving toward an understanding of software architecture and patterns; and a User-Centered Designer has designed, evaluated, and iterated on user-facing systems. The introduction of these meta-achievements was partially inspired by an alumnus who, when I told him about this redesign, said that most courses were like a Call of Duty game, but that this was more like Deus Ex, and that it would benefit from having more quests.

I want to help students succeed in this kind of learning environment, and so the first month is designed to help them understand what it means for them to have more ownership over course activity. In the first week, I will provide a review of important CS1 and CS2 topics, focusing on the mechanics of object-oriented programming in Java. Knowing that student experience and comfort with this material is highly variable, I can use this week to focus on how to navigate the course structure. In fact, I will strongly recommend that their first achievement is Studious, which requires them to read William Rapaport's excellent How to Study guide and write a study plan for the semester.

We will still use Clean Code as a shared focus, but I have changed how I am scaffolding students' experience in integrating its expert tips. Previously, I had identified specific tips or chapters for students to read and then asked them to apply these tips to their projects—past or current. There was little evidence that this "stuck" with the students, as later coursework would violate the concepts supposedly learned. This was, in part, due to lack of mastery learning, where students would accept a low grade and move on without having learned the material. Furthermore, because we paced Clean Code readings and assignments through the semester, I would frequently encounter examples in class and in critiques that afforded application of a tip that the students had not yet studied. In the revised course, we will read the book relatively quickly, but then keep returning to it in informal expert critiques, adopting an iterative approach that is more apropos to how one learns and remembers these tips: a bit at a time, and a bit more next time.

In my game design course, students are required to present to the class to earn their achievements, but the enrollment in that course is half that of this one. I toyed with the idea of having my CS222 students post their work on the walls, a portion of the class each day, but I feared that this would have too many negative consequences. Instead, I am requiring students to post their artifacts to a wiki, and my intention is to review the wiki changes between class meetings to identify notable entries. This way, I can bring up the wiki on the projector and model an appropriate critical process. I can also insert mini-lectures as necessary to clarify misunderstandings. I am eager (or perhaps anxious) to see how the density of misunderstandings in this redesigned course, which encourages mastery learning through reflection, compares to that of my current status quo, which encourages correctness up front. We'll be using BlackBoard's wiki only because it's easy for all the students to find; it was frustrating to me during my experimentation that it has no wikitext editor. If the wiki turns out not to work for us, we can always move to an in-person poster-style presentation.

Since I will be doing more just-in-time teaching—reacting to students' insights and confusions—I have decided to expand the conventional six-week, two-milestone project to three milestones over about eight weeks. The students like this part of class the most, and I like the idea of adding another milestone, since this gives them another opportunity to learn from mistakes in tools, design, organizational structures, and presentation. For both this large project and the small project, I will provide common requirements that everyone must follow, such as using distributed version control. Previously, these things were simply worth some points on the project; however, if I want to focus on assessing their reflections while also requiring some shared technological experience, it's fairly easily done with a hard-and-fast requirement. With multiple milestones, there will be an opportunity to ensure each team is following the requirements. For example, if I see a team not using DVCS, then I can remind them.

I am eager to see how students take these changes. I expect to be able to blog about some of the day-to-day experiences, and I anticipate writing some academic papers on aggregate student performance in all these modified classes.

Here are the critical links again:

• CS222 Course Description
• CS222 Achievement List

Revising Courses, Part II: Game Design

Following up on my previous post about revising my game programming course, today's post is about a revision to my game design course. This course is an honors colloquium, a special topics course only open to honors students and with an enrollment cap of fifteen. Every honors student has to take six credit-hours worth of colloquia, and the topics depend on who is available to teach them in any given semester.

This colloquium in part of a two-semester immersive learning project undertaken in collaboration with the Indianapolis Children's Museum, and it is being funded by internal funds through the Provost's office. Teaching in the Honors College is not a normal part of my load, and a major portion of the grant is "assigned time," allowing me to teach this colloquium. I mention this because the behind-the-scenes machinations are likely opaque to those outside academia: if I didn't have the grant, I would be teaching a Computer Science course instead of the colloquium. I am grateful to the Provost and his committee for approving my proposal, which allows me to teach this course that aligns so well with my research interests.

I taught a similar colloquium last Fall, and it had the same fundamental objectives: engage students in the academic study of games and learning, and, in collaboration with a community partner, have them produce prototypes of original games.There were two significant differences: I was team-teaching with my colleague Ronald Morris, and the community partner was the Indiana State Museum. It was my first attempt at achievement-based (i.e. badge-based) grading, and I wrote a lengthy reflection about the experience. For the redesign, I decided to focus on a few specific pain points from last time:

1. The students put off achievements until the end of the semester.
2. Some students made zero or nominal changes between game design iterations.
3. Not all the students were engaged in reflective practice: they were not adequately learning from the process.
4. In-class prototype evaluation time was rarely meaningfully used due to the points already mentioned.

I have met a few new colleagues through the conference circuit in the last year, and I am grateful for their willingness to share tips and tricks. In particular, the following changes reflect some specific ideas I have picked up from Scott Nicholson at Syracuse University and Lucas Blair at Little Bird Games.

Perhaps the most important revision to the course is the introduction of reflective essays. As in my game programming course, I was inspired by the participatory assessment model to grade reflections rather than artifacts. The students will present their weekly artifacts to the class, where artifacts might include summaries of essays and articles, posters, one-page designs, or prototypes. These artifacts will be subject to peer and expert formative evaluation, following the studio "crit" model. However, it will be students' reflections on these artifacts that are actually graded. As in my game programming course, I have decided to frame these reflections around essential questions and grade them based on (a) how they characterize an essential question, (b) the implications to practice, and (c) potential criticisms of the characterization. The research I have read predicts that this combination of achievement criteria and reflections should encourage students to produce high-quality artifacts without sacrificing intrinsic motivation.

Last time, the students had to choose a topic and iterate on it fairly early. The students with the best designs at the end of the semester were, for the most part, those who had to throw away major elements of their design or change themes entirely. This leads toward the desire to do more rapid iteration on ideas, not just prototypes; however, I still want each student to create a significant prototype by the end of the semester. To address this, I have divided the semester into three parts. In the first part, we will survey major themes of the course, such as games, fun, learning, museums, and children. The second part of the semester will be rapid creation of design sketches based on specific themes at the Children's Museum, about one sketch each week. The students will then choose one of these to prototype for their end-of-semester deliverable. I hope that this approach improves all the students' prototypes: even though they have less time to work on their prototypes, that time will be more focused and based on having had more reflective practice earlier.

Going along with the three-part division of the semester, I have organized the achievements into groups, some of which are tied to one of these parts. There is also an "unrestricted" category that can be earned at any time. I have introduced a throttle of one achievement submission per week plus one revision per two weeks. The students' grade is tied to the number of achievements they earn, and so this should help the students pace themselves while keeping me from having to evaluate an inordinate number of submissions at the end of the semester.

Following the most popular design principle for assessing learning with digital badges, I have introduced a "leveled" system of achievements. Certain achievements have gold stars attached to them, designating them as requiring special effort. These stars are tied in with number of achievements and reflection points in order to determine a student's grade. Note that two out of the three of these are directly in the student's control, and the one that isn't—reflection points—permits revision. Hence, students can essentially pick their grade based on their level of legitimate participation in the class. 

The full list of achievements is available online, and you can view it on its own page or embedded into the course description. Each badge is defined by a name, a blurb, criteria, and an image. The blurb and image are new this year, and I think they represent a major improvement. I used OpenBadges.me to design the badge images, making significant use of icons from The Noun Project. In case you're curious or want to sketch up your own, the border is the Ball State red taken from our logo (#ed1a4d) and the starred achievements use light yellow background (#ffff66).

Note that the core class activities in the second and third parts of the class are associated with achievements that lead up to stars. For example, a student who shows design revisions every week for the five weeks of prototyping will earn two gold stars, which are half of those required to earn an A. Indeed, I intend for the standard path through the course to consist of some student-directed inquiry in the first part, then two stars from one-page designs, then two stars from prototyping—and I will make this clear to the students at our first meeting! However, the system gives the students agency to choose a different path: if someone wants to focus on games criticism, or reading and reporting on game design texts, these are still rewarded and earn course credit.

One other change this year is based directly on student feedback from Fall. Last time, I had an achievement that was earned by playing several games that exhibited specific mechanics that I had identified. My intention was that I could guide students to experience genres, mechanics, and themes that I found interesting, but it ended up making the achievement hard to earn and delayed rewards for legitimate course activity. Note that there was no formal "leveled" achievements as I have this year with starred achievements, so this one achievement took much more time for the same reward as any others. This year, I have given the students much more freedom to choose a the games they will study and critique. They can choose analog, digital, or hybrid games, including sports and gambling games. I still provide some scaffolding through the games I chose to put on course reserves, but students who want to go in a different direction are free to do so.

The one weak spot in the course design, as of this writing, is the identification of essential questions. I have come up with two so far:

• What is the relationship between games, fun, and learning?
• How do you design an educational game for children?

I thought about introducing a third about design generally, such as "What is design?", but it seems that this is embedded into the second question. I worry that adding such an EQ would diminish the impact of the design-related one I already have. Finally, I will point out that the first essential question has explicitly guided my work over the last several years, and it was the explicit topic of my seminar at the Virginia Ball Center for Creative Inquiry: it is such a big question that others pale in comparison.

After having spent about three weeks this Summer revising my Fall courses, I find myself looking forward to the start the semester. It is good to have the time to rest, reflect, and revise. Of course, all this work has been without compensation, but, as the Spirit of Christmas once said, the rewards of virtue are infinitely more attractive.