Notes from "Grading for Growth"

I read portions of Grading for Growth as part of my preparations for the Spring's preproduction class. This book makes a case for "alternative grading," establishing four pillars for their efforts. These are:

1. Clearly defined standards
2. Helpful feedback
3. Marks indicate progress
4. Reattempts without penalty

I've been reading Talbert's blog for some time, and it's that last one that gives me some difficulty. I was hoping that reading the book would help me understand some practical matters such as grading management and dealing with skills that build upon each other. However, I found myself taking more notes about CS222 Advanced Programming and CS315 Game Programming than about CS390 Game Studio Preproduction.

I have read Nilson's Specification Grading and many articles on alternative grading, so I skipped through some of the content and case studies. The first case study is the one that was most relevant to me: a case of a calculus class in which the professor used standards-based grading (SBG). This was contributed by Joshua Bowman at Pepperdine University. This case study is published at the Grading for Growth blog.

One of the tools that I had not fully considered before is the gateway exam. Bowman gives a ten-question exam early in the semester. Students must pass at least nine questions to pass the exam. Students get five chances to retake the exam, and a passing grade is required for a B- or better grade. This is potentially useful to deal with some of the particular problems I have faced in CS222, where students come in with high variation in understanding of programming fundamentals while also suffering from second-order ignorance. A formalized assessment could very well help with this. 

Another useful idea from the reading is the distinction between revision and new attempt. In my own teaching, I have allowed revisions, but I frequently in CS222 find myself suggesting that students begin assignments anew with new code or contexts. This was never a clear requirement but rather a strong suggestion. Separating these two ideas could increase clarity about the significance of error or misunderstanding. In particular, this could help with a particular error mode that I have seen in CS222: a student submits a source code evaluation, I critique the evaluation, and the student resubmits an evaluation that restates what I just pointed out in the critique. This masks the distinction between a student who has learned the material and one who can effectively parrot my commentary. The problem could be avoided if I required new attempts in cases where I am using my feedback to direct the student's attention to what they have missed rather than to point out small oversights.

Regular readers may recall that I experimented with specifications-based grading in my section of CS222 in Fall 2023. I only laid out cases for A, B, C, D, and F grades, similarly to how I have implemented specs grading in CS315. The reading suggested that +/- grades can also be laid out in a specification, using them for "in between" cases.

I regularly air my frustrations with the incoherent concept of "mid-semester grades," but a piece of advice from the book struck me as useful. There was a recommendation to only give A, C, or F grades at midsemester. This is probably the right level of granularity for the task. The alternative, which I also recently came across in a blog somewhere, was to have students write their own mid-semester evaluations as a reflective exercise.

Bowman and others separate their standards into core and auxiliary. This could be useful in both CS222 and CS315, where I tend to weave together content that students are required to know from the syllabus with those that I think are useful from my experience.

The authors directly address the problem that reassessments have to be meaningful. Unlimited resubmissions will inevitably lead to students' throwing mediocre attempts at the problem in hopes that it goes away. The authors suggest two techniques for ensuring assessments are meaningful. The first is to gate the possibility for reassessment behind meaningful practice, which probably works better in courses with more objective content such as mathematics courses. The other is to require a reflective cover sheet. I have required students to give memos explaining the resubmission, but I've never given them a format for what this entails. This has led to my accepting many "memos" that show little evidence of understanding, usually when my patience is exhausted. Formalizing the memo process would benefit everyone involved.

Those are all helpful ideas for this summer, when I will likely take elements of CS222 and CS315 back to the drawing board, but what about the resubmission rate issue that I was actually looking for? Well, I found quite a surprise. The authors suggest exactly what I have been doing for years: using a token-based system or throttling resubmissions. The real puzzle here then is what exactly they mean by "reattempts without penalty," since it's not what those words actually mean together. Only being able to reattempt a subset of substandard assignments is a penalty, since from a pure learning point of view, there's no essential reason to prevent it. That is, the penalty is coming from the practical matter that teachers cannot afford to teach every student as if they are their only responsibility. This finding was anticlimactic, but part of me expected that it would be what I had found. There's no silver bullet, and if I haven't seen nor invented something better in 20+ years of alternative grading experience, then it does not exist.

(It's funny to actually type out "20+ years of alternative grading experience," but it's true. It's also one of those things that's making me feel old lately.)

On the ethics of obscurity

Years ago, I experimented with what is now called "specifications grading" in CS222. I set up a table that explained to a student how their performance in each category would affect their final grade. These are not weighted averages of columns but declarations of minima. For example, to get an A may require earning at least a B in all assignments, an A on the final project, and a passing grade on the exam. This gave a clarity to the students that was lacking when using more traditional weighted averages. While publishing weighted average formulae for students technically makes it possible for them to compute their grade for themselves, in practice, I have rarely or never found a student willing to do that level of work. Hence, weighted averages, even public ones, leave grades obscure to the students, whereas specification tables make grades obvious.

What my experiment found was specifications grading made students work less than weighted averages. The simple reason for this is that if a student sees that their work in one category has capped their final grade, they have no material nor extrinsic (that is, grade-related) reason to work in other columns. Using the example above, if a student earns a C on an assignment and can no longer earn an A in the class, they see that they may as well just get a B in the final project, too, since an A would not affect their final grade.

This semester in CS222, I decided to try specifications-based final grades again. It probably does not surprise you, dear reader, that I got the same result: students lost motivation to do their best in the final project because their poor performance on another part of the class. It's worse than that, though: the final project is completed by teams, and some team members were striving for and could still earn top marks while other team members had this door closed to them. That's a bad situation, and I am grateful for the students who candidly shared the frustration this caused them.

The fact is that students can and do get themselves into this situation with weighted averages as well. A student's performance in individual assignments may have doomed them to a low grade in the class despite their performance on the final project, for example. However, as I already pointed out, this is obscured to them because of their unwillingness to do the math. What this means—and I have seen it countless times—is that students will continue to work on what they have in front of them in futile hope that it will earn them a better grade in the course.

And that's a good thing.

The student's ends may be unattainable, but the means will still produce learning. That is, the student will be engaged in the actual important part of the class. 

Good teaching is about encouraging students to learn. That is why one might have readings, assignments, projects, quizzes, and community partners: these things help engage students in processes that result in learning. It is a poor teacher whose goal is to help students get grades rather than to help them learn. Indeed, every teacher who has endeavored to understand the science of learning at all knows that extrinsic rewards destroy intrinsic motivation. 

What are the ethical considerations of choosing between a clear grading policy that yields less student learning and an obscure one that yields more? It seems to me that if learning is the goal, then there is no choice here at all. How far can one take this—how much of grading can we remove without damaging the necessary feedback loops? This is the essential question pursued by the ungrading movement, which I need to explore more thoroughly. 

I also wonder, why exactly haven't we professors banded together and refused to participate in grading systems that destroy intrinsic motivation? 

MDA and the Game of School

We had an excellent discussion in my game design class yesterday about emotions in games. I have a stable set of readings and assignments that I have used several years for this meeting, and it's become one that I really look forward to. One little mix-up between my notes and the course plan, though, was that I thought I required them to prepare posters to present their work, but in fact, I had only asked them to post on a discussion board. We ended up having one volunteer share his work, and that was enough for us to have a great discussion. 

Because they didn't have posters on the wall, I didn't know who had completed the assignment and who had not. I was surprised, then, when I student came up to me after class to share with me what they did instead of the assignment. (This was a public hallway discussion, so I don't mind sharing it in general here, though I will dodge some particulars.) This student shared that they were feeling frustrated with school in general and so they decided to do an MDA analysis of "school." 

The student's conclusion from the analysis was that school is a game of managing diminishing resources. They saw that resources like sleep, socializing, and course grades are always running out, and you have to determine which one to prioritize. An example was that you might have a "B" in a course, but you might let that diminish in favor of putting effort into a course where there is greater perceived need. In analyzing the dynamics, the student observed that the extant systems induce an expectation of particular grades or achievement, which in turn induce a feeling of disappointment when these goals are not met.

We had just briefly discussed in class how theorists like Johan Huizinga and Roger Caillois attempted to use play as a framework for describing anthropology or sociology, and I agreed with her assessment: that school seems to have all the properties of being a game. The student seemed a bit pessimistic about it, although by their own admission, they were operating on no sleep. I congratulated her on the appropriate use of the MDA framework to analyze her situation, but I also encouraged her to take a step back and ask why the system is the way that it is. I shared my favorite reference here: John Taylor Gatto's famous essay, "Against School." In it, Gatto describes the six goals of compulsory education as presented in Alexander Inglis's 1918 book, Principles of Secondary Education. I expressed to the student something I've said often before: if we understand the goals for which school was designed, we can come to understand some of the deleterious secondary systems that have emerged. However, by the same token, we can also think like designers and consider how to either re-design core systems or hack existing ones. I am a radical, meaning literally that I want to return to the roots, but wholesale redesign tends to tilt too hard against the windmills.

It was a heavy hallway conversation, and I am honored to have been trusted with the story. At the same time, the student did this MDA analysis instead of doing the assigned work. The result was that, while the student arguably had an better sense of MDA and school, they were also not as well prepared to discuss the topic of the day.

Incidentally, I don't think the student's analysis is correct. While it might be true to model school as a game, I think that modeling it strictly as a game of diminishing resources is incorrect. While it's true that time is more measurable than understanding, I think in the common and good cases, the former trades for the latter. The resources that goes into "school" can produce good outcomes, although, by design, they also produce bad. For example, "school" necessitates my making a decision about how to grade a student's participation when they do something personally fulfilling instead of what I asked to be done.

Notes on Spreadsheet Features to Support Effective Grading (Or, What Canvas Can't Handle)

Here are some quick notes about my grading requirements. Each item is evaluated using triage grading, which implies in part that it is graded out of a convenient number of points. All the items in a week contribute to a weekly score. For simplicity, let's say each item in the week is evenly weighted. Note that this does not imply that each is graded out of the same number of universal points. One item might be graded out of four, and another out of eight, but they contribute equally to the week's score. If someone gets 4/4 on the first assignment and 4/8 on the other, then they get 75% for the week. 

If that seems confusing to you, maybe you are a lead engineer at Canvas. It is clear from their interface and the tenor of their discussion boards that they cannot conceive of a model of a course where you don't have some predetermined number of points, and then everything you do doles out those points. I will quickly put it to you, dear reader, that this is a bad model. One of the main reasons is that you have to deal with too many points. Let's say you make your course worth 1500 points, maybe reasoning that there's 100 points available each week of a fifteen-week semester. If a week has two activities in it, maybe one is worth 25 and another is worth 75. Now, you have at least two problems. The first is that you have to determine the exact semantic value of 3/75 of one assignment and make it equivalent in weight to 1/25 of the other, if you're doing an honest job. The other is that you have to make it clear to students the difference between, say, 55 and 56 out of 75 points. By contrast, consider the elegance of triage grading. If I give an assignment that is worth 3 points, and someone does a middling job, they get 2 out of 3 points. Their classmates who also did middling jobs get 2 out of 3 points, but of course, each one gets specific feedback about what aspects were not up to snuff. No ambiguity, no arguing for points, no false equivalence between units. 

My point in this post, however, is not to convince you that you should adopt a coherent grading philosophy, but rather to note for myself how I was able to compute midsemester grades today. This is important given that Canvas—my university's required "learning management system" (ack)—is basically useless for anything but the naive, conventional, point-based course structure.

My game programming course was easy to process because the students turn in one thing each week, but in my game design and human-computer interaction courses, students are doing a lot of different things each week. I decided to use Canvas modules to present each week's required submissions. In Canvas' embedded gradebook, you can view assignments by module, but once you download the data for offline processing, that information is lost. It appears that the columns may be sorted in the order they were created, but the important thing to note is that the connection between column and week is lost.

The first step, then, was actually for me to rename all the assignments on Canvas. I prepended each with an annotation of its week as a two-digit code, so the first week's work was "01", the second was "02", and so on. Then, I downloaded the CSV and was able to import it into Libreoffice Calc. The next step was to wipe out all the Canvas cruft, which was maybe two dozen column headings that I don't care about.

After a lot of tinkering this morning, I determined that life would be easier if I added an extra row to represent the week number. This has a simple formula:

=NUMBERVALUE(MID(AssignmentTitle,1,2))

Let's call that value AssignmentWeek. It allows me to use SUMPRODUCT to do the heavy lifting. My first step was to make sure I could compute the right scores for the first week, so I ended up with a formula like this:

=SUMPRODUCT(NUMBERVALUE(MID(AssignmentTitles, 1, 3))=1, AssignmentScores/TotalPoints)

Once or twice a year I have to remember how arrays work in spreadsheets, so let me say a little more about that function above. SUMPRODUCT automatically works over arrays, unlike other fuctions that require either ticking a checkbox, using curly braces, or pressing Ctrl-Alt-Shift-Meta-Something while pressing Enter. The first argument to SUMPRODUCT is going to be zero or one depending on whether the numeric value of the first two characters is equal to 1. In my case, does the assignment start with the prefix "01" or not. This essentially filters out everything except those columns which are actually Week 1 scores. Then, with the magic of array processing, we can add together the result of dividing the points earned range by the total points range, and this evens out the scores regardless of out of how many points they were graded. 

Now, to generalize this, I set up a series of columns, one for each week, with the week number given at the top. Let's call this WeekNumber. The above formula then can be revised so that we can see the score for each week as follows:

=SUMPRODUCT(NUMBERVALUE(MID(AssignmentTitles,1,3))=NUMBERVALUE(WeekNumber), AssignmentScores/TotalPoints)

The result of that expression above needs to be divided by the number of work items given that week. This cost me a lot of time trying to sort out, because it doesn't work the same way as the WeekNumber reference above. Instead, the expression looks like this:

COUNTIF(AssignmentWeek,"="&WeekNumber)

This gives a final form like this, which is copied into each cell of a matrix that shows the week's score for each student:

=SUMPRODUCT(NUMBERVALUE(MID(AssignmentTitles,1,3))=NUMBERVALUE(WeekNumber), AssignmentScores/TotalPoints) / COUNTIF(AssignmentWeek,"="&WeekNumber)

Looking at each student's score for each week allows me to drop in a simple formula to drop the lowest and determine their midsemester grade.

It took a lot of time this morning to sort this out, so hopefully by recording my notes here I can save myself time later, and maybe even help folks who come across this via search results. If you happen to know an easier way to meet my requirements through a spreadsheet, let me know. It's clear to me that Canvas is not robust enough to handle something like this. That's a shame, since my students are so addicted to the illusion of data that comes from seeing "current grades" or "midsemester grades". I'd rather blow up the whole thing, but I'm not ready to move to Evergreen just yet.

Specifications grading and wiggle room

I have been using specifications grading in my game programming class. I just finished evaluating my class' third mini-project, and now I am encountering some interesting problems. I want to share them here, in part to be able to review my notes later, and in part to see if my readers have any thoughts or advice.

First, a very brief background. Specifications Grading is the name of a technique popularized in the eponymous book by Linda Nilson. I actually used this approach several years ago in CS222 without giving it a clever name, but I abandoned it when I had a student tell me that he gave up on the course material because he knew he could not get to the level he wanted. That's just one case, and clearly from my plans for Game Programming, I decided to give it another shot. You can find all the specifications for the four Game Programming Mini-Projects on the projects page of the course site.

In the first two rounds of mini-projects, I had some cases where I ran into subtle problems with the specifications. One specification requires that students follow our established naming conventions. The problem arises, what to do if a student misnames one asset? Is one violation enough to say the whole specification was failed? Clearly not, but how many then? I am thinking about recasting such elements into something like "No more than two violations of the style guide." The problem then is, of course, that I have to actually look for them and keep track, doing the kind of bookkeeping that good specifications should make unnecessary.

A similar problem came up in the project report requirements, where I require that students describe the third-party assets they used and the licenses under which they are using them. Turns out, my students either were unbelievably lazy about this or really didn't understand the requirement. After the first Mini-Project, I encouraged them to take this more seriously; after the second Mini-Project, I realized I needed to intervene. I gave a class presentation about IP and licensing, including specific examples of violations from student work. I thought they should have learned this before a junior-level elective in college, but maybe they didn't; however, even in the third Mini-Project, I had students doing this wrong. I made the project report a low-grade specification: a student needs to have a well-formatted project report in order to get a C, but "well-formatted" includes all this licensing info. According to the specification, students who do not track the licenses properly should get a D. Is that right? Maybe, maybe not. I am thinking of separating the specifications for the project report to make this more generous to the students who really haven't yet internalized concepts of intellectual property.

Another area where students are having trouble is in working with Perforce. I sympathize: it took me some time to make sense out of it, and I had the benefit of working with many kinds of version control systems. It doesn't help that Perforce's nomenclature of "depot" and "workspace" is idiosyncratic. Having a working version of the project in our Perforce depot is a D-level requirement: fail that, and you fail the project. However, many students demoed games in class that are clearly not what they submitted. I am torn on this one: it's a clear, explicit D-level criterion that "Project is correctly configured on the depot so that a new client provides a runnable game." Students are turning in project reports with that box checked, but I doubt they have actually verified that this is the case. Indeed, one student even submitted a project report with that box checked and emailed me to say that he had trouble with the depot before submission. Hence, while I am sympathetic to the frustrations of learning new version control systems, I have very little tolerance for conscientiousness and none for deception.

If you look over the specifications, you will see that are simply for levels D, C, B, and A. In the report, students are supposed to tell me what grade they earned. My intention behind this was twofold: first, it would make them doublecheck the specifications and reflect on what they have done, and second, it would make my grading easier. I am surprised by how many students, in their reports, will make a claim like "I earned a B+" or "I earned at least a C because I worked hard on this." Neither of these are in line with the specifications system at all. It's really not clear to me where they get these ideas, if they are reading into rules something that's not there or, possibly more likely, they are not reading the rules at all.

My friend and colleague David Largent has been using Nilson-style Specifications Grading for several semesters, and so I look forward to picking his brain about some of these issues. He deploys a system called "Oops Bits" wherein students can get another chance if they misunderstand or misrepresent a specification, but I don't know much more about the system than that. I am thinking of sending out an email to my class to give them some kind of timeboxed period in which they can deploy an "Oops Bit", e.g. if they realized that the game the demoed was not properly submitted to the depot. The obvious negative here is that then there's no lesson really learned: I have to grade their work again, when they already claimed in their reports to have met that criterion.

As an aside, I require that the project reports themselves be written in either HTML or Markdown. I am surprised how few students are fluent with one or the other of these. Like understanding intellectual property, it seems to have become a major unexpected learning objective of the class that students understand plain text formats. I provided the students with a Markdown report template for Mini-Project 1, and yet many of them manage to create non-standard or nonsensical reports. I wonder if I can easily modify the Javascript that creates the specifications articulation on the course Web site to automatically generate downloadable Markdown templates for each separate project, which would potentially reduce students' copy-paste hassle and, ideally, provide a more convenient way for students to fill in the blanks and meet the report specs correctly.

Rubric-based self- and peer-evaluation on team programming projects

I have written before about grading rubrics, and I am taking this opportunity to share some of my recent developments here. In my game programming class (CS315), I have eleven students working together on one project. We're using Scrum, and it's still going well. The early part of the semester was ungraded, formative, small-project work, and I found myself needing a way to assign grades to the students. It may be worth mentioning that I don't actually want to assign grades to them, but it is a requirement of the course, and so I must do it responsibly.

I happened to stumble across a rubric I created in 2008, before my blogging days, while I was cleaning up my Web space. As I recall, this was an amalgam of several other rubrics I had found on the Web with an ounce of invention. This was my inspiration to create the following rubric:

	3	2	1	0
Commitment	Attended all scheduled team meetings or notified the team of absence.	Missed team meetings, with notifications, with enough regularity to be problematic.	Missed one or more team meetings without notifying the team.	Regularly missed team meetings without notifying the team.
Participation	Contributed to sprint planning, sprint review, and sprint retrospective meetings.	Did not contribute to one of the following: sprint planning, sprint review, or sprint retrospective meetings.	Did not contribute to two of sprint planning, sprint review, and sprint retrospective meetings.	Did not contribute to any of sprint planning, sprint review, and sprint retrospective meetings.
Communication	Clear standup reports on what was accomplished, what is planned, and identification of impediments, to allow the team to continue to make progress.	Some unclear standup reports that inhibit team progress.	Regularly unclear standup reports that inhibit team progress.	Did not give standup reports.
Technical contributions	High quality technical contributions that facilitate success of the team.	High quality technical contributions that do not directly facilitate the team's success.	Low quality technical contributions that frequently require redress by other team members.	Low quality technical contributions that inhibit success.
Attitude and Leadership	Listens to, shares with, and supports efforts of others, and actively tries to keep the team together.	Listens to, shares with, and supports the efforts of others.	Frequently fails to listen, share, or support teammates.	Displays an antagonism that inhibits team success.

I created this after the team's first sprint, and parts of it are reactions to problems the team was having. For example, in the team's first sprint retrospective, they identified that many had spent inordinate time on aesthetic issues that were unrelated to the user stories. Hence, there is a distinction between those technical contributions that facilitate success and those that do not. Note that the numbers across the top relate to my triage grading approach; some day I will write a blog post about my experiences with it, but for the time being, suffice it to say A=3, C=2, D=1, F=0.

I shared this rubric with the team during Sprint 2, and at the end of the Sprint, I asked each person to do a personal evaluation. This was designed as a formative evaluation for the student as well as a trial run of the rubric. From reading the students' self-evaluations, I think the instrument worked as intended, and we're going to do self- and peer-evaluations at the end of this sprint.

In my other course (CS222), students are working in teams of four on six-week projects of their own design. They just finished their mid-project milestone presentations, and I've modified and distributed the rubric to this other group as well. I've highlighted those cells that have changed.

	3	2	1	0
Commitment	Attended all scheduled team meetings or notified the team of absence.	Missed team meetings, with notifications, with enough regularity to be problematic.	Missed one or more team meetings without notifying the team.	Regularly missed team meetings without notifying the team.
Participation	Contributed to project planning, implementation, testing, and milestone presentations.	Did not contribute to one of the following: project planning, implementation, testing, and milestone presentations.	Did not contribute to two of the following: planning, implementation, testing, presentation.	Did not contribute to three or more of the following: planning, implementation, testing, presentation.
Communication	Clear reports on what has been accomplished, what is in progress, and what stands in the way, thereby facilitating progress.	Sometimes is unclear about what has been done, what is in progress, and what stands in the way, creating minor impediments to progress.	Is regularly unclear about what has been done, what is in progress, and what stands in the way, creating significant impediments to progress.	Communication patterns directly disrupt team progress.
Technical contributions	High quality technical contributions that facilitate success of the team.	High quality technical contributions that do not directly facilitate the team's success.	Low quality technical contributions that frequently require redress by other team members.	Low quality technical contributions that inhibit success.
Attitude and Leadership	Listens to, shares with, and supports efforts of others, and actively tries to keep the team together.	Listens to, shares with, and supports the efforts of others.	Frequently fails to listen, share, or support teammates.	Displays an antagonism that inhibits team success.

These students are not using and are not familiar with Scrum, and so the Scrum-specific references needed to change. I am not that happy with the articulation of participation above, in part because we invested a lot of time on test-driven development, and I fear the students will be confused about separation of testing and implementation. The "communication" entry is clearly influenced by the idea of the Daily Scrum, but I have not previously suggested this particular reporting strategy to the students.  My hope is that by asking them to conduct rubric-based self- and peer-assessments, they will gain some insight into better ways to organize themselves. As with the game programming students, I am using this first pass as a formative assessment, and also as an instrument to help me identify teams that are having problems with team members—an unfortunate but oft-inevitable situation.

The students in CS222 had their first group experience with rubric-based evaluations this week when they evaluated each others' milestone presentations using this instrument:

Presentation requirement	Satisfactory (3)	Probationary (2)	Unsatisfactory (1)	Unacceptable (0)
Executable release	Stable and appropriate demo of executable release	Unclear demonstration of stable executable release	Unstable executable release	No demo
Milestone #1 Estimate	Clearly met milestone estimates	Clearly met a reasonable subset of milestone estimates	Did not meet an adequate subset of milestone estimates	Progress on initial estimates was not addressed
Milestone #2 Estimate	Clear presentation of well-articulated estimates for next milestone	Clear presentation of poorly-articulated estimates for next milestone	Unclear presentation of poorly-articulated estimates for next milestone	Milestone #2 estimates not addressed
Software architecture	Good presentation of reasonable software architecture	Decent presentation of reasonable software architecture	Unclear or awkward software architecture	No presentation of software architecture

I printed these on small slips of paper. On Tuesday, we had four presentations, so I instructed the students to pick up four slips as I talked through the structure and purpose of the rubric. On Thursday, we had five presentations, so I left the slips in the same place in the back of the room and I put on some mood music.

I was a little disappointed that none of them stopped to tell me how incredibly clever this was.