For context, here are a few links:
- Revising Courses, Part III: Advanced Programming
- Tweaking Achievements and Reflections
- Fall and Spring course descriptions
My intention was to use the same format in the Fall and Spring, to get more consistent data for the study, but there was one piece that I felt I had to change between semesters: the tight binding of achievements and reflections. In the Fall, students' weekly writing assignments had to tie directly to their achievements. Late in the Fall, I realized that this was an unnecessary restriction on the topic of their writing, and that what I really wanted was for them to be reflecting on any course experience, whether it was the achievements or not. The Spring reflection essays were more varied and much less formulaic. In fact, some delved into personal stories that were quite moving and inspiring. Of course, I cannot say more about that publicly, except to point out that it was the kind of moment that makes me proud to be a positive influence in a person's life.
There were still some hiccups with the reflective writing. There were three parts to each essay in the Spring—one less than Fall—but still students submitted without hitting each part. I can probably mitigate some of this by spending more time with Blackboard, copying the requirements from the course description to each assignment rather than referencing the course description. The programmer in me wants to simply reference the authoritative document, but experience has shown that students fail to check the reference.
I am also concerned about students' balance of attention between course activities. The achievements are mostly programming-related and are designed to integrate with the final projects. The best students saw this and leveraged it, but it feels like the capacity to recognize this dropped exponentially with students' effort or attention. Reflections, similarly, were supposed to tie all these experiences together, and the best students got this; many other students wrote trite, predictable, platitudinous, or probably-false reflections just as a way to earn course credit. I believe that anyone can learn from reflective writing, and I suspect many of my students did, but I also suspect that many more did not learn significantly from this activity, since they treated it as a box to be checked. This makes me wonder if the whole system of reflections should, itself, be an achievement, rather than required. More on this in a moment.
The achievement system led to students engaging in a wider variety of learning activities than I had seen before. Furthermore, much of this was self-directed: I gave criteria whereby achievements were earned, and students were responsible for finding the resources to earn them. Of course, I myself was a resource, but one could almost draw a line between high and low grades as to who recognized this. This is a common problem, not related to achievements I think, where "strong" students will talk to professors, ask questions, seek guidance, and "weak" students suffer from culturally-imposed fears that prevent this (fear of showing weakness, fear of authority figures, etc.).
All of the achievements were posted to a shared folder on Google Docs where students could see all that each other have posted. All of my evaluations were posted as comments on these shared documents. It is not clear to me the extent to which students read each others' posts and learned from them, but I know that it did happen at least sometimes. I am uncertain whether students read each others' posts for content or for form, but it seems to be a bit of each: from what I gather, ambitious students would scan others' posts and see where I left comments, and then try to avoid these pitfalls. As above, however, we see the weaker students making the same mistakes that their peers made just a few lines above them.
Several achievements dealt with what I called "production code," meaning that it had to be the students' final project code or parts of Open Source software. This definition, which seemed clear to me, caused students trouble in both semesters. Regularly, students would submit achievements that used code that did not fit this definition, sometimes the same students week after week. The confusion between current project code and old project code seems like a lack of attention to detail; however, some students confused public open source code with snippets on StackOverflow or tutorial sites. I need to clean up this definition in the future, making it clear that the point is to look at code that is part of a system, not didactic or decontextualized.
This leads to another unexpected oddity: in the final round of reflections and in the final exam, students used terms like "production code" as if they were universally meaningful, as opposed to only being meaningful in our class. I did not make it clear that this was a definition for us, and so I worry that students may go into interviews or other discussions and mistakenly express this as vocabulary. This was much worse with the achievements, in fact, many of which I tried to name in clever ways. For example, one achievement deals with Java's
toString, equals, and hashCode methods, and I called the achievement, "Master of the Big Three." I didn't know what else to call it when I was writing it up, and that seemed like fun, so I called it that. However, then I overheard students talking about "The Big Three" as if that meant something in an absolute sense. Similarly, I had several achievements dealing with design patterns, and since I see patterns as a way-of-programming, I called these "Pattern Disciple" achievements. Then I heard students talking about "pattern discipline"—they didn't even get the word right, but they were still using these corny terms as if they were accepted jargon. I am not sure how to fix this aside from coming up with much drier names for the achievements, but the punster in me will likely find this impossible.To get an 'A' in the class, students needed to earn "meta-achievements," which were coherent sets of related achievements. For example, the "Engineer" achievement required using UML and design patterns. (In retrospect, it may have made sense to call these "quests" instead, to better show the metaphor of an 'A' being at the end of the journey.) The members of one of the Spring teams did earn this achievement, although because of how I articulated the achievements, they had to incorporate the Decorator pattern where it really had no place. When I saw this, I was disheartened: what's the point of a pattern if it doesn't actually solve a problem in the domain, and would the students recognize the difference? When I expressed this doubt to the team, one of them actually defended my policies, saying that he was happy to have an excuse to study the pattern and use it, regardless of whether it "fit" the project. At least this guy saw the balance of pragmatic and academic here, and the result was positive for us both; however, I think I can circumvent this in the future by articulating quests in a way that ensures the activity is still motivated by the project context. Most students have difficulty finding the kind of intrinsic motivation that this one had!
Right at the end of the semester, I had an enlightening conversation with a student regarding the entire metaphor of achievements. I asked why he had not been more active in pursuing achievements when they were clearly tied to the course grade. His answer may have been an excuse, but it's interesting nonetheless. As a gamer, he conceptualized "achievement" as they are often treated in game design: as ancillary rewards, secondary to the primary gameplay. The discourse around the semester focused on the nine-week final project, and I can certainly see how students would see that as "the point of the course." From a game design perspective, these weren't really "achievements," then, as used in contemporary game design: they were essential, as missions or quests, waypoints or progress markers. Another student put it another way, saying that they weren't achievements at all, they were just a set of optional assignments. This is valuable feedback, and I see two ways out of this situation: re-name the achievements to reflect their essentially-mandatory nature; or actually make them optional, perhaps as gatekeepers to tiers of grades ('B' and 'A', for instance).
Achievement submissions were organized by topic, so all the "Master of the Big Three" achievements were on one page, for example. This was convenient for students to see what their peers had been working on, and what kind of feedback I had given to them. The resulting course document structure was like a wiki, then, with content organized by topic. I wonder how different the learner's experience would be if we took a portfolio approach, organizing the content by contributor. This, I think, would make it easier for individuals to see and think about what they did all semester. On the other hand, it would work against the goal of helping students see themselves as part of a community, and I suspect it would be less likely that students would look at each others' work, since one would not know whether a particular student earned a particular achievement or not. Also, it might lead to many mid-level students honing their sites on the portfolios of a small number of strong students; this would be problematic since, from my reading of the literature, students learn better from people at similar skill levels. This leads me to think that the wiki-structure approach is superior to the portfolio-structure approach, and now that it's on my blog, maybe I will remember this point as I consider what ideas to incorporate into my summer course revisions.
When I first imagined the achievement system, I envisioned the contribution of high-quality content on which I could provide meaningful, thoughtful responses. In fact, in my original design, I would not evaluate the achievement submissions at all, since each one had crisply-defined requirements, but I added a professor-review before the start of the Fall semester. In practice, there were many submissions that were not up to snuff. On one hand, this is predictable: if students will often submit assignment submissions without actually fulfilling their requirements, why wouldn't they do the same with achievement submissions? On the other hand, why would they do this? The achievements were not graded on some archaic percentage scale. There were only three grades: incorrect, acceptable, and correct-with-distinction (coded as 1, 2, and 3). The best I can figure is that students uploaded incorrect submissions because schooling culture encourages "submit and hope for points" as the dominant idiom, rather than "consider whether or not this is correct." (This is just one of many soapbox issues for me, where I point my finger squarely at... well, pretty much all the rest of formal education, which is difficulty because I have pretty small fingers. I could go on about how the dominant schooling culture discourages reflection, accountability, creativity, and deep learning while encouraging risk-averseness, but this parenthetical phrase is long enough.) I wonder, perhaps, if the numeric coding itself brought to mind students' conventional experience with points. Indeed, one student told me that he really wanted to get those '3's, even though, from a grading perspective, they were equal to '2': the only difference was one of pride. If I keep this aspect of achievements, I will try turning these into symbols, perhaps thumbs-down, thumbs-up, and ... I don't know, maybe Chuck Norris.
As a preliminary conclusion, I call the achievement-based grading system a success for this course. Students were able to focus on their nine-week projects and engage in meaningful content-oriented and reflective activity, with freedom to choose activities that aligned with their interests. At a minimum, I want to redesign some achievements to align more clearly and obviously with the projects, but I will hold off on major redesigns until I take some time to read my study data.
No comments:
Post a Comment