I mentioned in my last post that I decided to experiment with story mapping as an alternative to Scrum-style task tracking boards. My team had their first release on Wednesday, which led into an excellent playtesting session at Connection Corner. However, the team only was able to make the release due to the herculean effort of a few technical team members, who worked through the day on Tuesday to add missing critical features to the prototype.
It's worth dwelling on that phenomenon itself for a moment. This is part of a three credit-hour course, which means--according to federal standards--that we should expect nine effort-hours per week from the student during the fifteen weeks of the semester. The team has scheduled meetings for six hours per week, which I set up in order to maximize face-to-face time for this multidisciplinary team. At least one student put in about nine hours on Tuesday alone to help us meet this deadline. He didn't have to do this, in the sense that there would be no punitive grade assigned, and the shame at not having met our goal would have been shared by the entire team. Worse, we would not have been able to conduct our playtesting session, which would mean we would still be grappling with fundamental uncertainties today. On one hand, we need to congratulate the students who put in significant extra effort to help the team meat its goals (and indeed we gave a round of applause on Wednesday); with the other hand, however, we need to admonish ourselves for putting us into a situation where so much work was required before an important deadline. How did we get to that point? It seems there are only two options: either it was unclear to team members that we were headed toward missing the goal, or they did not care that we were going to miss the goal. The behavior of this subset of students who put in extra hours leads me to conclude the former.
On Monday, many stories remained on the spine of the story map, all or nearly all with student names on them as responsible parties. On Wednesday--the day of the deadline--I wrote a note on the board in the morning to ensure that the story map was up to date for our review & retrospective meeting, and even then, the story map was untouched.
Every student team goes through some growing pains where they realize that they need to be more careful about communication and planning. The retrospective meeting on Wednesday confirmed that the students were aware of the need to coordinate more prudently and tactically. I am left wondering, however, how much of this is related to the story map? I could look at the story map and tell that we were not making progress, as notes were not moving, but I think they were blind to this. I suppose this iteration will demonstrate to us whether they will hold themselves more accountable to the practice or not. During the retrospective, a student contributed what I think will be a valuable action item: immediately after the stand-up meeting, gather around the story map to review where we are. Today was a planning meeting for the next iteration, and so it will take until Monday to get any sense of whether this helps or not.
Another item that gives me some unease is the tracking of stories (in a story map) vs tasks (with a conventional Scrum board). User stories are necessarily brief, but to address their need requires a decomposition of the story into discrete tasks. Using Scrum, this was done explicitly during the Sprint Planning meeting. For example, given a story like "As a player, I want to create my character" might break down into a design and an implement task, to make it clear that we should figure out a good user experience before programming it. However, in the last iteration, we saw several UI components that were thrown together by programmers--surely with good intention, but just as surely without any background knowledge in usability and game interface design principles! As above, there's too much ambiguity to drive a particular conclusion: did the students not know that they should step back and work with the team to design a good UI, or were they just working quick-and-dirty on a two-week prototype? The truth may be somewhere in the middle, and I suppose I can hope that if it is purely ignorance that this two-week prototype was a learning experience about the importance of critical design!
This leads me to one last point that I want to record here, so I don't forget. This two-week prototype was "slapped together" intentionally. The team agreed on several methodological rules governing source code, but they also agreed that these rules would not hold for the two-week prototype: this could be done "quick and dirty." Late in the iteration, I started looking more carefully at the code being committed to the repository, and I was struck by how one developer's "dirty" might be another developer's "abominable." When I said dirty, I meant we could be lax about things like having rigorous unit tests, or allowing several parameters to constructors rather than introducing a builder. Others clearly had a different idea! This strikes me as a fascinating study for software engineering research: how do different developers approach a project when told they can do it "dirty" vs. "clean", and what are the contributing factors in their decisions, and what are the inherent properties of the code they produce? Maybe someone has done this already, but if not, it sounds fascinating---and it clearly resonates with the kinds of discussions I have daily in CS222.
Showing posts with label evaluation. Show all posts
Showing posts with label evaluation. Show all posts
Friday, February 12, 2016
Monday, October 20, 2014
Managing Student Peer- and Self-Evaluations using Google Forms
I have written before about the rubric I use for students' self- and peer-evaluation. Here is the latest version that I have deployed this semester, copied from my 2011 post.
I have been conducting the evaluations on paper and transcribing the data into a spreadsheet for analysis. I realized last academic year that the analog form was preventing me from using self- and peer-evaluations more frequently. I suspect that if teams had to give and receive this feedback more regularly, more teams would be able to identify and mitigate collaboration problems. I decided to require self- and peer-evaluations for each iteration of my Game Programming and Advanced Programming courses, in part to push me to explore a streamlined, digital solution.
My first stop was Blackboard, which is required at my university at least for entering students' final grades. I found no support for the kind of group-restricted peer evaluations that interest me.
Enter Google Forms. If you have not used this before, it is an easy-to-use tool for creating forms, distributing them, and analyzing the results via integration with Google Sheets. I created a template that converts the tabular rubric into a series of five questions on a zero-to-three scale. To use the form for any particular class, I copy the list of student identifiers from Blackboard and paste them into the first two questions—the selection of the evaluator and the subject under evaluation. I could not find a way to make the template publicly readable without also making it publicly editable. If you want to try the form yourself, contact me and I'll share it with you via Google Docs, then you can make your own copy of it.
Once the students have completed the form, I use a pivot table to summarize the results. Specifically, I add a row for the subject of evaluation ("evaluatee"), then add values for the five rubric categories, summarizing by median. The pivot table is itself a sheet, and so I can add a new column that sums the medians to reach a student's contribution score.
Getting this information back to the student is still a manual process. My current approach is to make a new column in Blackboard's grade book and then sort the grade book table by student identifier to match the sequence in the pivot table. I step through, student by student, entering the contribution score and, in the comments field, the five medians in a simple, space-separated format. While this is somewhat tedious, it does force me to look at each line and identify whether I should intervene with a team or an individual.
The form cannot validate that each student completes the form exactly once for each peer. Another quick pivot table makes this easy to verify, however: add a row for each evaluator and a column for each evaluatee, add the evaluatee data, and summarize by "COUNTA". Anyone who has submitted more than one evaluation of the same person will be notable in the table, and each row's total should be the number of person in that student's team. This technique allowed me to recognize that a few students submitted multiple evaluations of a partner, presumably by mistake since the data matched up.
This approach presumes that students will be honest in identifying themselves, but the whole self- and peer-evaluation system already presumes honesty. I talk to my students about this, that I assume they will be honest with me because I have been honest with them.
Hopefully these notes will be useful to others who are thinking about how to conduct peer evaluations. If nothing else, this will help me remember how I handled the first iteration so that I can replicate it in the second!
| 3 | 2 | 1 | 0 | ||
|---|---|---|---|---|---|
| Commitment | Attended all scheduled team meetings or notified the team of absence. | Missed team meetings, with notifications, with enough regularity to be problematic. | Missed one or more team meetings without notifying the team. | Regularly missed team meetings without notifying the team. | |
| Participation | Contributed to project planning, implementation, testing, and presentations. | Did not contribute to one of the following: project planning, implementation, testing, and presentations. | Did not contribute to two of the following: planning, implementation, testing, presentation. | Did not contribute to three or more of the following: planning, implementation, testing, presentation. | |
| Communication | Clear reports on what has been accomplished, what is in progress, and what stands in the way, thereby facilitating progress. | Sometimes is unclear about what has been done, what is in progress, and what stands in the way, creating minor impediments to progress. | Is regularly unclear about what has been done, what is in progress, and what stands in the way, creating significant impediments to progress. | Communication patterns directly disrupt team progress. | |
| Technical contributions | High quality technical contributions that facilitate success of the team. | High quality technical contributions that do not directly facilitate the team's success. | Low quality technical contributions that frequently require redress by other team members. | Low quality technical contributions that inhibit success. | |
| Attitude and Leadership | Listens to, shares with, and supports efforts of others, and actively tries to keep the team together. | Listens to, shares with, and supports the efforts of others. | Frequently fails to listen, share, or support teammates. | Displays an antagonism that inhibits team success. |
I have been conducting the evaluations on paper and transcribing the data into a spreadsheet for analysis. I realized last academic year that the analog form was preventing me from using self- and peer-evaluations more frequently. I suspect that if teams had to give and receive this feedback more regularly, more teams would be able to identify and mitigate collaboration problems. I decided to require self- and peer-evaluations for each iteration of my Game Programming and Advanced Programming courses, in part to push me to explore a streamlined, digital solution.
My first stop was Blackboard, which is required at my university at least for entering students' final grades. I found no support for the kind of group-restricted peer evaluations that interest me.
Enter Google Forms. If you have not used this before, it is an easy-to-use tool for creating forms, distributing them, and analyzing the results via integration with Google Sheets. I created a template that converts the tabular rubric into a series of five questions on a zero-to-three scale. To use the form for any particular class, I copy the list of student identifiers from Blackboard and paste them into the first two questions—the selection of the evaluator and the subject under evaluation. I could not find a way to make the template publicly readable without also making it publicly editable. If you want to try the form yourself, contact me and I'll share it with you via Google Docs, then you can make your own copy of it.
Once the students have completed the form, I use a pivot table to summarize the results. Specifically, I add a row for the subject of evaluation ("evaluatee"), then add values for the five rubric categories, summarizing by median. The pivot table is itself a sheet, and so I can add a new column that sums the medians to reach a student's contribution score.
Getting this information back to the student is still a manual process. My current approach is to make a new column in Blackboard's grade book and then sort the grade book table by student identifier to match the sequence in the pivot table. I step through, student by student, entering the contribution score and, in the comments field, the five medians in a simple, space-separated format. While this is somewhat tedious, it does force me to look at each line and identify whether I should intervene with a team or an individual.
The form cannot validate that each student completes the form exactly once for each peer. Another quick pivot table makes this easy to verify, however: add a row for each evaluator and a column for each evaluatee, add the evaluatee data, and summarize by "COUNTA". Anyone who has submitted more than one evaluation of the same person will be notable in the table, and each row's total should be the number of person in that student's team. This technique allowed me to recognize that a few students submitted multiple evaluations of a partner, presumably by mistake since the data matched up.
This approach presumes that students will be honest in identifying themselves, but the whole self- and peer-evaluation system already presumes honesty. I talk to my students about this, that I assume they will be honest with me because I have been honest with them.
Hopefully these notes will be useful to others who are thinking about how to conduct peer evaluations. If nothing else, this will help me remember how I handled the first iteration so that I can replicate it in the second!
Thursday, January 2, 2014
Tweaking achievements and reflections
Before the start of the Fall 2013 semester, I wrote about the changes I was incorporating into CS222, the Advanced Programming class. With that semester behind us, I can say that it went well, although not without the need for tweaks. For the Spring, I am keeping the fundamental structure. Here is a summary and justification for the tweaks.
I am scaling down the number of achievements required for each grade level. The original plan in the Fall was two achievements per week. However, students were not as adept as I had hoped at aligning achievements and projects. The intention was that students would earn achievements by doing their projects, but the majority of the class tried to do achievements separately from the projects. In the Fall, we ended up making a change midstream, reducing the number required during the nine-week project to only one per week. For the Spring, I'm pulling back a little bit more, expecting two achievements per week during the first three weeks, then only one per week during the two-week "warm-up" project and the nine-week major project.
Another change addresses the intended achievement-project synergy: restricting the code that can be used to complete achievements. When working toward achievements, Fall's students were permitted to use any code to which they had a license. In practice, I had students submitting trivial modifications to terribly-conceived CS1 and CS2 projects all the way to the end of the semester. In Spring, students can use any such code during the first third of the semester, but once they start their nine-week projects, they must use either that code or public open source code.
I have also changed the how and when reflections are required. In Fall's section, students wrote one reflection per achievement. I am sure that the best students—those on the road to being reflective practitioners—found this fruitful, albeit time-consuming. The students with worse writing skills spent inordinate time on these reflections, time that should have been spent understanding the material instead. Their action was justified, perhaps, by the fact that reflections were a significant part of the course grade. Also, I allowed resubmission of the reflections, but I think my attempt to encourage mastery learning instead fostered the"submit and hope" approach that I find so endemic and disdainful. In the Spring, the students will instead be asked to write one reflection per week, and this reflection may focus on any course-related experience: achievements, in-class discussions, project team discoveries, and so on.
The structure of the reflections will change as well. In the Fall, students were asked to do three things in their reflections: characterize an essential question, describe implications to practice, and consider alternative perspectives. Many students struggled with these all semester. Although I believe that tackling these problems is good for students, there is a limited about of time students can devote to this class. Hence, I have reduced and rearticulated the requirements: in the Spring, students will be asked to address one of the essential questions ("characterize" caused undue problems) and describe implications to practice.
Despite the addition of achievements and structured essays, perhaps the most uncertain of my Fall experiments was in not grading the final projects. As mentioned above, the course design was supposed to foster students' intrinsic motivation to work on the projects, which would provide context for the achievements. My ambitious designs failed to account for the pragmatic, goal-directed nature of the undergraduate! Students realized that the projects were not graded, and many students found this unmotivating: they wanted their work to be recognized directly in their grades. I will be returning to my conventional approach, grading both the projects and their ancillary artifacts such as presentations and a final report.
A final significant change that I expect to bring vast improvements is getting off of Blackboard. We used Blackboard's wiki in the Fall as a place for students to post and share their achievement artifacts. The wiki is ghastly, and because it was so difficult to give targeted feedback, we ended up with a very poor signal to noise ratio. Often, a student would post something that was wrong, and I would give formative evaluation to that student, encouraging revision; the student would not revise, and within a few days, I would receive several similar submissions from students who assumed the original was correct. What a mess. In the Spring, we're going to use Google Drive instead. I am uneasy about requiring students to register for a Google Account in order to participate, but most have them anyway, and the technology stack is so vastly superior that I'm going to try it. I will set up a shared folder that will contain one document per achievement, and students will post their artifacts within those documents. My feedback can take the form of comments, which are visible to all readers. I will have a bit of manual work to translate these data into Blackboard's gradebook, but that's worth it to help close the gap between my feedback and peer learning.
I finished up the course description this morning and linked it to my Web page. The achievements themselves are basically the same as the ones I designed last Summer, with a few notable additions that reward technology (integration with relational databases and structured data parsers) as well as community-building (interviewing alumni and senior capstone teams). Each achievement has also been clarified as to what exactly the corresponding artifact is—some had been vague in the previous iteration.
The semester starts in just a few days, and I'm looking forward to meeting the new crop of CS222 students. I am not looking forward to teaching at 8AM and Monday's high of -6°F, but I am trying not to complain.
But seriously, 8AM and -6°F high temperature.
I am scaling down the number of achievements required for each grade level. The original plan in the Fall was two achievements per week. However, students were not as adept as I had hoped at aligning achievements and projects. The intention was that students would earn achievements by doing their projects, but the majority of the class tried to do achievements separately from the projects. In the Fall, we ended up making a change midstream, reducing the number required during the nine-week project to only one per week. For the Spring, I'm pulling back a little bit more, expecting two achievements per week during the first three weeks, then only one per week during the two-week "warm-up" project and the nine-week major project.
Another change addresses the intended achievement-project synergy: restricting the code that can be used to complete achievements. When working toward achievements, Fall's students were permitted to use any code to which they had a license. In practice, I had students submitting trivial modifications to terribly-conceived CS1 and CS2 projects all the way to the end of the semester. In Spring, students can use any such code during the first third of the semester, but once they start their nine-week projects, they must use either that code or public open source code.
I have also changed the how and when reflections are required. In Fall's section, students wrote one reflection per achievement. I am sure that the best students—those on the road to being reflective practitioners—found this fruitful, albeit time-consuming. The students with worse writing skills spent inordinate time on these reflections, time that should have been spent understanding the material instead. Their action was justified, perhaps, by the fact that reflections were a significant part of the course grade. Also, I allowed resubmission of the reflections, but I think my attempt to encourage mastery learning instead fostered the"submit and hope" approach that I find so endemic and disdainful. In the Spring, the students will instead be asked to write one reflection per week, and this reflection may focus on any course-related experience: achievements, in-class discussions, project team discoveries, and so on.
The structure of the reflections will change as well. In the Fall, students were asked to do three things in their reflections: characterize an essential question, describe implications to practice, and consider alternative perspectives. Many students struggled with these all semester. Although I believe that tackling these problems is good for students, there is a limited about of time students can devote to this class. Hence, I have reduced and rearticulated the requirements: in the Spring, students will be asked to address one of the essential questions ("characterize" caused undue problems) and describe implications to practice.
Despite the addition of achievements and structured essays, perhaps the most uncertain of my Fall experiments was in not grading the final projects. As mentioned above, the course design was supposed to foster students' intrinsic motivation to work on the projects, which would provide context for the achievements. My ambitious designs failed to account for the pragmatic, goal-directed nature of the undergraduate! Students realized that the projects were not graded, and many students found this unmotivating: they wanted their work to be recognized directly in their grades. I will be returning to my conventional approach, grading both the projects and their ancillary artifacts such as presentations and a final report.
A final significant change that I expect to bring vast improvements is getting off of Blackboard. We used Blackboard's wiki in the Fall as a place for students to post and share their achievement artifacts. The wiki is ghastly, and because it was so difficult to give targeted feedback, we ended up with a very poor signal to noise ratio. Often, a student would post something that was wrong, and I would give formative evaluation to that student, encouraging revision; the student would not revise, and within a few days, I would receive several similar submissions from students who assumed the original was correct. What a mess. In the Spring, we're going to use Google Drive instead. I am uneasy about requiring students to register for a Google Account in order to participate, but most have them anyway, and the technology stack is so vastly superior that I'm going to try it. I will set up a shared folder that will contain one document per achievement, and students will post their artifacts within those documents. My feedback can take the form of comments, which are visible to all readers. I will have a bit of manual work to translate these data into Blackboard's gradebook, but that's worth it to help close the gap between my feedback and peer learning.
I finished up the course description this morning and linked it to my Web page. The achievements themselves are basically the same as the ones I designed last Summer, with a few notable additions that reward technology (integration with relational databases and structured data parsers) as well as community-building (interviewing alumni and senior capstone teams). Each achievement has also been clarified as to what exactly the corresponding artifact is—some had been vague in the previous iteration.
The semester starts in just a few days, and I'm looking forward to meeting the new crop of CS222 students. I am not looking forward to teaching at 8AM and Monday's high of -6°F, but I am trying not to complain.
But seriously, 8AM and -6°F high temperature.
Friday, November 11, 2011
Rubric-based self- and peer-evaluation on team programming projects
I have written before about grading rubrics, and I am taking this opportunity to share some of my recent developments here. In my game programming class (CS315), I have eleven students working together on one project. We're using Scrum, and it's still going well. The early part of the semester was ungraded, formative, small-project work, and I found myself needing a way to assign grades to the students. It may be worth mentioning that I don't actually want to assign grades to them, but it is a requirement of the course, and so I must do it responsibly.
I happened to stumble across a rubric I created in 2008, before my blogging days, while I was cleaning up my Web space. As I recall, this was an amalgam of several other rubrics I had found on the Web with an ounce of invention. This was my inspiration to create the following rubric:
I created this after the team's first sprint, and parts of it are reactions to problems the team was having. For example, in the team's first sprint retrospective, they identified that many had spent inordinate time on aesthetic issues that were unrelated to the user stories. Hence, there is a distinction between those technical contributions that facilitate success and those that do not. Note that the numbers across the top relate to my triage grading approach; some day I will write a blog post about my experiences with it, but for the time being, suffice it to say A=3, C=2, D=1, F=0.
I shared this rubric with the team during Sprint 2, and at the end of the Sprint, I asked each person to do a personal evaluation. This was designed as a formative evaluation for the student as well as a trial run of the rubric. From reading the students' self-evaluations, I think the instrument worked as intended, and we're going to do self- and peer-evaluations at the end of this sprint.
In my other course (CS222), students are working in teams of four on six-week projects of their own design. They just finished their mid-project milestone presentations, and I've modified and distributed the rubric to this other group as well. I've highlighted those cells that have changed.
These students are not using and are not familiar with Scrum, and so the Scrum-specific references needed to change. I am not that happy with the articulation of participation above, in part because we invested a lot of time on test-driven development, and I fear the students will be confused about separation of testing and implementation. The "communication" entry is clearly influenced by the idea of the Daily Scrum, but I have not previously suggested this particular reporting strategy to the students. My hope is that by asking them to conduct rubric-based self- and peer-assessments, they will gain some insight into better ways to organize themselves. As with the game programming students, I am using this first pass as a formative assessment, and also as an instrument to help me identify teams that are having problems with team members—an unfortunate but oft-inevitable situation.
The students in CS222 had their first group experience with rubric-based evaluations this week when they evaluated each others' milestone presentations using this instrument:
I printed these on small slips of paper. On Tuesday, we had four presentations, so I instructed the students to pick up four slips as I talked through the structure and purpose of the rubric. On Thursday, we had five presentations, so I left the slips in the same place in the back of the room and I put on some mood music.
I was a little disappointed that none of them stopped to tell me how incredibly clever this was.
I happened to stumble across a rubric I created in 2008, before my blogging days, while I was cleaning up my Web space. As I recall, this was an amalgam of several other rubrics I had found on the Web with an ounce of invention. This was my inspiration to create the following rubric:
| 3 | 2 | 1 | 0 | |
|---|---|---|---|---|
| Commitment | Attended all scheduled team meetings or notified the team of absence. | Missed team meetings, with notifications, with enough regularity to be problematic. | Missed one or more team meetings without notifying the team. | Regularly missed team meetings without notifying the team. |
| Participation | Contributed to sprint planning, sprint review, and sprint retrospective meetings. | Did not contribute to one of the following: sprint planning, sprint review, or sprint retrospective meetings. | Did not contribute to two of sprint planning, sprint review, and sprint retrospective meetings. | Did not contribute to any of sprint planning, sprint review, and sprint retrospective meetings. |
| Communication | Clear standup reports on what was accomplished, what is planned, and identification of impediments, to allow the team to continue to make progress. | Some unclear standup reports that inhibit team progress. | Regularly unclear standup reports that inhibit team progress. | Did not give standup reports. |
| Technical contributions | High quality technical contributions that facilitate success of the team. | High quality technical contributions that do not directly facilitate the team's success. | Low quality technical contributions that frequently require redress by other team members. | Low quality technical contributions that inhibit success. |
| Attitude and Leadership | Listens to, shares with, and supports efforts of others, and actively tries to keep the team together. | Listens to, shares with, and supports the efforts of others. | Frequently fails to listen, share, or support teammates. | Displays an antagonism that inhibits team success. |
I created this after the team's first sprint, and parts of it are reactions to problems the team was having. For example, in the team's first sprint retrospective, they identified that many had spent inordinate time on aesthetic issues that were unrelated to the user stories. Hence, there is a distinction between those technical contributions that facilitate success and those that do not. Note that the numbers across the top relate to my triage grading approach; some day I will write a blog post about my experiences with it, but for the time being, suffice it to say A=3, C=2, D=1, F=0.
I shared this rubric with the team during Sprint 2, and at the end of the Sprint, I asked each person to do a personal evaluation. This was designed as a formative evaluation for the student as well as a trial run of the rubric. From reading the students' self-evaluations, I think the instrument worked as intended, and we're going to do self- and peer-evaluations at the end of this sprint.
In my other course (CS222), students are working in teams of four on six-week projects of their own design. They just finished their mid-project milestone presentations, and I've modified and distributed the rubric to this other group as well. I've highlighted those cells that have changed.
| 3 | 2 | 1 | 0 | |
|---|---|---|---|---|
| Commitment | Attended all scheduled team meetings or notified the team of absence. | Missed team meetings, with notifications, with enough regularity to be problematic. | Missed one or more team meetings without notifying the team. | Regularly missed team meetings without notifying the team. |
| Participation | Contributed to project planning, implementation, testing, and milestone presentations. | Did not contribute to one of the following: project planning, implementation, testing, and milestone presentations. | Did not contribute to two of the following: planning, implementation, testing, presentation. | Did not contribute to three or more of the following: planning, implementation, testing, presentation. |
| Communication | Clear reports on what has been accomplished, what is in progress, and what stands in the way, thereby facilitating progress. | Sometimes is unclear about what has been done, what is in progress, and what stands in the way, creating minor impediments to progress. | Is regularly unclear about what has been done, what is in progress, and what stands in the way, creating significant impediments to progress. | Communication patterns directly disrupt team progress. |
| Technical contributions | High quality technical contributions that facilitate success of the team. | High quality technical contributions that do not directly facilitate the team's success. | Low quality technical contributions that frequently require redress by other team members. | Low quality technical contributions that inhibit success. |
| Attitude and Leadership | Listens to, shares with, and supports efforts of others, and actively tries to keep the team together. | Listens to, shares with, and supports the efforts of others. | Frequently fails to listen, share, or support teammates. | Displays an antagonism that inhibits team success. |
These students are not using and are not familiar with Scrum, and so the Scrum-specific references needed to change. I am not that happy with the articulation of participation above, in part because we invested a lot of time on test-driven development, and I fear the students will be confused about separation of testing and implementation. The "communication" entry is clearly influenced by the idea of the Daily Scrum, but I have not previously suggested this particular reporting strategy to the students. My hope is that by asking them to conduct rubric-based self- and peer-assessments, they will gain some insight into better ways to organize themselves. As with the game programming students, I am using this first pass as a formative assessment, and also as an instrument to help me identify teams that are having problems with team members—an unfortunate but oft-inevitable situation.
The students in CS222 had their first group experience with rubric-based evaluations this week when they evaluated each others' milestone presentations using this instrument:
| Presentation requirement | Satisfactory (3) | Probationary (2) | Unsatisfactory (1) | Unacceptable (0) |
|---|---|---|---|---|
| Executable release | Stable and appropriate demo of executable release | Unclear demonstration of stable executable release | Unstable executable release | No demo |
| Milestone #1 Estimate | Clearly met milestone estimates | Clearly met a reasonable subset of milestone estimates | Did not meet an adequate subset of milestone estimates | Progress on initial estimates was not addressed |
| Milestone #2 Estimate | Clear presentation of well-articulated estimates for next milestone | Clear presentation of poorly-articulated estimates for next milestone | Unclear presentation of poorly-articulated estimates for next milestone | Milestone #2 estimates not addressed |
| Software architecture | Good presentation of reasonable software architecture | Decent presentation of reasonable software architecture | Unclear or awkward software architecture | No presentation of software architecture |
I printed these on small slips of paper. On Tuesday, we had four presentations, so I instructed the students to pick up four slips as I talked through the structure and purpose of the rubric. On Thursday, we had five presentations, so I left the slips in the same place in the back of the room and I put on some mood music.
I was a little disappointed that none of them stopped to tell me how incredibly clever this was.
Tuesday, August 3, 2010
Student evaluation plan for game programming
Continuing the series of posts on planning for Fall's game programming class, I would like to share my student evaluation plan. Thanks to all of my friends and colleagues who have taken the time to discuss these issues with me.
By way of background, I used both self- and peer-evaluation in my HCI class in Spring 2010, where students worked in teams on Android applications. I have always been suspect of both kinds of evaluations --- they are neither reliable nor valid, statistically speaking --- but it seemed like the best way at getting reasonable feedback. While I believe that most of the evaluations were honest, I found the lies to be much more interesting. Of course, the cynical mind leaps immediately to bloated self-evaluations, and there were some of those. Simple tricks like taking the median of the set of self and peer evaluations can address such issues assuming most people are honest. By the end of the semester, some students had an axe to grind and gave brutally honest feedback. As an observer, I think that much of this anger was justified. However, it was the inflated peer-evaluations that I found more interesting. Without breaking teacher-student confidentiality, I will say that these led to some fantastic discussions where the reason for "inflated" evaluations was not from a desire for reciprocation, but rather sympathy, maybe even pity for those who had made mistakes.
The point relevant to this post is, as Dr. House would say, people lie. Specifically, students lie to teachers. We have set up a cultural system in which it is inevitable, in which cheating is rewarded with material gain and, as The Spirit of Christmas put it so eloquently, "the rewards of virtue are largely spiritual, but all the better for it."
However, there is a force more powerful than material gain, a force that can strengthen leaders or destroy futures: peer pressure.
Without further ado, then, my experimental evaluation method is rubric-based, using the rubric shown below.
The numbers are based on my general grading rubric. In a nutshell, 3=A, 2=C, 1=D, 0=F.
As part of the Sprint Retrospective, students will grade themselves according to this rubric. Then, in a meeting with the team, they will announce what values they assigned to themselves. While students may lie to me in a moment of weakness, I think it's less likely that non-sociopaths would do this in front of their team of peers. Teams can reorganize between sprints, and so I will collect all of the self-evaluations in a spreadsheet that is shared with the other teams.
I do not have any middle management in the course design: there is no one between me (Scrum Product Owner) and the students (Scrum Developers / Team Members) that externally evaluates them. In the event that there are students who are not carrying their weight on the project, I have set up a mechanism for firing them from the studio. Here's how I put it in the draft of the course description:
As always, I welcome feedback and discussion.
By way of background, I used both self- and peer-evaluation in my HCI class in Spring 2010, where students worked in teams on Android applications. I have always been suspect of both kinds of evaluations --- they are neither reliable nor valid, statistically speaking --- but it seemed like the best way at getting reasonable feedback. While I believe that most of the evaluations were honest, I found the lies to be much more interesting. Of course, the cynical mind leaps immediately to bloated self-evaluations, and there were some of those. Simple tricks like taking the median of the set of self and peer evaluations can address such issues assuming most people are honest. By the end of the semester, some students had an axe to grind and gave brutally honest feedback. As an observer, I think that much of this anger was justified. However, it was the inflated peer-evaluations that I found more interesting. Without breaking teacher-student confidentiality, I will say that these led to some fantastic discussions where the reason for "inflated" evaluations was not from a desire for reciprocation, but rather sympathy, maybe even pity for those who had made mistakes.
The point relevant to this post is, as Dr. House would say, people lie. Specifically, students lie to teachers. We have set up a cultural system in which it is inevitable, in which cheating is rewarded with material gain and, as The Spirit of Christmas put it so eloquently, "the rewards of virtue are largely spiritual, but all the better for it."
However, there is a force more powerful than material gain, a force that can strengthen leaders or destroy futures: peer pressure.
Without further ado, then, my experimental evaluation method is rubric-based, using the rubric shown below.
| Category | 3 | 2 | 1 | 0 |
|---|---|---|---|---|
| Code Quality | Uses pair programming or all commits are signed by a team member. | Usually uses pair programming or most commits are signed by a team member. | Rarely follows pair programming or few commits are signed by a team member. | Has not contributed code |
| Commitment to Meetings | Attends and is prompt and attentive for all meetings. | Attends most meetings but is either late with significant frequency or is not focused during meetings. | Attends most meetings but is routinely late or unfocused. | Regularly misses meetings. |
| Facilitation (ScrumMasters only) | Facilitated the success of the team. | Mostly facilitated the success of the team with some problems such as failure to remove impediments or unnecessarily prioritizing owned tasks before team tasks. | Regularly failed to facilitate the team's success | Did not adequately serve the team. |
The numbers are based on my general grading rubric. In a nutshell, 3=A, 2=C, 1=D, 0=F.
As part of the Sprint Retrospective, students will grade themselves according to this rubric. Then, in a meeting with the team, they will announce what values they assigned to themselves. While students may lie to me in a moment of weakness, I think it's less likely that non-sociopaths would do this in front of their team of peers. Teams can reorganize between sprints, and so I will collect all of the self-evaluations in a spreadsheet that is shared with the other teams.
I do not have any middle management in the course design: there is no one between me (Scrum Product Owner) and the students (Scrum Developers / Team Members) that externally evaluates them. In the event that there are students who are not carrying their weight on the project, I have set up a mechanism for firing them from the studio. Here's how I put it in the draft of the course description:
If I receive reliable reports that a student is not contributing adequately, I will schedule an individual meeting with the student to discuss the matter. Failure to attend this meeting or resolve the problems will result in the student's being fired from the studio: the student fails and may no longer contribute to the project.I am hopeful that the combination of rubrics and public accountability will contribute to students' sense of ownership over the project. I am putting a great deal of trust in these students, and I want the evaluation plan to reflect that. As I intend to introduce the course, we're not just playing pretend: we are a game studio, and we have work to do.
As always, I welcome feedback and discussion.
Wednesday, July 28, 2010
Android class evals
I just read through the student evaluations from Fall's CS345/545 HCI class, a topic of many blog posts in Spring. They are quite positive, with the vast majority claiming to have enjoyed and learned significantly from a team-oriented, "real-world" project experience.
The request that shows up the most is for more explicit milestones during the semester. I had mostly left milestones up to the individual teams since each project had different characteristics, but in retrospect, I could have offered more assistance here. Perhaps next time I will use timeboxing techniques, forcing the students to complete vertical slices and deliver working code on a regular basis; without this, several groups meandered through feature development.
A few students requested more time invested at the beginning of the semester explicitly teaching Android development techniques. That is a reasonable request, although I would like to be able to sit with these students to discuss what they really want. For instance, one student requested more examples, but there are myriad examples online and packaged with the Android SDK. Another student complained about the online resources feeling "sort of scattered". These students clearly missed the point that such is the nature of modern software development: examples and resources are scattered throughout the Web and the library, and gaining experience in dealing with this was an explicit goal of the course. In the student's defense, I did not directly assess their capacity and learning with respect to navigating this information. Still, I think that this shows that students often come away from coursework with the wrong impression, that there is a Single Truth and the professor has it, and that every problem has a solution in the textbook somewhere.
Most of the departmental evaluation form deals with 5-point Likert scales, though under "Overall rating of the instructor", I received my first 10 out of 5. The student actually wrote in 6, 7, 8, 9, and 10 so that he or she could circle the ten. It made me grin, though I still recorded it as a five in my database.
I will share with you, loyal reader, the best of the bunch:
What aspects of the course did you especially like?
The freeness
How can the content and the instruction in this course be improved?
More formal structure.
Such are the paradoxes of higher education!
The request that shows up the most is for more explicit milestones during the semester. I had mostly left milestones up to the individual teams since each project had different characteristics, but in retrospect, I could have offered more assistance here. Perhaps next time I will use timeboxing techniques, forcing the students to complete vertical slices and deliver working code on a regular basis; without this, several groups meandered through feature development.
A few students requested more time invested at the beginning of the semester explicitly teaching Android development techniques. That is a reasonable request, although I would like to be able to sit with these students to discuss what they really want. For instance, one student requested more examples, but there are myriad examples online and packaged with the Android SDK. Another student complained about the online resources feeling "sort of scattered". These students clearly missed the point that such is the nature of modern software development: examples and resources are scattered throughout the Web and the library, and gaining experience in dealing with this was an explicit goal of the course. In the student's defense, I did not directly assess their capacity and learning with respect to navigating this information. Still, I think that this shows that students often come away from coursework with the wrong impression, that there is a Single Truth and the professor has it, and that every problem has a solution in the textbook somewhere.
Most of the departmental evaluation form deals with 5-point Likert scales, though under "Overall rating of the instructor", I received my first 10 out of 5. The student actually wrote in 6, 7, 8, 9, and 10 so that he or she could circle the ten. It made me grin, though I still recorded it as a five in my database.
I will share with you, loyal reader, the best of the bunch:
What aspects of the course did you especially like?
The freeness
How can the content and the instruction in this course be improved?
More formal structure.
Such are the paradoxes of higher education!
Subscribe to:
Posts (Atom)