Peer evaluation rubric

I read through a portion of Computer Science Project Work. If found some of the book to be difficult to read because it is written for the UK higher educational system, whose structures I do not fully comprehend. It makes matters worse when the authors throw around terms that I can only assume are equivalent to AP, credit-hours, FTE's, and such in the United States. Ignoring the specifics of the implementations, I did find some good tips that I can adapt to my own practice. In fact, one of their key contributions is a studied observation that transfer expects transformation. That is, the transformation that one applies to found scholarship is an indicator of transfer of practice.

Most of my notes from reading the book fit handily into one of my notebooks, but there is one piece I would like to share and archive here: a sample peer- and self-evaluation form from pages 250-251. It is presented as part of bundle 8.6, "Moderation Using Student Input." The form is in two parts: management characteristics and technical contribution. Both use a seven-point Likert-type scale, but while the first part uses distinct semantic labels for each end-point,
the second simply uses "well above average," "average," and "well below average" for all categories.

These are the categories and scale labels for the management skills portion.

Category	High-end	Low-end
time management	highly organized	unreliable
responsiveness to others	respects views	domineering
coping with stress	always calm	panics easily
cooperation	always cooperates	goes own way
self-confidence	able to take criticism	can't take criticism
leadership	takes initiative	follows others
problem analysis	incisive	woolly
project management	best practice	activity lacks coordination
project evaluation	systematic and objective	casual and subjective

The categories for the technical contribution portion are

• Task Analysis
  
• Conceptual Design
• VDM
• Manager's Meetings
• Team Meetings
• Low Level Design
• Coding
• Testing
• Documentation
• Demonstration

Both forms provide room for open-ended comments as well.

VDM? I have no idea. Unfortunately, while the book acknowledges that portions are written by different authors, it does not identify who wrote this part, nor whether or not is based on existing scholarship. I would like to see an evaluation of this instrument in particular, since it seems to be the only recommendation in the book that is provided in such specificity—despite knowing and withing acknowledging that it should be transformed to be transferred!

On first glance, I was impressed by the custom labels. I love the use of words like "incisive" and "woolly" to describe how a student interacted with a learning experience, and I think it's good for students to think about the capacity to take criticism as an indicator of self-confidence. As I looked closer, I noticed scales like "leadership," which here is measured in terms of taking initiative versus following. I'm not sure I like that dichotomy, but again, that's where I would like to read a formal evaluation of this instrument in particular. The more I read about using this instrument (which was less than one page), the less I liked it, as the author suggested using a fairly straightforward scoring mechanism: average students' self-assessment with their peer assessments, sprinkle with up to 5% secret sauce, and call your work done. Once again I have to admit my ignorance of the grading environment in which this scheme was designed, but I would be unhappy with this in an A/B/C/D/F scheme. Specifically, where would one cut off, say, "A=Excellent" with such an approach?

In any case, there are some interesting categories here for student self- and peer-assessment, and I can see the value in handing out a form like this at the start of a project to get students thinking about how their contributions may be classified. No one wants to be woolly.

Allergies

Regular readers of my blog may wish to know that, as of this week, I am back on my allergy medicine. Also, I was not spontaneously promoted.

My morning in pictures

Ah, the first day of the Fall semester. My game programming class meets at 8am. Let me tell you about the morning so far.

I came into the Math lounge to put my lunch into the refrigerator. I pulled it out of my backpack and slung my backpack up onto my shoulder. Not having latched my bag shut, my favorite mug fell from it and shattered into a million pieces on the ground.

Favorite mug, before the accident

I picked up the bigger pieces, cutting my finger in the process, then asked the receptionist in the Math department to call custodial to get the rest of the shards.

Returning to my office, I realized I still had my lunch bag in my hand, so then I went back and put it in the 'fridge.

I poured my coffee from my travel mug into the back-up mug in my office, my good-old ACM mug.

Old Faithful

A bit of background on my morning coffee: Last night, I noticed that both of my french presses were in the dishwasher, which was nigh full, and I expected that the dishwasher would be run overnight. Of course, I did not actually run it, and so in the morning, I was left without a clean press. My backup plan was already in place: use my cold-brew coffee concentrate! I mixed it with some boiling water and half-and-half, and that's what I brought to the office with me. The result was that when I tested the coffee, it was not quite as hot as I would have liked.

Problem easily solved. I went into the CS conference room to put my coffee in the microwave...

Conference room microwave, outside

and when I opened the door, I saw this:

Conference room microwave, inside

Well, ick. The red sludge looks like jam to me, like someone tried to microwave a peanut butter and jelly sandwich, realized it was a bad idea, and left the detritus behind. By this point, it was already about 7:55,  and I didn't want to dally—I just wanted hot coffee. I went ahead and put my mug in for 30 seconds, hoping it wouldn't come out tasting of smoked jam. When I opened the door after thirty seconds, a little flying insect came out of the microwave. Can that even happen?

I took the coffee out of the microwave and headed down to my classroom, a cozy little room we call RB104. It's a conventional "smart classroom," meaning there's a projector, dongles for attaching a laptop, and a badly designed teaching kiosk. Some of the students were waiting outside the door, so I proceeded to unlock the door with my key. In fact, I locked the door, which I only discovered after unsuccessfully trying to open it; it had been unlocked from the beginning. Properly unlocking the door and flipping on the lights, here's what I saw when I came in:

The land that custodians forgot

This is 8AM on the first day of classes for the semester, and there's a pile of crap on the desk! Notice that there are two unclaimed notebooks from at least the Summer session, though I think it might be from last Spring, since they looked familiar. Let's take a closer look at that pile on the near side of the table.

"Stuff"

The mess of cables is normal for the room. There's cabling for the mouse, ethernet, and video. They are sitting on top of a discarded notebook, beside the room's nonfunctional remote control. On the left is a chunk of chalk, an uneaten but ratty-looking Ricola, and a small leaf.

The real winner, however, is this thing, from the middle of the table.

What is that thing?

I moved the mouse next to it for scale. I'll say it again: What is that thing? At first I thought it was a wad of candle wax that had been handled by someone with soot-covered fingers. It looks kind of like a big ugly toenail. I really have no idea. I did not touch it.

After cleaning up the desk, I plugged in my laptop to the projector and found that the autofocus was not working. A friendly student adjusted it manually from the middle of the room.

The good news is that everything got better from here. I talked to my students about how I got interested and involved in game development, showed them some of my and past students' work, and talked about the goals for the semester. They seem a good sort, and despite a rocky start, I think it will be a good semester.

Good morning!

The Escape

When a student reminded me that the next Ludum Dare competition would be this weekend, at first I was excited about using it as a creative way to ring in the new semester. My plan was to learn more of the toolchain involved in making a simple game in Unity in order to leverage that experience in teaching Game Programming this semester. I spent some time this past week tinkering with Blender for the first time, watching tutorials and making some very simple models. As the weekend drew closer and the real stresses of starting the semester began to rear their heads, I realized that my time would probably best be spent in a combination of last-minute planning and spending quality time with the family.


Ludum Dare announced their theme for the 48-hour game development competition Friday evening: "Escape." After sleeping on the theme, I was inspired Saturday morning to make an entry. So, for your gaming pleasure, I present to you The Escape.

• Windows version: Extract the zip file and run the enclosed exe.
• Mac OS X version:  Unzip it and it should magically work.

You could call it a puzzle game.

I'll write more about the development in a later post.

Continuing Academically Adrift

I was excited to read the final chapter of Academically Adrift, entitled "A Mandate for Reform." Although I enjoyed reading it, the final chapter was neither as revolutionary nor as specific as I hoped. The authors discuss several inhibitors to change, the most daunting of which is the lack of demand for learning, which I wrote about a few days ago. Their suggestions match much of what my own Future of Education Task Force concluded: we need a greater focus on student learning. This support must take the form of both respect and rewards, since as long as teaching is institutionally considered a lesser responsibility to research, then there will be no change.

Let me be clear that I enjoyed reading the book and consider it a significant contribution to the field. Perhaps it was unfair of me to expect that their conclusions would be more immediately actionable, since their stated goal was research, not reform. Consider, for example, the authors' suggestion that effective reform needs to happen from within the current system, allowing actors to innovate within the existing space of higher education. This compares to my Future of Education Task Force recommendation for what we called "The University Sandbox," an initiative that would be authorized to work within the university while ignoring conventional assumptions of higher education. Our recommendation was not codified to the point of being immediately implementable, but I was hoping that Academically Adrift would provide more examples of how institutions might consider approaching the problem of fostering educational innovation.

In the final chapter, the authors to begin to fall into the social science trap of claiming their results are broader than what they actually found. Their study used the Collegiate Learning Assessment (CLA) to look specifically at critical thinking, complex reasoning, and writing, measuring a cohort as they entered college and as they finished their sophomore year. They found that individual studying improved students' scores while group study did not, and the last chapter begins to conflate CLA achievement with learning in general. This matches my experience and intuition: a student's ability to individually write an essay should only be improved by that individual practicing writing essays. However, they neglect to mention that CLA-measured skills are not the only skills that need to be learned. For example, there is a body of research in Computer Science education that has shown that pair programming—undeniable a form of group study—improves learning and retention. The authors clear bias for individualized rigor could have been balanced with a caveat that there are disciplinary idiosyncrasies.

The final chapter's message could be summarized as, "Be deliberate about fostering student learning in higher education." The criticism of cargo-cult active learning was refreshing, given how many papers I have reviewed and presentations I have heard where the researcher has not demonstrated any significant understanding of the fundamental tenets of learning. I also appreciate their argument for more widespread teacher-training within doctoral programs. The crux of their argument is that for the foreseeable future, many more Ph.D.s will be adjunct faculty or in teaching roles than tenure-track at research-intensive schools.

There is one immediately actionable recommendation for faculty that the authors return to time and again: be rigorous. Their own study and several related studies have shown that rigorous coursework improves student academic achievement. This may seem obvious to those outside the ivory tower, but those of us on the inside have seen how easy it is to trade away rigor for easier teaching. This comes back to the delicate equilibrium I wrote about the other day, in which learning is generally not considered worth institutional sacrifice. The good news is that students are hungry for it.

Despite any nitpicking, I do strongly recommend the book, especially to anyone who is just starting to dip their feet into higher education reform. It is probably best read in conjunction with a book more specifically about learning, such as How People Learn, How Learning Works, or, for like-minded technocrats, Pragmatic Thinking and Learning. (NB: I have not yet read the second book mentioned above, but I have heard good things about it.)

Academically Adrift and the Status Quo

I am nearly finished reading Academically Adrift: Limited Learning on College Campuses. In case you are not familiar with the book, here's the short version. The authors studied 2000+ students from 24 four-year institutions of higher education, giving them tests that measure general (as opposed to discipline-specific) skills that practically all colleges claim to enhance: critical thinking, complex reasoning, and writing. The tests were given before starting college and halfway through the sophomore year, and so they were focusing on the effects of common core curriculum experiences. The results were dismal, with the average gain being very low and about 45% of students showing no improvement.

The fifth chapter is entitled "A mandate for reform," and it makes an important point about higher education: there is no crisis. There is no crisis because there is nothing that, right now, is threatening the existence of higher education as an institution. All of the stakeholders are in good position to maintain the status quo. Parents grumble about tuition but continue to pay it in exchange for credentials for their children. Students readily seek out courses that provide little challenge, require minimal work, and allow for plenty of time for social activities, while still maintaining progress towards credentials. Professors minimize interaction with students, exchanging unchallenging courses for high course evaluations and therefore more pay, while also gaining more time for other scholarship—which also leads to more pay. Administrators keep students happy in order to improve recruitment and retention, focusing on revenue streams. Legislators and other politicians gain a credentialed citizenry, improving metrics against competing states, nations, etc.


This gives me another perspective on why it has been so challenging to try to push for learning reform within higher education: "learning" is completely absent from the status quo. There is nothing about the current balance of powers that supports, rewards, or encourages undergraduate learning, and in fact, to do so would upset the balance.

From page 127, "Many higher-education administrators and faculty today have largely turned away from earlier conceptions of their roles that recognized that providing support for student academic and social development was a moral imperative worth sacrificing for personally, professionally, and institutionally."  There were some on the Future of Education Task Force who were willing to bring up the concept of institutional sacrifice, but I have yet to see this manifest on the Strategic Planning Task Force.

Looking at it positively, this book gives me more ammunition to support what I believe is the moral imperative to fundamentally improve the higher educational system. The book is well-written and contains plenty of references to supporting literature.