Wednesday, March 28, 2018

Students' preference for discussion over prototyping, despite instruction to the contrary

Monday afternoon was a perplexing one, forcing me to look back at my goals and direction for a variety of reasons. What kicked it off was my one o'clock meeting with my HCI class. After Spring Break, we started in on our final project, and given their surprising reaction to our pre-break mini-project, I decided we would use the final project to gain a better understanding of the double diamond approach rather than try to introduce a different model. Briefly, before break, we did a quick run through the double diamond; in evaluating their results, it was clear that the vast majority of students did not invest the time to understand the context, let alone to identify a real problem. Indeed, what seemed to happen is that they chose to do something they could do rather than trying to solve a real problem. That shook me pretty hard—hard enough that I realized I couldn't let that be their broken understanding of the process.

We spent a week on the discovery phase, and I pushed them out into the field to talk to real human beings. Based on this, they had to make a few empathy maps and personas. They then interpreted these into journey maps, almost all of which were identical—not because of academic dishonesty but because of collective myopia. From this, they identified the problems they would work on, and that brings us to Monday: the first day of the "develop" phase, in which we would review and practice making low-fidelity prototypes.

I started by asking the class to list the tools of low-fidelity prototyping. They quickly came up with paper, markers/pens/crayons, and PowerPoint, along with other lightweight drawing tools not specifically designed for prototyping but certainly amenable to it. Their next answer was people, which surprised me but I think is appropriate. I added whiteboards, and I pointed out that one of the students had previously deployed a system specifically for UI prototyping (uxpressia by name, but that's just one example among many).

I created a second column, which—since I had sort of backed myself into a taxonomic corner—we called the "Anti-Tools" of low-fidelity prototyping. What sorts of things should we avoid? One student quickly mentioned code, which I agreed is exactly right: avoid code until it's the best tool. Another mentioned templates, which at first I didn't understand, but as he explained it, he was really talking about locking yourself in to particular approaches too early: a template is a reusable abstraction, but we don't know a priori that the given abstraction is appropriate. They paused here, and it was my chance to introduce two critical ideas; indeed, the primary reason for the exercise was for me to share these two points. I added brainstorming, which required me first to define the term—I am regularly frustrated by how students want to use this as a trendy synonym for "thinking." I briefly explained to them that brainstorming in a group will tend to push early convergence rather than divergence: that the best approach was to just start making. The second one I put up was related: analysis paralysis and discussion. A kissing cousin of brainstorming, I explained that I have seen practically every team I have mentored fall into this trap, thinking that sitting and talking about a problem will help us solve it. It won't. Primed by these observations, I returned to the positive column and suggested that timeboxing is one of the greatest tools of creative prototyping.

With that, they voted on a 15-minute timebox, I set the timer, and they got to work.

Or something that looked like work to them, anyway.

There was one group whose only discussion was about distributing index cards—I had told them a low-fi prototyping exercise was coming up, and so a few people brought supplies—and then they set to work, cutting, drawing, crumpling. Another group did a brief powwow before going in what appeared to be a similar direction, although that might be up for interpretation. The rest of the class, roughly 70%, engaged in discussion. One group took to the whiteboard to draw something like a flowchart, the rest sat in their clusters and discussed while they drew.

I observed all of this happening, of course, and as the timer kept ticking, I kept thinking, "Any moment now they will break and start actually prototyping, right?" At about ten minutes in, it was clear that this was not going to happen, so I wrote two questions on the front center board: What makes it a prototype? and What makes it a good prototype?

When the timer went off, I invited them to look at these questions. Honestly, I wasn't very hopeful in getting good answers, based on what I saw, but we went to the board anyway. In answer to the first, a student (from that first group) said, "It's testable." That's exactly right: a prototype has to be testable, otherwise it is something else. I pointed out a secondary part of the definition is that the prototype points to a possible future. I couldn't think of the word for this, but a student suggested, "portent." In retrospect, the word is "portentous," but still, that's a 10-cent word. We moved to the second question, and right away a student (from a different group) said, "It gives you good feedback." That's the right idea here too, in my opinion, although I tell you what I told them, that I prefer to frame it as, "It answers a design question."

This is interesting, isn't it? They seem to understand the theory. I asked them, then, how many of them had prototypes from the 15-minute timeboxed exercise? The students in that first group, they all raised their hands, and rightfully so. The only other hand that went up was from one student in a group of three. I asked them to dive into this: they had all worked together, on one artifact, in discussion, during the timebox, and only one of them characterized it as a prototype. I invited them to describe their disagreement, and one of them attempted to justify that what they made was a prototype because it had arrows and indications explaining to someone how it would work. It was clear to me, and I think to the rest of the class, that this was not a prototype at all, but some kind of sketch or schematic.

My next question to them was, "What did you notice that was different about how the groups worked?" They all seemed to recognize that the people who came up with prototypes used their fifteen minutes to create their prototypes, while everyone else engaged in discussion. I explained to them again how this phenomenon was something I had seen many times before, particularly on immersive learning teams, where I advise working on prototypes and instead, students talk to each other—at length, with no real output.

I ended class, then, with a challenge: first, that they actually follow the instructions I give; second, that perhaps they consider breaking their teams in half, with half doing timeboxed prototyping and half focusing on brainstorming and discussion, and compare the results at the end. Honestly, I would rather they do the first, but I'd be satisfied if they did the second.

As this post was bouncing around my head this morning, I realized that I have seen a similar phenomenon before, regularly, in my teaching. It is the phenomenon of CS222, where I tell the students that they need to start with CRC cards, then make a list of tasks, then use test-driven development to approach those tasks. From many years of teaching the course, I my best estimate of what actually happens is that teams get together, talk a bit, and then start programming. This comes, in part, from student essays admitting to it. This inevitably leads to failure of the two-week project or the first iteration, and I provide vociferous feedback about what is wrong and how to improve. The "how to improve" is, essentially, to follow the steps. Yet, students don't. Even to the third iteration, I regularly have 20-40% of teams who are still not following the steps that I have laid out and that they know they will be evaluated on. Whether or not they ever read the requirements is moot here, since they all look at the feedback I provided and claim to be seeking to improve; yet, if we judge intention by results, the real intention seems to be maintain the status quo rather than learn something new.

I am left with a burning question about how to push my role as a mentor. Should I be interrupting their 15-minute timebox to point out what I see, in the hopes of pushing them more quickly in the right direction, or do I need to let them make this little mistake so that they can learn from it? I am afraid of them treating me like an exam proctor: if only the proctor weren't here, we could just collude on the exam and get out of here. I have been thinking about how this applies in CS222 as well, and I have been thinking about having more formal check-in points; for example, if teams had to turn in their CRC cards two days into an iteration, it would show them that I mean business. However, it would also mean that they would do it because I was collecting it and not because they should in order to learn what is being discussed.

My next meeting with the HCI class is coming right up. I think I may begin class by asking them to share the processes they used to construct their prototypes, and perhaps I will push them a bit further into a root cause analysis, to consider why they didn't follow the instructions they were given.

No comments:

Post a Comment