Reflecting on CS445/545 Human-Computer Interaction, Fall 2020

This is the third in my sequence of end-of-semester reflections [1,2], and today's topic is my Human-Computer Interaction class. This was the course that I ended up designing twice this past summer. The end result was a great experience. Out of my three different courses in Fall 2020, I think this is the one where I was most consistently impressed with students' dedication to tackling the ideas of the course. And how many ideas there were! With the removal of face-to-face discussions, and knowing the limitations of discussion boards, I presented a wider variety of ideas to the students this semester than in the past. I will proceed by giving some brief thoughts on these additions.

One of the most obvious, sustained additions was adding Open Mind Platform, which I wrote about in October. It went as I had hoped, with my seeing evidence that students learned from the content of Open Mind itself but also thought critically about its implications for HCI design. I have also been in touch with the team that manages the platform, and I have reviewed some revisions that they are making to further improve the teacher's experience. Their dedication to improving Open Mind excites me, and I am glad that I was able to play a part in it. In fact, I recently added Open Mind Platform as an Achievement in my sketch of CS222 plans for Spring 2021, which will be the topic for another day's blogging.

I think I made good use of Scott Klemmer's lectures, balancing them against several of my own lectures that gave my perspective on various HCI topics. Whereas Klemmer's presentations were generic, I was able to focus mine on the specific assignments and tasks that I had in mind for my students. I do not know when next I will be asked to teach this course, but I will need to consider how I might reuse some of this course design for face-to-face classes.

Another valuable addition was Nielsen's Usability Heuristics. Given the challenges of empirical testing in the face of a global pandemic, performing heuristic evaluation was much more practicable. Nielsen's approach combined well with the ideas in Norman's Design of Everyday Things, Gestalt principles, and the accessibility heuristics we explored, showing students that there were multiple valid and useful ways to evaluate a system.

I am pleased with how the final project went. I assigned them a challenge that I myself faced earlier this year: taking a traditionally in-person Student Showcase from the CCSC Midwest conference and making it an online asynchronous event. Rather than just throw them at the problem, I scaffolded a series of six weekly assignments that walked through a reasonable, human-centered process. Only two weeks out of the six were spent on building, which I think surprised many of the students. The technical artifact was not the primary outcome of the process but rather a secondary one: the primary outcome was the report that students built on week by week. In the end, these ranged from about 15 to 50 pages long, depending on students' use of images and their attention to detail. After reading their final project submissions, I recorded a response video in which I encouraged them to consider the extrinsic value of this document, given that it provides clear evidence to potential employers that they themselves can follow best practices of design—not just parrot a few definitions from a textbook.

The final project was not without problems. Some students fell behind, and there was little recourse for them. I provided verbal instruction on how to progress; for example, if someone was unable to get a build working, I told them to reach out to classmates and continue the project using someone else's build. However, no one did that: they either made something up or gave up on the project. It is worth noting that the view counts on my weekly feedback videos were noticeably less than the course enrollment, so I am not even sure that the students in trouble knew that I had given them instruction. 

The best of the reports explicitly compared their results to the one that I used. I did not require this, but I was surprised that only of the students thought to use my solution as a benchmark. I was a little disheartened to see how many students praised their solutions based on the limited understanding and testing, but most seemed to be honest about their strengths and shortcomings.

While planning for this semester, I went back and read my post about the Fall 2019 offering, which regular readers may recall was particularly challenging. Seeing the high quality work of these students, I wish I had them in a semester with a properly community-engaged project! The CCSC Midwest Student Showcase was a fine case study, but it was still primarily about my work as opposed to a community member's. Ah well, if my biggest regret is that I could not get these students working on something even more public, then I think that means it was a great semester.

Wrapping up OpenMind in HCI

My HCI students wrapped up their experience with The OpenMind Platform this week. Back in September, I wrote a bit about how it was going. Since then, they completed modules three through five. I gave them reflective activities after each of the first four modules, tying the module to the week's activities:

OpenMind	Course Topic
Irrational Mind	Goal-Directed Design
Moral Matrix	Cultural Constraints
Intellectual Humility	Errors
Value of Diverse Perspectives	Heuristic Evaluation
Constructive Disagreement

For the final module, rather than tie it into the week's topic of accessibility, I asked them to reflect on the OpenMind experience writ large and relate it to their study of human-computer interaction. This reflection was submitted confidentially to me rather than as a discussion board post, where practically everything else has lived this semester. I wanted to ensure that I was getting as honest feedback as I could expect from them.

The results were overwhelmingly positive. Everyone said it was useful to their studies, and most agreed that its impact reached well beyond the course. The most commonly cited portions of the program were the importance of a growth mindset—particularly in the face of receiving feedback on designs—and the elephant-and-rider metaphor. One student adroitly recognized the connection between OpenMind's themes and the Norman's themes in The Design of Everyday Things. Another pointed out how his background in social science made him skeptical of the program, but that going through, he found nothing questionable nor overreaching about it. One of the more curious but interesting responses was a student who said he wasn't sure that he got anything out of it, yet he enjoyed the exercise of tying a module's ideas to the week's topics.

One student argued that we didn't go far enough with the ideas brought up in OpenMind, that they have deeper connection to HCI than we gave them credit. He pointed out that OpenMind prepares you for meaningful interaction with different others, but that we stayed in conversation with the same classmates all semester. This is quite an observation, and I'm proud of this student for pushing in this direction. I had two themes in my response to him. One is the easy one, that the global pandemic complicates a lot of things, but I could certainly consider rearranging other parts of the course to make room for such activities in the future. The other is that I retain a fear of tokenization. Disability is diverse just like everything else, but the observation leaves me a stake in the ground to consider for future offerings of the course. Also, he may not yet have looked ahead at the just-published final project specification, which does actually require interviewing people as part of the empathy-building process, although not necessarily along lines of disability.

Overall, then, I call the integration of OpenMind an unqualified success. All those who mentioned course integration lauded it and encouraged continuing it. One suggested that it would be even more valuable if given in high school, to counteract the tribalism that arises there. This leans into my continuing consideration of OpenMind in my teaching: would it be valuable to roll this out in my CS222 class next semester? I need to give it serious consideration once I have a chance to start planning that course. (I am still waiting final word on the room and meeting arrangements, which is of crucial importance to planning, of course.)

My main criticism of OpenMind from a teaching point of view is that it's very difficult for me, as an educator, to get in and browse the material. I went through it myself, but memory is imperfect. They provide a dashboard of sorts for teachers who assign the program, but I had no use of this: I wanted to be able to browse or even dump out the contents of each module so that I could review it and mention specific parts in my feedback videos. I understand that they don't want people accessing the material outside of their platform, for research and analytics purposes, but it would still be quite valuable to me. I would rather not go through it again myself each time I am planning a course and trying to remember the details of how a student interacts with it.

Summer Course Re-Revisions 2020: CS445 Human-Computer Interaction

Yes, you read that title right: this is a re-revision of my Human-Computer Interaction course, which I wrote about revising back in early July. Since then, I learned that all my courses would be asynchronous online (after being told that we would almost certainly be able to do them with synchronous online meetings). As I mentioned in my game design revision notes, then, I have had to come back to the HCI course to figure out what to do about it.

Like basically all of my in-person course plans, my HCI course traditionally relies upon in-class activities to help students understand the concepts and build a community together. The original synchronous plans assumed that we would have time to work together, even if distributed physically. Removing that constraint meant that I had to revisit the design to make all the required conversations asynchronous, which means more formal structure around discussion board posts.

After some consternation, I decided to follow a model similar to what I am using in my game design course. I will set up small peer groups who are required to read and comment on each other's work, in the hopes that this not only helps students sharpen their skills but also build some community and get to know each other. I thought about adding a requirement that each group get together online to discuss the readings and work, which is something I've seen in a colleague's online graduate course, but I decided that I wanted to keep this in fairly small, contained steps.

Along those lines, I copied over and lightly edited some of the additional prose I have been adding to course plans back into the HCI one, including several paragraphs about commitment. This may be one of the hardest parts for the students. It has always been the case that I have designed my courses to take nine hours of effort per week, which some students struggle with anyway; without the valuable three contact hours per week of structure, discussion, rapid feedback, and encouragement, it is not clear to me how many of my students will be able to keep their heads above water. Yet, it seems to me to be the only honest option: if we're going to take their money for an education, and federal financial aid law makes these demands on the effort-hour-cost equivalency, we cannot pull back on expectations without a concomitant decrease in tuition and fees. Assuming I can keep my head above water, my plan is to record a short talking-head video each week to release to my students, talking about what excites me about their work and where I see challenges. I hope that will at least help them remember that I'm rooting for them.

I reviewed all my notes and did some additional research as part of my re-redesign, and I added two new elements to the course which I hope will work well with the distributed mode. I added several videos from Scott Klemmer's series of lectures based on his Stanford course. I watched a lot more of the series than I will be assigning. It certainly seems prudent to use his nicely polished videos rather than record inferior ones myself. Incorporating his content made me rethink the ordering of some of the topics I wanted to cover as well.

The other formal element I added was a requirement to complete the five-step OpenMind tutorials. I heard about these through Heterodox Academey I think, and this past week I took some time to actually go through them. There was nothing in it that I had not heard before, but I suspect there was quite a bit that my students would not have. Some of it is relatively simple contemporary psychology, such as the idea that the emotional system is fast and dominant but trainable (the riding-the-elephant metaphor). Other parts mix pragmatism and ethics, such as the importance of intellectual humility. Taken as a whole, I liked the presentation, my only major criticisms being that it starts on the wrong foot and has an imbalanced obsession with Buddhism. Knowing the unprecedented breakdowns of discourse in modern times, and how the Internet seems to be fueling the fire of ideological demonization, it seemed like the HCI course was a good place to inject some of this content. I am eager to hear what my students have to say about it.

We are less than two weeks from the start of classes, and I find myself nervous about the semester. I am grateful that I can do my work from home, but I am concerned about my students' experiences. I still don't really have a plan for how I will do anything like office hours aside from a passing fancy about livestreaming my course-related relevant work. At the same time, I've put in an unprecendented amount of effort this summer to revise my courses for Fall, and honestly, I feel like I could use a vacation, just as the meetings are kicking up for the semester's start.

Summer Course Revisions 2020: CS445 Human-Computer Interaction

After wrapping up my preliminary planning for my Game Programming course before the holiday, I turned my attention to the necessary revisions for my Human-Computer Interaction course. I had a few ideas swimming around my mind, but I decided it would be best to go back to my notes from last time I taught the course, which was Fall 2019. I re-read my end-of-semester blog post, and this reminded me of how terrible that semester was. I had sort of forgotten how painful that course was, but it was good to be reminded.

I decided not to include a community partner for the Fall. I have been disappointed with what my students have been able to provide in this course. While the pandemic makes it easier to work with partners who are distant, it also removes the crucial ability to sit in a circle and have an honest chat. The risks are just too high right now to ask an external partner to donate their time to the students. Of course, if I get good learning outcomes from the students, I would love to have them engage with a community partner in subsequent experiences (such as the game studio course). The topics are just too hard for the students to grasp to ask them to be able to learn them and apply them to a partner's benefit in one semester, given all the other constraints.

The course plan that I just put online lays out the first several weeks of the course in good detail. During these weeks, students will be reading Don Norman's The Design of Everyday Things and completing weekly exercises to help them build their understanding. I have used this book in the past several semesters, and I like that the students can both learn from it and practice learning from expert writings (rather than conventional “textbooks”). The weekly exercises are all in a format that involves individual reading and work from Thursday to Tuesday followed by a period of online peer review from Tuesday to Thursday. In this way, I hope to simulate some of the discussions and ad hoc peer reviews that I would normally do face-to-face in the classroom.

I have been disappointed in the past with students' ability to bring knowledge from DOET into their future work in the semester, and so I added a new culminating assignment that asks them to apply the book's principles to their CS222 projects. In part, this copies the inspiration I use in CS222 when I have them look at their CS121 projects: having them look back on their work, after they have had a chance to step away from it, should give them more objectivity while also reminding them that, despite imperfections, that work was important to their journey. This report is currently designed as a two-week writing assignment with an explicit, graded check-in point halfway through. This will give me and the students' peers an opportunity to give them feedback on which parts of the report are working and which are unclear before the students complete their final submissions.

The course plan as of this writing only goes through the DOET readings and analyses, but my planning notes go several weeks farther. My plan is to get them into Flutter, specifically the Web support that is currently in the beta channel. The thing I like about Flutter is the “code as design” approach, in which the code clearly reflects the application structure. I have seen many students in this course and CS222 get hooked on a drag-and-drop UI builder such as scene builder, leading to two dangerous outcomes: they lose the insights and power about software structures that are gained through programming, and they still design bad-looking interfaces. I'm hopeful that Flutter will give us the right level of abstraction to avoid these two pitfalls.

What I would like to do is have them learn Flutter by creating a simple project, which at this point will likely be a grade calculator. It's simple and familiar, but there is still plenty of room for students to go in different design directions. Then, I would like to use this simple example to introduce readings and techniques of usability evaluation, Gestalt vision, and accessibility. That last point is a new one for the course despite its importance: it has frequently been one of the topics I have cut when doing community-engaged projects because, inevitably with community projects, there are problems that the teams need to put their heads down on and fix.

I would like to wrap up the semester by having them make a larger project that they can use to bring together all the analytical tools and techniques that they have learned. I am still a little uncertain about the context for this project. I may just open it up to them, as I do in CS222, but this runs the risk of students falling into a quagmire of code and not appropriately deploying the other cognitive tools we are learning. I may just give them a canned project for that reason. Two that have come up recently in my own practice are a tool to judge student showcase entries, as in the CCSC Midwest Conference, or a tool to help students manage checklist-based specifications grading, as in my Game Programming course plan.

I think I will return to the CS445 course plan in the next few days and add details for those other assignments. Looking at it now, I realize that I forgot to add the due dates to the Web site; since I cannot yet even access the Canvas page for the course, I may as well trick out my lit-html templates to include the deadlines online. In any case, I had a productive morning of planning, and it seemed like a good time to write this blog post in the gap before lunch.

As always, thanks for reading, and feel free to share any feedback or questions you have about the plans.

Reflecting on the Fall 2019 CS445/545 HCI Course

I have to start by saying that this was one of the strangest classes I have ever taught. We were continuing my series of collaborations with the David Owsley Museum of Art, and as I wrote about in July, I had a great meeting with them to set up some tighter constraints around how we work with the students. There were only nine students in the class, which I thought would be really exciting: small team, a couple of graduate students, working with a partner on exploratory software to enhance the visitor experience. It seems like the recipe for an excellent learning experience, but in truth, this was one of the most frustrating classes I have ever taught.

I have never had a class where I have to "pull" so hard to get them to do, well, anything. On the very first day of class, there were about ten people in a room that seats over thirty. When I walked in, they were all sitting apart from each other, staring into their phones. I commented on how quiet it was, and I encouraged them to move in toward the front. Nobody moved. We repeated this ritual basically every class meeting. It became something of a joke after a while—a sad, sad joke. On a few occasions, I forced them to rearrange the furniture so that we could sit in a circle for discussion, and these meetings were always better, but they didn't seem to have any impact on the de facto standards. One time, I came into class, and two or three students were talking to each other. I heaped praise on them in hopes of some positive reinforcement. I got a few smiles, but again, no real change.

When we moved into working as one big team, I let them follow their own path for the first two week sprint. After a structured reflection, I told them that I would be scaffolding their improvement by providing a methodology. I used one based on my immersive learning teams, which have been roughly the same size. One of the rules of the methodology is to work in pairs whenever possible. They ostensibly read the methodology and we got to work... everybody working silently at their own laptops. I paused for a minute or two, then I interrupted, pointing out that they were all in violation of the methodology, and that they should pair up. Some of them still did not. At that point, I feel like I just have to throw up my hands.

With only nine people, attendance irregularities are easy to notice, and many people missed many class meetings. It's not like it didn't hurt their grade either: they had work to complete and then discuss almost every class meeting. I think it's fair to say that it's disheartening for everybody in the room to look around and see that only five or six out of nine people are there. In the latter part of the semester, we worked as one consultancy, and there was work for everyone to do; even here, people missed critical planning and reflection meetings.

The point of this writing is not just to complain, though. We did have some real stand-out meetings. In one of them, we talked honestly about their past team experiences. They acknowledged that none of them had ever really been on a non-dysfunctional student team before, and also that they didn't really know what a successful team looks like. This is invaluable to me as an educator, because it makes me realize that it's not just enough to give them guidance: I think we have to work harder to show them examples of successful teamwork to model. I am still not sure how to do that, except maybe by filming one of my high-functioning immersive learning teams. Another important part of this discussion was their acknowledgement that in the prerequisite course (CS222), they learned to fear feature branches and pull requests and, generally, GitHub. That is, they saw these tools as impediments to their success rather than what they are: critical parts of a healthy and productive work environment. This again points to some specific actions, to make sure that the prerequisite course is not accidentally teaching counter to its purposes. Conveniently, I've been assigned to teach CS222 in the Spring, so I will be able to pilot a few interventions here.

One of the frustrating outcomes of this semester is that I am not really sure whether the students learned anything or not. We studied some of my favorite theories of HCI during the semester, always embedded within the context of our collaboration with the art museum. My plan was, near the end, to return to those theories and frame our work within them. We ended up having to declare a failed sprint a few weeks from the end of the semester in order to produce a barely-testable digital prototype. This ate up the time that I was hoping to use to close the loops. Looking at the work that the students did on the project at the end of the semester, while technically competent, I didn't see any consideration for any of the theories we discussed earlier in the semester, like Don Norman's action cycle or Gestalt vision principles. Instead, I saw what it looked like they would have done if they had never taken the course. This is disheartening.

In our final meeting, our museum partner was kind to point out that the prototype these students created was the highest quality of any that students made in the previous semesters. I generally agree, and this is in large part because we had one group and one consistent series of conversations. From a teaching point of view, it's a completely different thing to have one team of students to engage with than it is to say, "Get into your groups and talk about this." Also, because we all worked together, I could give more direct guidance on some of the technical issues of the implementation as well, which prevented them from getting caught in amateur's dead-ends. For example, there's always someone who thinks copying and pasting code will do no harm, and then you end up with an unmaintainable mess that collapses under its own weight; I was able to work with students to refactor such solutions, teaching both process and techniques for refactoring along the way.

Although our partner was positive and right to be so, I remain concerned at how it seemed the students didn't really think about what they were saying or doing—they were not critical. For example, they liked to mention that they used journey maps, but they didn't do this well. I graded all the journey maps and provided feedback about the parts that were good and bad, but in their presentation and final essays, they wrote about their journey maps as strictly virtuous. I mentioned Falk's theories of museum visitor motivation in class, and the students latched on to parts of this; in the presentation, though, they made it sound like they had actually studied and applied these theories. In fact, they had done the equivalent of a standard contemporary undergraduate practice: hear of something, Google it, put some buzzwords in, and call it satisfactory. Truly, their presentation of their knowledge of Falk's model bordered on lying to the client, and I just wasn't prepared for how to react.

If you got this far into this essay, you can understand why this class that seemed like it should be so good would actually be so frustrating. I'm still not sure about the root cause, but I'll share the best thought I have. I don't know why the department administration thought this was a good idea, but we've offered HCI as an upper-level elective for something like five straight semesters—including summers—when it used to be a biannual course. The number of students in my classes has gone down each time, and with only nine completing this time, I have to think that a majority of this small group didn't really want to be there. I don't think they had any motivation to either take HCI or to take a class with me. (Not to toot my own horn, but there are some students who just want to study with me, regardless of the course.) In the absence of motivation, the course is perceived like the methodology I wrote up for them: a hurdle to be cleared rather than an idea to explore. I suspect I could have just given them readings and assignments, and they still would have skipped a bunch of classes and earned their C and B grades, and I would have been able to sleep at night. But you know, that's not how I roll.

I had one day where I was feeling particularly frustrated as well as low on physical and mental energy. I don't remember where we were on the project, but I asked the class to give an update on their progress. Nobody in attendance had made any. It was one of those times when I had to seriously think about just leaving the room, just walking away and letting them do whatever it was they thought they should be doing instead of contributing to the class. I even said out loud, "I have done more work on this project than you have collectively," which is probably not exactly true with a class of nine people, but I think it was dangerously close. I took a minute or two to collect my thoughts and, with some grace, turned it into a discussion of how to move forward. After this, one of the students—one with whom I had some struggles earlier in the semester but with whom we grew into mutual respect—started occasionally thanking me for my work. He would say something like, "Thanks for taking the time to put this collaboration together," and he really meant it. This may seem like a small thing, but it wasn't. It's possible that some of the students still see faculty generally as some kind of automata. I think this student saw that I cared, and because I cared, I hurt. We talk about helping students build empathy, and here's a case where it actually worked, not in some abstract social justice sense, but in a very concrete, local-community sense.

This post is a bit long, but I have had a strong desire to try to capture these stories. In truth, I'm not even entirely sure why, because in conclusion, when I consider what I would do differently next time, I honestly don't know. One thing I would do is absolutely put my foot down on the "nine people spread around a classroom" on day one, though. That was just ridiculous.

Thanks for reading. Feel free to share your thoughts and suggestions. The next one will be more cheerful, I promise.

Reflecting on the Spring 2019 CS445 Human-Computer Interaction Class

Regular readers may recall that I was given the Spring 2019 HCI course to teach on rather short notice, so I only made a few structural changes between it and the Fall 2018 section. The most relevant to this post are the increased attention to software architecture and the switch to specifications grading. I also gave the teams nominally more time for their final project, but not enough that it was noticeable from my point of view. We retained our collaboration with the David Owsley Museum of Art (DOMA) and the overall theme that student teams would identify and address real problems they face.. Yesterday, I shared my sending-forth message to the students, and today, I would like to share my reflection on the semester's experience. Feel free to reference the course page, which provides the policies, procedures, assignments, and assessments for the semester.

I used a similar approach to specifications grading as I did in the Fall 2018 Game Programming course, in which there were discrete criteria for each level of grade. I added a separate category of criteria for the project reports, which were designed to provide the process documentation that corresponded to the technical artifacts produced. As before, students had to submit a self-assessment along with their source code and report, the self-assessment's consisting of a checklist of criteria that were met. Unlike the game programming class, where there was rarely disagreement between the students and I about whether a criterion was met, there was a lot of friction this semester. This was especially the case on the final project's two iterations. As I wrote to several students in my formal feedback, I have serious doubts that many of the teams honestly conducted a self-evaluation at all. Consider, for example, one of the most missed criteria asked students to explain how their projects manifested particular design principles from Don Norman's The Design of Everyday Things. Teams submitted a list of examples, with no explanations of them. It seems to me that if a team sat together and worked through the checklist, as I expected them to, someone would have said, "Have we explained this?" I don't think they had anything approximating such a discussion: I think they surveyed the checklist, said "Good enough," and marked the box. That is, I think they defaulted to the "hope for points" model rather than the "ensure success through unambiguous choices" model. Of course, the idea of self-assessment is not to save me grading time but to foster reflective practice and remove ambiguity. When students do it honestly, it does save me grading time, and when it is done dishonestly, I suspect that students might learn something about the importance of self-reflection. I need to think about what I might change in future uses of specifications grading to get around this.

An honest student approached me near the end of the semester, in part to share what he claimed was a voice of many students who were frustrated with the specifications grading system. I explained to him that the goal of the system is to remove ambiguity, both for students and for assessors. I honestly never got a good explanation of what precisely he or other students did not like about it, except that there were standards at all. I think the status quo is that students believe they start a project with full credit, and then I take away points for mistakes. One of the things I like about specifications grading is that it follows my contrary philosophy, which is that students start with nothing and must earn their credit. I think it is this idea, not specifications grading in particular, that students are upset about, because it holds the accountable to demonstrating understanding to earn credit. The fact that I get complaints about grading regardless of the scheme I use is probably testament to students' pushing back against having expectations rather than the particulars of the system. However, foreshadowing some of what is to come below, part of why a subset of students complains is likely that I actually draw upon knowledge they should have from prerequisite courses—knowledge that they may not actually have.

In the first half of the semester, I used a running demo project ("archdemo") to demonstrate some ideas of how to separate the layers of a user-facing software system. In the previous semester, I had done something similar, but using a context separate from our class collaboration with DOMA. Many students that semester did badly with the "warm-up" project, and so in order to help with on-boarding and consistency, archdemo showed a sample use of the DOMA data via the ContentDM database. The resulting application was called "Naïve Search," named thus because it didn't really solve any reasonable search problem: it just showed how to separate the layers of a system. While this worked in the short term, I think it also caused problems as students perceived more value in the example than it was meant to have. It was never intended as a template, but only an example of very specific course concepts.

One of the changes I made from Fall 2018 was that I required final project teams to use a subset copy of the ContentDM database in their projects. My intention here was that each team would have to demonstrate that they could separate the layers of a user-facing software system, regardless of what creative direction they wanted to take the project. The result, however, was that nearly every project looked a lot like archdemo with an added bell or whistle. Last semester, we had a broad range of concepts on a plethora of platforms; this semester, it was dullsville, as the teams just added some minor idea to archdemo. One team even consistently referred to their solution as "an improvement over Naïve Search," despite my repeatedly telling them in their formal feedback that this was not even close to our goal. I have no doubt that our partners at DOMA were uninspired by this semester's projects, although we have not had our wrap-up meeting yet. I would be remiss not to mention one exception, which was a clever interactive map that tied into the database in interesting ways despite the tight project timeline; those guys really nailed it, so if you're on that team and reading this, kudos to you.

Throughout the semester, we returned to five principles of design brought up in Don Norman's The Design of Everyday Things: affordances, signifiers, mapping, feedback, and conceptual models. Despite this being a theme of the class, there is scant evidence that students understand or applied these principles. Instead, my professor's eye tells me that they designed whatever they wanted, and then they tried to shoehorn those designs into these principles, or to justify their work after the fact. Although they had several assignments and much formative feedback about these principles, students continued to show misunderstandings through the final exam.

What was missing? I believe a big part of it was that they didn't follow my advice. This is exemplified with one key example: taking notes. The course plan says explicitly that students should always have their notebooks available for taking notes and jotting questions, and furthermore, that they should not have their distraction machines (laptops and phones) in their way during class discussions. Taking a friend's advice, I even made a first-day quiz in which students had to answer questions about this aspect of the class. Yet, very quickly (and for some, instantaneously), their old habits took over, and I would stand in front of class looking at the backs of open laptops rather than faces. Almost no students took any notes on any of our discussions, and is if to drive the nail of the coffin, some of them only got out their pens when I wrote something on the board. Even if they had a glimmer of understanding about affordances and signifiers during class discussion, there is no way that they held on to this fifteen minutes after class unless they actually expended the effort to do so.

I wrote on Facebook the other day about how I was feeling conflicting emotions about this class. On one hand, I am unsympathetic that they did not learn the material because they chose not to follow my advice on how to do so. The advice is not complicated: it primarily involves reading and taking notes. However, at the same time, I pity the students, because I think a large majority of them—if not all of them—know neither how to read for understanding nor how to take notes while reading or discussing. To me it begs the question, "Where does the buck stop?" If I get undergraduates in my upper-division elective Computer Science courses who lack these skills, is it my responsibility to teach them or just to assess what is in the master syllabus?

There is a related puzzle, which was foreshadowed in my sending-forth message to the class. An uncomfortably large proportion of the class showed very little proficiency in fundamental programming skills. When I brought this up in honest, private conversation with trusted undergraduates, they showed no surprise: they said that it was fairly easy, and common, for students to "cheat" on assignments. This manifests in two ways: either copying the work of a peer and submitting it as their own, or stringing together bits and pieces of code found online. Neither approach forces the learner to confront the useful struggles required to build firm understanding. It reminds me of the advice from Make it Stick that I wrote about last December, and that in its absence, students really don't know how or what it means to learn.

Going a little further, I witnessed a curious phenomenon several times during a guest presentation by a CS alumnus and successful professional. The speaker is currently in a position with a lot of creative flexibility, and he has his own team of programmers to implement parts of what he designs. However, many students seemed to misunderstand his story, thinking that this meant one could just "have ideas" and tell others to program them. They missed the part where he worked on rather tedious programming tasks for ten years to prove his capability, vision, and leadership. Instead, they rejoiced, saying things like, "I cannot program, but I want to tell programmers what to do, so now I see that there's a job for me!" This sentiment was shared primarily among CS minors despite their having taken at least three programming courses in the prerequisite chain to this course.

These students don't seem to see a connection between the HCI goals of the class and the fundamental skills of software development. It appeared to me that these students were not being tripped up by the accidental complexity of software development (such as the placement of braces or the quirks of a UI framework) but rather by its essential characteristics, which include precision and sequential reasoning. How I frame HCI as a Computer Scientist is essentially that it is user-centered precision and sequential reasoning. What happened on the students' final project teams seemed to be that those who had programming skill were relegated to doing persistence- and model-layer data manipulation tasks—required tasks for the program to work at all—while those who could not program worked on the UI. The result is that the UIs were badly conceived and executed, because those working on them couldn't conceive of the problem as requiring precision and sequential reasoning. Part of my evidence for this manifestation is the difference between teams' paper prototypes and their final products. Every team decided upon a paper prototype that was developed from a user-centered design process, but practically no team's final product looks at all like their prototype. Instead, their products looked like the archdemo sample, but with a few more widgets added via SceneBuilder. One could say they did what they could rather than what they wanted, or more pointedly, they decided to fail conservatively rather than succeed differently—which was exactly one of the human failure modes we discussed in class.

A student confided in me that, when he signed up for the course, he expected he would learn how to design a good user interface. By this, he meant that there would be some thing I could teach him that would suddenly make him good at it—a silver bullet. He pointed out that some students seemed to think that it was all in the tools: if they learned the tools, they would be able to make good UIs. I am grateful that he took the time to share this with me. I asked him if, after studying this topic for fifteen weeks, he understood why I could not meet his desires, why I could not dump ideas into his head that would suddenly make him good at UI design. He indicated that he did understand it, but he also sounded disappointed.

As I wrote about yesterday, I had forgotten about the emotionally powerful reflection session I had with my students at the end of the Fall 2018 HCI course. I didn't schedule for it this semester, and so it didn't happen. I think it would have helped all the students to frame their difficulties and challenges within their authentic context: yes they struggled, but they did so because what we are doing is legitimately hard.

The puzzles I face in considering how to change the course for Fall 2019 are significant. I expect this week to be able to meet with representatives from DOMA to talk about their take on the experience. Our primary contact is their Director of Education, Tania Said, and she has intimated that she would like to see the class work together to produce something with more staying power rather than a series of prototypes. This makes me a bit nervous, given my previous experiences having entire classes taking on a project, but there may be a possibility to set it up like competing consultancies rather than trying to do a whole-class team. One of the reasons I wrote this up now is that it has helped me serialize and articulate some of my thoughts in preparation for meeting with her. I hope that conversation will help me turn some of these reflections into actionable course plans for Fall. As always, I expect I will be able to share my summer planning activity in a blog post in the coming months. Until then, I think it may be time to spin up my summer project.

Two missing specifications in HCI

I finished grading my students' final projects for the Spring 2019 HCI class (CS445, used to be CS345). Before the start of the semester, I wrote about how I would try specifications grading in the course. After the afternoon of grading, I realize that I missed two important specifications. I will share them here so that I have a better chance of remembering when planning Fall's class, since I've been assigned to teach the course again.

I should have had a specification requiring all non-trivial processing to be done off of the event thread. This is, of course, a requisite for any kind of multi-threaded UI programming. I specifically chose a data source that would require handling slow load times and long processing times so that my students could practice this technique. I developed a sample project in the first half of the semester based around this common practice, and I explained to them why it was important. However, I neglected to have a specification about it. Three hours before the final project was due, I had a student ask for some last-minute troubleshooting. He said that he added a spinner while some images loaded, but it wasn't showing up. Of course, it wasn't showing up because he was loading the image on the event thread. I showed him (again) the example from earlier in the semester and explained (again) why this pattern was necessary. From their final technical presentations, it was clear that he was the only person in the class of roughly twenty students who understood this crucial point. I believe this is an instance of the old standard motto: if it's important, make it worth points. I simply missed it in my specifications.

The other specification deals with acceptance testing. There are two relevant specifications in the evaluation plan, one at B-level and one at A-level. Specification B.5.R says that the final report " describes the methods by which the solution was evaluated," and A.2.R says that "The documented solution evaluation includes both quantitative and qualitative elements that explicitly align with this semester's readings." The B-level specification is designed to be broad: you can earn a B on the project by doing any kind of acceptance testing. The A-level specification is designed to be more focused: do a mixed methods evaluation based on a theory we studied this semester. None of the five teams explicitly aligned their evaluations with the semester's readings. This didn't stop two of the groups from marking that specification as complete in their respective checklists, casting serious doubt on the implicit claim that they had conducted the required self-assessment for which the checklist is a result. (Perhaps, then, I need to add more rigor to the self-assessment itself, requiring them to link their claim to the artifacts.)

The problem with the acceptance testing actually goes much deeper than dishonest claims of completion. Among those who conducted any kind of acceptance testing, there was no evidence of their having learned anything from the assigned readings and exercises relating to Steve Krug's and Jakob Nielson's theories. Instead, they followed ad hoc approaches that were poorly designed and yield unreliable results. They did actually use quantitative and qualitative approaches, in keeping with A.2.R, but they did not do these well. For example, many groups asked questions like "What did you think of the application?" and then reported "3/6 users say they liked it." I pointed out in my feedback that 50% of users claiming they liked it is different from 50% of users liking it. More importantly, "liking" the application was not one of our design goals: we were designing systems to be used. Yet, only one of the groups conducted a task-based evaluation, where the user was asked to accomplish a goal using their system. Task-based evaluation is what I expected, and task-based evaluation is what I wanted. However, I wanted the students to realize that this kind of evaluation was the right choice, so I left the specification open to other options. The other options were demonstrably worse. Hence, in the future, and particularly in this introductory HCI course, I should just require them to follow the best practice rather than give them the choice to shoot themselves in the proverbial feet.

I have to wonder if the students would have spontaneously met these criteria if they had taken notes during our discussions

Planning CS445 Human-Computer Interaction for Spring 2019

Around the Christmas celebrations, I have spent many hours the past two weeks planning my Human-Computer Interaction course for Spring 2019. I wrote my reflection about the Fall semester back on December 8, and I just put the finishing touches on the Spring section's course plan before lunch today. Now, I would like to write up a few notes about some of the interesting changes I have in place.

First, though, a funny caveat. Since I originally designed the HCI class for CS majors, it has been CS345. Due to administrative busybodies, the course will now be numbered CS445 instead. Which label should I use for my blog posts? I think I'll switch over to "cs445" now, and I'll have to remember to use both codes when I'm searching for my old notes.

Canvas Grading Caveat
Prompted in part by my frustrating experience at the end of the Fall 2018 Game Design class, I have a more explicit statement on the course plan telling students to ignore Canvas' computed grade reports. I would always say this in class, but I did not have it explicitly in the course plan before. Also, I found out that I could mark assignments as not contributing to the final grade, in which case students will be able to see their assignment grade but not a false report of their "current" final grade, so I need to remember to mark all the assignments that way. Also also, please take a moment to consider the epistemological tragedy that is the concept, "current final grade."

Writing
I was surprised in the Fall when one of my more talented HCI teams brought up their project report, highlighted a place where I pointed out grammatical and spelling errors, and asked if they had lost points because of it. There are two things wrong with this question. The first is that it assumes the students had some points to lose in the first place, which is simply not true. I don't take away points from anyone; instead, I award points for demonstrating competence. You cannot take away something that someone doesn't have. The second, more pragmatic problem is that the students also had significant conceptual and descriptive problems in their report, but they seemed more concerned about the spelling and grammatical errors.

Last semester, I included a link to the public domain 1920 version of Strunk's Elements of Style, along with advice to read it. This time, I've made my expectations more explicit. On the evaluation page of the course plan, I have explained briefly the importance of writing and the fact that Elements of Style will provides my expected standards. I also explain there that I expect to give feedback on both conceptual problems and spelling or grammar problems, along with a primer about how to interpret that feedback. I thought about making an assignment around Elements of Style, but I decided against it, partially because I did not want to shift my early-semester plans ahead by a day. My professional opinion is that the book should be remedial to anyone who has a high school diploma, but I am also a realist about the variable quality of writing instruction these students may have received.

Software Architecture
It was a little disappointing to see so few teams really engaging with principles of quality software construction last semester. I have written about this before, and the students are aware that the culture of other classes is one that values only software output rather than software quality. I have carved out some design-related topics from the HCI class to make more time to work through examples of refactoring toward better architectures. I'm still working on the exact nature of these assignments, but I have a few notes to draw from. The schedule I have online right now actually goes right up to the point where I want to switch gears from design theory to software architecture practice.

Specifications Grading
After a positive experience in last semester's game programming class, I have converted my HCI class project grading scheme to specifications grading. I laid out my expectations for each level of poor (D), average (C), good (B), and excellent (A) grades. This was an interesting exercise for me, especially around the source code quality issues. Last semester, students could earn credit for following various rules of Clean Code, and a mixed grade simply meant that they got some and not others. Now, I have put all of these rules at the B level, to reflect the fact that "good" software is expected to follow such standards. For the A level, I've included demonstrated mastery of the software architectural issues mentioned above.

I had some fun with the Polymer-powered course site as well. My new custom element for presenting specification grading criteria uses lit-html to concisely express how they are presented. It took a bit for me to wrap my head around lit-html, but I think I have a good sense of it now. The other fun new feature I added was the ability to download the specifications as Markdown. The specifications are internally represented in a Javascript object, and that object is transformed into the Web view. Of course, with this model-view separation, it's reasonable to provide other views as well, such as Markdown. I used this StackOverflow answer to write some functions that convert the JSON object to downloadable Markdown. I hope that this makes it easy for students to write their self-evaluations in Markdown. I did not use checkboxes on the Web view, as I did for game programming last semester, because they don't copy and paste well. I hope that having the Markdown version available removes the need for students to manually copy over each criterion into their self-evaluation.

More Time on the Final Project
In addition to switching the evaluation scheme, I am giving more time to the final project. I decided to keep the short project as a warm-up, since it also provides a safe-fail environment as students pull together ideas such as personas, journey maps, user stories, and software quality into a coherent process. Some of my dates for the final project aren't quite sorted out yet, as I'm debating whether to take a purely agile approach or a milestone-driven one. The advantage of the latter is that I can specify exactly which design artifacts I want to see at each stage, and the level of fidelity I expect. However, I expect that I will follow the more iterative and incremental approach, but I've put off the final decision until the semester gets underway and I can get to know these students a bit.


We are continuing our relationship with the David Owsley Museum of Art, who have been a great partner. I look forward to working with them and seeing what my students can develop during the semester.