Wednesday, May 11, 2016

Parting thoughts about Studio 368

As I finish up my obligations from the Spring semester, I want to take some time to reflect on my Spring Game Studio course--the group that would come to call themselves Studio 368 and produce the original educational game, Traveler's Notebook: Monster Tales. I have written some about this during the last few months, so here's a collection of relevant links:

There are a few topics that I have not written about, so I will take the opportunity here to share them.
The sign on our door
By the way, that little monster on the top of the sign there is Debbie--full name, Little Debbie Snackcakes. One of our team members sketched her onto our "Keep Calm and Find Monsters" poster (which sadly I don't seem to have), and she became our mascot.


This semester had the best mix of majors I have ever had in a studio course, better even than the mix at the VBC. The team included: four Computer Science majors; one Telecommunications major with an emphasis in audio production, who also had a CS minor; one Philosophy major with a CS minor; one Art major specializing in animation; and three English majors specializing in Creative Writing. There were some tasks that people fell into pretty naturally: the CS majors got right into the programming, the artist into illustration and visual design, the English majors into writing. Our audio was all designed and integrated by our Telecommunications major, and one of the English majors also contributed significantly to the visual design, drawing upon her minor in Professional Writing (which includes a significant media design element).

Those are some of the common discipline-oriented production tasks involved in game design and development, but there are also important interdisciplinary aspects. Here, the team really came together and took initiative. Some of the students joined me in writing a scholarly article about our work, which is currently under review. Several contributed to the visual design of the game over multiple iterations. When it came time to conduct playtesting sessions, our Philosophy major showed her humanistic and analytical prowess in designing, executing, digesting, and disseminating our evaluation protocol. About halfway through the semester, we became aware of fundamental design problems, and this resulted in a beautiful, rapidly-developed paper prototype for a new design that was assembled by a CS major and our Philosophy major. This particular artifact served as an important intermediary work product, serving to align the team's disparate visions about where the project was heading.

Working on the paper prototype


While I have been using GitHub and Travis for projects since last summer, this was the first time I worked with more than one or two other people using this technology. It was a little spotty at first, in part because so many of us were a little green, but once we agreed on how to coordinate feature branching, we hit stride. I think we still could have been a little more formal about how to handle pull requests and who is responsible for doing them. I wonder if this is a place where I should be more central in the process, teaching by example how to conduct a review of a pull request, and slowly doling out the responsibility to others on the team as they learn. I am uncertain, however, as to whether this would slow the team beneficially or artificially--if they need to make those mistakes with git in order to understand how to proceed.

A particularly tech-savvy team member knew a lot more about GitHub than I did, and he pointed out that we could set up an organization page as well as a project page. Given that our project is open source, this gives us free hosting for a web presence and the game itself, which is pretty slick. Our custom domain name simply redirects to our page, with Google Domains' subdomain forwarding taking you to the info page or directly to the game. Thanks, GitHub!

We adopted Slack at the recommendation of one of our technically-minded team members, and this proved to be a great boon to the team. Over the past several years, I have struggled to find a team communication mechanism that fits the right balance of convenience for a maximal number of users. Usually, I have fallen back on Google Groups, which is essentially a mailing list; the problem is that kids these days, they don't conscientiously check their email. Slack hit us right where we needed it, with multiple entrypoints and notification settings for different users. If I were to use it again--which I expect I will--I will do a little more research on best practices for the use of channels. We used one #development channel, where programmers talked to each other and we had Travis sending continuous integration messages. This was much too noisy. The team had a lot of fun with Slack, which also meant that sometimes it was hard to separate the signal from the noise, so something like a new pull  request might get lost. This is probably something that regular users have bot-based solutions for, and one of the team members even started writing such a bot during finals week; next time, I need to come in a little better prepared.

All together, we built prototypes in three different programming platforms. In our first push for a playable prototype, the team decided to use Java with JavaFX because it was familiar to them, even knowing that it would not likely be our target platform. While they worked on this, I worked on proof-of-concept systems in both PlayN and Polymer. After that first iteration was passed, we looked at all three platforms and centralized on PlayN due to the ability to write in Java and use familiar tooling while also being able to publish to HTML5 via the GWT compiler. 

Team T-Shirts were an important part of our technology stack


We never quite hit a good cross-functional integration rhythm this semester as has happened in past semesters. Art integration was an awkward manual process, with files being uploaded to Google Drive, downloaded by a programmer, probably renamed to match conventions, and then added to the project. By contrast, in Bone Wars, we had the artists integrated directly into our repository, so they could freely update files themselves. It's worth noting, though, that we had two artists that semester and for twice the credit hours, so we had more breathing room: our one art major this semester really went beyond the call of duty in producing the amount of original work that he did.

We once again used a design log, and this came up frequently in students' reflections about what facilitated team work. I think we could have been even more deliberate in using it to explore design decisions: a few times, some ad hoc contribution would end up in the code and then become seen as normal, even though it broke the game's theme. It is also possible that some team members never did reference the design log, as very late in the game, there were still some team members using the game's nomenclature incorrectly, which invariably leads to confusion. Of course, this was a highly-experimental course, and we came in with inspiration but almost no design at all. It is really amazing that this team was able to accomplish what they did in just one semester and three credit hours. It helped that I had an extra course release for research this semester, as it meant I also had one less thing to worry about and could focus on helping Studio 368 succeed.

Story integration was also awkward. I initiated the design of a JSON-based story specification format, so that a single JSON file could describe the world. (This actually came out of my Polymer-based prototype.) One of the team members undertook the challenge of creating a graphical level editor in JavaFX that could read and write this format, which--in theory--would make it easy for those focused on writing to integrate with the game. Unfortunately, that never really happened: the writers never adopted the editor as a primary tool, instead collaborating on Google Docs. This led to some conflicts, where their creative writing was not always aligned with the formal needs of the project. It was the first time since my 2012 VBC seminar that we had to write our own editor, but this one was not integrated well enough into the rest of the stack to maximize its utility. In all of our iterations, I don't think the story team ever saw and debugged their work in the game until after the iteration was formally over. Looking forward, if I were to continue this project or undertake something similar, I would want to more formally include the editor in sprint planning and ensure that it supports both rapid integration to the game. 

Inspirational messages from Debbie


This project was supported by an Immersive Learning Micro-Grant from the College of Sciences and Humanities, to whom I am extremely grateful. We would not be able to make it a real studio without the generous support of my home department, Computer Science. One of the reasons I had such a great team was because of trusted friends on campus who sent me their best and brightest, and I owe them gratitude as well. Special thanks to Jennifer Grouling and Joyce Huff from the English department at Ball State, both of whom volunteered their time to share their scholarship with my team; Cathy Hamaker at The Children's Museum, for sharing her stories and time with the team; and to Rebecca Parker and Stuart Cotton at Muncie Public Library's Connection Corner, for being willing to work with us on this venture.

And of course, thanks to those students who took a risk in signing up for the game production studio. You did great work, and I am proud of you.

It's a long story.

Thursday, May 5, 2016

What we learned in CS222, Spring 2016 edition

It's that time of year again, where I like to share a short retrospective on CS222. You may recall that I had thought about making some significant revisions to the course, particularly with respect to the trio of assignment, project, and achievement. In particular, I thought about incorporating student portfolios, but as I read more about this approach, I saw strong encouragements that students should be able to pick what goes into their portfolio. With the short, holiday-packed winter break, I didn't have time to really dig into the research on portfolio-based grading and integrate it into CS222. Summer is a more relaxed time for making this kind of structural change.

As always, I have several notes sketched out, in various places, about what went well and what didn't go well. I keep coming back to the idea of a more structured approach to project planning, particularly with respect to user stories. So many students do this badly, even after repeated feedback, and then have difficulty with their project for lack of concrete requirements. I really don't want to muddy the iterative approach by having additional items due in the middle of it, so maybe I need to structure the days leading up to the first iteration more carefully... but that might mean dropping or shortening the two-week project, which tends to be an important learning experience and turning point for students. Clearly, there are a lot of unknowns, and I need to spend some time thinking about it.

Without further ado, here's the student-generated list of what they learned in CS222 this semester, with the number of votes:
Clean code*24
Descriptive naming2
It's OK to start over4
CRC cards2
User stories8
Mode-view separation5
Team development*12
Pair programming 4
implementing libraries2
object vs data structure
code for the readers2
commenting best practices1
definition of OO9
functional programming1
stack overflow is KING
test coverage
test coverage exploitation
builder desing pattern1
observer design pattern
Drefus model of skill acquisition
game & mobile development
user-centered design2
dependencies on List
Scala (presentation)
quantum computing
AI efficacy
Acceptance testing2
critical components
intellectual property5
8am is troublesome ("sucks")3
classroom is awkward
walk on the left
BNL- not as good as in the past
IntelliJ is more featured than Eclipse
IntelliJ is better than Elipse2
Gradle is fragile
Github is complex2
Github is powerful
github has a steep learning curve
IntelliJ has a steep learning curve3
Finishing / shipping a project2
Time management3
Slack is great
Don't generate code you couldn't write1
Team communication is key5
Horizontal & vertical formatting
Egyptian vs K&R braces
DocumentBuilder (XML)
Package naming conventions
Unit testing4
Constant naming
Magic numbers
Essay reflections
Idea -> reality2
Don't procrastinate2
Critical thinking5
NPR is phasing out RSS
Making an app
Design thinking framework
Jar files
Managing scope1
Burndown chart
Technical debt
Timers are a pain
Compiler warnings1
Don't update mid-iteration
Network interfacing
UI design3
HBox & VBox
Scene Builder
Android Studio
Pizza makes apps
Adjusting sleep schedule
Open Source Licensing1
Dr. G dislikes textbook authors
Donut sticks -> no real wheat
RMS is eccentric
There is no silver bullet1
Studio 368 only has 4 CS majors
Caffeine is good3
Dr. G likes jazz
Ragtime is racist
They won't kick you out of RB1091
How to take constructive criticism1
How to give constructive criticism
Better to have compiler warnings than nothing at all
Working with unfamiliar APIs
Don't program alone3
Inner classes
Take breaks when coding
Review code in short segments
Commit frequently
Don't push broken code
Working in iterations
Sometimes you have to break code to fix it

The three marked with asterisks were the top three on which students could write their final reflective essays.

There are no real surprises here. It was an 8AM class, so I would have been surprised if this had not come up on the list. There are usually a few personal things about me on there; I know someone said "Dr. G. likes tea," which I did not actually write up. The Ragtime reference is from a discussion we had in class the morning after I had seen the musical for the first time, and---through what I remember to be a far-reaching discussion---brought the racist themes of both Ragtime-the-art and Ragtime-the-production into comparison with diversity in software development.

Git, GitHub, and IntelliJ came up frequently, although not with very strong weight. These are the tools that I myself only started using last summer, and sometimes they still trip me up. As I have mentioned before, I do think the switch to GitHub is worth the effort since it's such a popular tool: if we're going to deal with a real version control system, we may as well use one that also has cultural purchase and can land students internships and jobs.

I get more and more students taking on Android projects. Whether this is related to the similarity between Android Studio and IntelliJ, I cannot say, but it does complicate evaluation. I really don't know gradle besides the very basics, and I have no reason to learn it with my current projects, where Maven is working fine for me. I suppose it's OK, though, for students to see that I don't know everything, and I have to also poke around on Google and Stack Overflow when things go wrong. With all the different versions of IDEs and Android libraries, it tends to take longer to clone and build a project than it does to evaluate it.

That's it for CS222 this semester. I have around thirty essays to grade before I can really call it done, but those are pretty easy: they are student reflections, a final stab at tying their real, authentic experiences to the big essential questions of the course.

Someday, I should do something with all this "what we learned" data...

Wednesday, May 4, 2016

A Project Retrospective and a Final Exam for the 2016 Spring Studio

I announced the release of Traveler's Notebook: Monster Tales a few days ago. This was the semester project of Studio 368, a multidisciplinary undergraduate team that I mentored during the Spring semester. After each of our four iterations, we held iteration retrospective meetings, during which they contributed answers to these four questions:

  • What did we do well that if we don't write down, we might forget?
  • What did you learn?
  • What should we do differently?
  • What still puzzles us?

I am fond of this format, and I use it regularly with my student teams. (In fact, I've written an empirical evaluation of this approach, should you care to read it.)

Studio 368 finished production on April 27, and on our final meeting of May 2, I led the team in a project retrospective. First, we collaborated on drawing a timeline of significant events from the semester, including production schedules, guest visits, important decisions, external playtesting, and dissemination opportunities. Then, I challenged them to think about what they had learned--their answers to the second reflection question above--and to annotate these on the timeline with sticky notes. Unlike a normal retrospective meeting, where people work individually and then find clusters, this time I encouraged them to post items as soon as they thought of them, and to talk to each other about refining the articulations where appropriate. I circulated some colored dots, which could be used to affirm or agree with items others posted.

The result looked like this:

That's too small to read, but it gives you the overall shape. Below, I have transcribed each note, attempting to maintain the orthography while prudently fixing spelling errors. If an item had dots affixed to it, I mark this in parentheses.
  • Don't be afraid to step out of your bubble and try something NEW (2)
  • Narrative-driven games are really interesting and redefine regular games (2)
  • FOOD is important (1)
  • game design for academic research is DOUBLE hard
  • Slack is AWESOME (1)
  • Articulating how we record qualitative data is difficult
  • Getting qualitative data is important to understanding the project (& how it meets its goals)
  • Github and feature branching is amazing (1)
  • How to integrate my artwork into a working platform / game (1)
  • The importance of physical space (2)
  • The usefulness of the React library
  • How to build a game efficiently in an interdisciplinary team (1)
  • How to communicate with a team consisting of members of various disciplines (1)
  • How a culture's "monster" represents a real fear or threat to that culture (5)
  • Monsters are windows into culture (1)
  • Everything we make will be remade. Care about it but don't get attached. (2)
  • Pair programming accelerates production while minimizing errors (2)
  • How to adapt to a new set of programming libraries in a timely manner (2)
  • Implementing creative limitations makes for richer encounters (character limit, cultural implementation, multiple encounter reactions, etc) (1)
  • How to condense my writing to successfully meet character limits (1)
  • Design log is law... Design Lawg
  • Establishing visual metaphors as game mechanics (3)
  • In game design, the best idea wins! (1)
  • A good paper prototype takes time, but adds great clarity to the project (3)
  • Lack of specific roles in Agile/Scrum
  • It's okay to call a sprint a failure (5)
  • Jumping in on what others have been working on is difficult, yet a great learning experience (that should be had)
  • Don't throw up (1)
  • Going to events with the team really helps us bond (3)
  • I'm interested in game academia (1)
  • What we are making is something refreshing but innovative (3)
  • Google analytics: how to implement it
  • Sound design enhances the gaming experience (2)
  • Kids will find the limits of whatever you're developing (2)
  • We learned a lot (1)
I think this is an excellent list, full of beautiful learning outcomes for an interdisciplinary undergraduate course. The students seemed to enjoy the activity as well, engaging in the process with both quiet thoughtfulness and friendly laughter. 

One of the intentions behind this activity was to get them ready for their final exam. I have tried several different approaches to giving final exams for this kind of immersive learning class. This semester, I went back to our essential questions (on the course description) as well as the learning outcomes from my project proposal. I wanted to crack these open and expose the various ways that students can start to tease apart the lessons from the context. I am including the final exam verbatim below, which includes both prompts and a justification for itself:

What is the final exam?

Most of our work this semester was bound up in the making of Traveler’s Notebook: Monster Tales. Our team has built a shared understanding that is necessarily bound up in the context of our collaboration: the people, the problems, the places, the donuts, etc. The main point of the final exam is to help lift our thoughts out of the particulars of this context, to improve our ability to draw upon our understanding as we move on to other endeavors.

Choose two categories from those given below and respond to one prompt in each. If you think of a different prompt, or even a new category, let me know, and we can negotiate how to move forward. If you spent at least an hour at the Immersive Learning Showcase talking to guests about our project, then you only need to choose a prompt from one category, although you are always welcome to write more.

Transformative Games
  • Consider one or more of these essential questions:
    • How do games encourage or discourage the development of literacy?
    • What is the role of player culture in transformative game design?
    • What is the role of theory in research and development projects like ours?
What do you know about it now that you didn’t before the semester started? (Note that you don’t need to answer the question per se: the point of an essential question is to guide inquiry because it doesn’t have a closed-form answer.)

Expectations and Reality
  • What was your biggest surprise of the semester? Delve into it in an essay, considering: why was it a surprise (that is, what knowledge did you have or not have coming a priori)? When did you realize you were surprised? What did you learn from the experience?
  • Compare and contrast the final product to how you initially thought the game would turn out. What accounts for the differences?
  • If you could improve upon the game in one way (including game design, asset production, and technology platform), what would you improve, and why?

Reflective Practice
  • What was your biggest mistake of the semester? How did it come about, what did you do about it, and what do you think you learned from it?
  • Reflect on what you have learned this semester that is related to your involvement in the game studio, and choose one outcome that is most important to you. Write a reflection about the context of this outcome: Who was involved? In what places did it happen? Over what period of time? Was the experience mediated by technology writ large---software, designed spaces, furniture? What were the sights, the sounds, the smells, the tastes, and the tactile experiences involved? How did these factors contribute to this outcome?

  • Write an orientation document for future game studio students, something that could be given to them at the start of the semester to help them succeed in their work.


  • Produce an artifact in accordance with your academic focus that represents something significant you have learned this semester.
One of my students responded to this by sharing, “That moment when @paul.gestwicki makes the most reasonable final exam I've ever taken :raised_hands::clap::pray:"” I am not sure that reasonable is the first adjective most people use to describe my approach to teaching, but I am glad this student thought so.

I may write up a longer project retrospective of my own now that the semester is winding down, but for now, I wanted to share the list and thoughts about the final exam. As always, feel free to share your thoughts and reactions in the comments section.

Saturday, April 30, 2016

The Launch of Traveler's Notebook: Monster Tales

I am pleased to announce the release of my Spring Studio team's game, Traveler's Notebook: Monster Tales. It is a two-player, narrative-rich game in which you encounter monsters from various world cultures. The game grew out of my consideration of learning through games like Tales of the Arabian Nights, which I wrote about back in January, and which led to the team's identification of their goals and early design work. I plan to write a more thorough retrospective of the semester once it's all behind us. For now, check out the game at!

If you're in the vicinity of Ball State University, members of the team will be at the Immersive Learning Showcase on Monday, 4-6PM in Cardinal Hall at the Student Center. They would be glad to give you a demo and talk about their experience.

Wednesday, April 20, 2016

Metacognitive learning through a design thinking framework

I have written a few times about the design thinking model that I use, lightly adapted from George Kembel's:

Brian McNely and I wrote a paper about how we observed a team falling into a failure mode of looping like this:

That experience inspired me to start using this model as an exercise in CS222. Around the beginning of the third three-week iteration on the final project, I take a day where I begin by introducing this model. Then, I ask the teams to annotate their paths through the phases, starting with their project pitch, and annotating major milestones such as the ends of iterations. This is a great metacognitive exercise that gets students thinking about how they have been proceeding. Inevitably, I get a student who reports that they did it "wrong" by starting in somewhere other than empathy or by jumping between phases, but then I explain that we're not using this in a prescriptive way, but as a tool to help us think about our processes.

In this semester's CS222 class, we did this exercise last week Friday, which was the first week of the final iteration. This is the only photograph I have of the event, but rest assured that this is representative of the kinds of diagrams the students generate.

I usually tell the story about the study that McNely and I did, pointing out that sometimes we software development types get stuck in "ideate-build-test" loops without stepping back and remembering for whom we are building this thing. The arcs in students diagrams tend to have a similar shape, often skipping over the empathy step entirely (which is understandable since it's not really connected to our course learning outcomes).

On Friday, though, I saw something I had never seen before. The They recognized that they had made this transition early in their project:last team to present their models showed that they had a strong arc like this:

That is, they observed that they identified the real problems within a specific demographic, but they never actually decided which of those problems they would solve, and how they would do so. Instead, they went right from identifying the problems to programming. Having looked at their code in two formal evaluations and some information evaluations, I think this is astute: it perfectly describes the failure mode that they fell into. The team is pulling itself together in this last iteration, and I hope that this exercise is formative to that process.

As I was talking to my class about this model, I had a moment of inspiration where I realized there may be disciplinary differences in how one gets trapped in subcycles. This is only a hypothesis at this point, but I think I've seen this before in my multidisciplinary classes:

I see programmers fall into the trap where they think of something, build it, think of something else, build it, and so on, without ever meaningfully evaluating it or really returning to the user's needs. The humanists--such as English and History majors--tend to think about the problems that exist and think of solutions, then think of other problems, then think of solutions, and so on, without actually building anything to validate their ideas. I wonder if teaching this model early would help students to frame their processes in a useful way and help them catch themselves unproductive spiraling.

Wednesday, March 9, 2016

Leaving story maps behind

I wrote on the first of this year about how I was excited to try using story maps as a management aid for my Spring Studio team, and in the middle of February I followed up with some of my findings from one iteration's use. The team just finished their second iteration, and after talking about it with them during our end-of-iteration review and retrospective meeting, we are going to drop the story maps and go to Scrum.

The problem with the story map—as we reified it—is that it presumes that the reader already has knowledge of the fundamental processes that are required to complete a story. For example, we had an activity on the story map called "Narrate my friend's story" with a story underneath it called "Read conclusion and result." (StoryConclusion, and Result are all key terms from our game design and are defined in a design log.) When I look at that story, I can imagine the steps required to complete it: sketch the UI, make sure I have actual conclusions and results, make sure it fits with the overall game UX flow, rough it in code with placeholder assets, start replacing placeholders with real assets, test the integration. The problem is that I am not the one doing these steps: rather, it is a team of people who never did this before. What happened in the first two iterations with the story map is that they would look at "Read conclusion results" and, in the discussion, I would explain that these steps were required. These steps weren't tracked explicitly, though, and so we ran immediately into cognitive overload, and the team moved forward with the illusion that they knew how to proceed. Note that this is basically the same phenomenon as in a traditional class, where you can tell the class something and they will nod but not take notes, believing that they understand; of course these students then fail to follow the proper steps when it comes time to execute the task independently.

I am hopeful that switching to Scrum will make it easier for the team to track its progress on smaller tasks. It will mean more attention devoted to planning meetings, as we break down stories into individually-measurable tasks. One reason I was hoping for a leaner approach using story maps is that this is only a three credit-hour course, unlike the six credit-hour studios of the past few years, and so this will eat even more time away from production. However, if we're not producing the right thing, then it doesn't matter how quickly we do it.

Saturday, March 5, 2016

Rewriting departmental P&T documents to follow Scholarship Assessed

At the end of February, I gave a presentation at Ball State's Engaging Community series on the topic of publishing community-engaged scholarship. I was glad to be invited, although I don't generally describe myself as a "community-engaged scholar." I see community engagement as a necessary component of the authentic game studio experience: if my teams were not engaged with the community, we would not be doing authentic work. In any case, I took the opportunity to talk about the variety of research questions my colleagues and I have investigated, and the corresponding variety of venues where this work has been published. I have posted the slides, if you want to take a look, though as usual they may not make sense without the stories surrounding them.

Before me in the session were two respected colleagues, Chadwick Menning and Mellisa Holzman, who organize the Elemental sexual assault protection program. One of their main talking points was how to contextualize community-engaged scholarship for the promotion and tenure process. Given that a program such as Elemental is not conventional scholarship, some within the university perceived it as being solely a teaching experience (because it arose from a VBC seminar) or as service (because it involves helping people, I guess). Mellisa's sage advice was to contextualize and justify the work within your vita—a tactical maneuver that is facilitated by conceiving of the vita as a personal expression rather than a bureaucratic formula.

Inspired by their story, I began my talk with some extemporaneous remarks about my own P&T adventures. I told the audience that I didn't really have any trouble defending my work as scholarship because I had rewritten my department's P&T document to make it easier. Back in 2008, I chaired my department's P&T committee, and I had just recently read both Scholarship Assessed and Scholarship Reconsidered. With a favorable committee and authority from the department, I was the primary author of the changes that moved us from a laundry list of recognized activities toward a definition of what we mean by "scholarship". Here is the relevant section:
All works of scholarship will be judged according to the following standards (adapted from Glassick, et al., Scholarship Assessed: Evaluation of the Professoriate, John Wiley & Sons and the Carnegie Foundation for the Advancement of Teaching, 1997, p36). It is the responsibility of the faculty member to clearly demonstrate in writing that the standards listed below are satisfied by the body of work presented to document his or her bid for promotion or tenure.
  1. Clear goals. The scholar clearly articulates the basic purposes of the project, defines objectives that are realistic and achievable, and identifies the most important questions in the field.
  2. Adequate preparation. The scholar shows an understanding of existing scholarship in the field, brings the necessary skills to the project, and brings together the resources necessary to move the project forward.
  3. Appropriate methods. The scholar uses methods appropriate to the stated goals, applies effectively the methods selected, and appropriately modifies procedures in response to changing circumstances.
  4. Significant results. The scholar achieves the stated goals. The results of the project add consequentially to the field and open additional areas for further exploration.
  5. Effective presentation. The scholar uses a suitable style and effective organization to present the results of the project, uses appropriate forums for communicating these results to their intended audiences, and presents his or her message with clarity and integrity.
  6. Reflective critique. The scholar critically evaluates her or his own work, brings an appropriate breadth of evidence to her or his critique, and uses this evaluation to improve the quality of future work.
I should note that this section is followed almost immediately by a more conventional section describing what categories of evidence are recognized. However, whereas this was previously an exhaustive list, it is now a list of items that simply don't require much extra contextualization.
The following categories of evidence of scholarship are recognized by the department. However, support materials not fitting clearly into any of these categories may still be submitted. All evidence of scholarship should be documented according to the standards in [the section above].
  • Publishing refereed articles in journals or conference proceedings;
  • Publishing book articles, research monographs, or textbooks;
  • Writing reviews for national journals;
  • Presenting papers at professional meetings;
  • Conducting seminars, institutes, or workshops on computer science topics;
  • Evidence of success in developing, experimenting with, and implementing new teaching procedures and techniques;
  • Receiving grants or fellowships;
  • Adoption outside the department of software or electronically distributed materials that you have developed;
  • Serving as a professional consultant on matters of computer science to organizations or individuals outside the department.
There was no trouble with my department's approving these changes, and these policies have been in place ever since. When I have been considered for tenure, promotion, graduate faculty status, or merit pay, I simply provide a paragraph or two that establishes any unconventional work as scholarship given the six criteria.

This change did not instigate a visible cultural shift within my department: my colleagues are generally engaged in conventional disciplinary research. However, as far as I can tell, no one has pushed back against these changes either. Although I may be something of a black sheep in terms of how I conduct my research program, I don't sense anyone questioning the scholarly nature of it. I am sure it helps that, in addition to my more progressive work, I regularly produce traditional forms of scholarly output as well.

Although I am happy with this departmental policy, I also believe it does not go far enough in embracing Boyer's philosophy. Our document is patterned after a college document that uses a traditional three-category taxonomy of professorial activity: teaching, research, and service. That is, it misses the point that Boyer made, that everything a professor does should be scholarly—that the way of the scholar is scholarship. Worse yet, the college document reuses the word "scholarship" in place of the more traditional "research," which only further confuses the matter. Boyer's four scholarships simply don't fit into a model that separates scholarship from teaching and service. While I would like to see my department's document align with Boyer's model more explicitly, it's not a battle I choose to fight. Regardless, it was an epiphany for me when, as a young scholar, when I realized I could stop thinking about teaching, research, and service, and instead think about doing all things as a scholar.

(Erin Moore helped organize the university's session on Publishing Community-Engaged Scholarship, and she told me that she has received a few requests for more information about my department's P&T policies. I hope this post helps provide a bit more context than I could in some off-the-cuff remarks. As always, comments are welcome.)