A proposal for the disclosure of the use of generative AI in scholarly writing

I recently encountered a scholarly manuscript in which the author acknowledged ChatGPT as being instrumental in composing the work. This is fraught with similar problems to students' use of such generative AI tools: how is the reader to trust that the author actually means what is written? 

I propose that those who wish to publish such manuscripts must also release their original draft, prior to its being passed through AI tools. This recognizes that the purpose of these tools is to translate between natural language dialects, and so they should be treated like other translations. While I am reading an English translation of Francois Mauriac's Viper's Tangle, I can access the original French in order to distinguish between the connotations of the original and the interpretation of the translator.

An exercise in assessment of scholarship: Another look at the Year of Fam Jams

I'm sure I mentioned it before, but reading Boyer's Scholarship Reconsidered was formative in my development as a young scholar. It helped me abandon the uninspiring taxonomy of teaching, research, and service and instead to think about scholarly ways of being. Boyer's taxonomy includes Scholarship of Teaching, Discovery, Application, and Integration. It makes me sad to see how frequently these terms are misunderstood at my own institution, where people use them as neologisms for teaching, research, and service rather than really engaging with what Bloom was saying.

Glassick's Scholarship Assessed deals with the significant problem of determining whether or not an activity aligns with that scholarly way of being. This model can be used regardless of domain and regardless of what type of scholarship is being assessed. I have never encountered any opposition to Glassick's model except, if my memory is right, by one old crank who insisted that peer-reviewed publications were the only form of scholarship. I have met plenty of people who neither thought about them nor engaged with them, but I don't remember ever seeing someone try to refute them.

A ridiculous series of events leads me to an interesting writing and thinking exercise. I agree with Paul Graham's recent essay, which asserts that "no one who hasn't written about a topic has fully formed ideas about it." Regardless of the series of events, then, it's good to take the time to write. Indeed, I originally made this blog post last night, but after reflecting on it, I've returned to it this morning to tighten it up.

The topic for consideration is whether my 2020–2021 FamJam project represents scholarship. To address this, I will take Glassick's criteria as a given. One could frame this primarily as Scholarship of Integration, although it has elements of Teaching, Discovery, and Application as well.

Clear Goals

The project started with a few goals. One was to build original video games with my family through the application of best practices, contemporary scholarship, and appropriate tools for game design and development. This serves dual subgoals, which are that the family would learn from the experience and we would make something to bring joy to others. Combined with this goal was the desire to investigate the extent to which what I had learned about mentoring undergraduate game design and development teams would be applicable to my own family. That is, what elements or ceremonies from my scholarship of teaching would apply in this context, and how? Another goal from the beginning of the project was to use this experience with my family to inspire other people to try something similar themselves.

As the project continued and our process matured, I thought more about how I could communicate our findings to the right audience. That is, having effective dissemination was a goal from early in the project.

Adequate Preparation

I've been studying game design and development for over fifteen years and have written multiple peer-reviewed and published papers on the topic. I have mentored many student projects, and I myself have made many games and participated in jams, alone and in teams. Before starting my project, I also spoke extensively with a colleague who has guided families in Fam-Jams before, and we stayed in touch intermittently throughout. Maintaining my YouTube channel has also helped me gained adequate skills in video production, which would prove useful for meeting the dissemination goals.

Appropriate Methods

My family ended up doing 13 consecutive monthly Fam Jams, starting in March 2020 and ending in March 2021. Each month, we carved out a day to work on an original video game project. We assigned a creative director, worked on our project, and finished it before the day was out. We incorporated various reflective practices inspired by agile software development, engaging in semi-structured retrospectives after each project.

Doing this work was a kind of action research, getting into the field and participating in the work. As action research, the goal here was to understand particulars. Elements of the project may also be viewed in an autoethnographic light. Part of what makes this method appropriate is that we were making lemonade with the lemons we were given: a global pandemic had emerged, and lockdowns and distancing requirements were imposed, and so my family was the unit that I had to work with. 

We chose to release all of our projects publicly on the Web and to make all the code available under a Free Software license. This way, interested parties could learn from these artifacts, use them in their own work, and also, if desired, seek evidence of my claims made about them.

Significant Results

Over the course of 13 monthly jams, we learned more about working together. Some of these findings were significant in the small in that they helped us improve and learn. I mention this because in educational action research and Scholarship of Teaching, significance is often in-the-small: if the process worked with the intended class, then it is significant. Many of my writings and presentations about the Fam Jam experience mention particular areas where improvement was seen, including the quality of the source code, the formal structure of the music, the elegance of the gameplay, and the visual aesthetics.

Taking a wider view, the project gains further significance. I developed a tested structure for one-day Fam Jams. Through iterative reflection on what we learned, I was able to develop a list of our learning outcomes, which can then be used as potential learning outcomes for others.

As far as I know, mine is the only work of this kind. It has served as a talking point for several of my formal and informal presentations about game design and development.

I have talked to my students about this experience and shown them how I approach it. In this way, one of the significant outcomes of this work is that it models a scholarly way. Students are often inspired to make games because of a love of playing games. There are countless articles and videos explaining how to make games, but what I provide in my work is a framework for how to make games thoughtfully and within a scholarly method. The projects themselves have also served as reference points in my teaching, to show students what is possible in a one-day timebox.

A final point here that could be argued is that the projects themselves were significant because they brought joy to people outside my family. One of our goals was to spread some joy during a dreary season, and we succeeded. Discussing whether this is significant is a philosophical question that could sidetrack us, and my defense of this point is rooted in the transcendentals.

Effective Presentation

There are many layers to the question of effective presentation. I started writing blog posts about my experience from the beginning so that I could come to a clearer understanding. My initial hope was that interested people would search the Web and end up here. 

There is a lot of game design and development content on YouTube, but I did not originally see a way to talk about my FamJam experience on my own YouTube channel. Hooking into GodotCon would increase the chances that I could inspire people and get my findings into the hands of those looking for it. Those seeking information about running a game jam with their family would watch a video and might read a blog post, but unless they are a scholar themselves, they would be very unlikely to find or read a journal article. 

When I saw the call for presentations for an online version of GodotCon, I realized this was an opportunity to reach a much wider audience. GodotCon 2021 was on Jan 23, 2021, and the whole event was streamed on YouTube. This excerpt includes only my presentation. The conference featured an online chat system whereby the online attendees could interact with the presenters. I engaged with the chat there after my presentation, which was another form of effective presentation.

As mentioned earlier, the software itself is also presented publicly, both the source code and the executables. Anyone who reads my reflections or is inspired by my work can see the actual fruits of our labor via GitHub.

Considering all this, I would argue that this project is potentially one of the most effectively presented works I have ever done. It hits multiple modes and gets information directly where it would be sought.

Reflective Critique

Critical self-evaluation is an essential element of this work: moving through the process toward my goals was a process of regular reflective critique. I have written numerous pieces about the FamJam project, all of which feature at least a modicum of reflection, while some are purely reflective. To illustrate the point, I've gathered links here to all my blog posts during the FamJam project.

1. Fam Jam #1: Joe Johnson Gets Captured
2. Fam Jam #2: The Rocket and UFO Game
3. Fam Jam #4: Get the Food (and a few words about #3)
4. Find the Ornaments: Thoughts about the December 2020 Fam Jam
5. Reflecting on almost a year of FamJams: What the family learned

The work also produced these additional blog posts about technical issues that came up while jamming.

1. Deploying Godot Engine Web Games to GitHub Pages with Continuous Integration
2. A slightly easier way to get started publishing to GitHub Pages with Workflows

These two have a corresponding YouTube tutorial.

My family has continued to do game jams together, and we have experimented with varying the format to reach different learning outcomes. I have contemplated running workshops locally to promote this kind of family event, but the ongoing pandemic and other responsibilities limit such opportunities. I think it would be fascinating to do an ethnographic study of a Fam Jam, and such work would easily find publication in the field of serious games. I am not currently pursuing this line of research, however, since the return to in-person teaching has turned my attention toward studying phenomena around my university students rather than the local community.

That's it. I think this makes a strong case for the 2020-2021 FamJam project being an example of scholarship. I'm happy to hear any refutations you have to the points made above, as I'm sure it would be an opportunity to sharpen my own understanding of Glassick's framework.

Revisiting the state analysis of Every Extend

Back in 2007, my first Computer Science education research paper was published in the proceedings of SIGCSE. The paper provides an overview of how game programming can be used as a motivating context for learning design patterns. One of the examples in the paper, which I have used for years, is the behavior of the player-controlled bombs in the inimitable Every Extend, which can be used to explain the State Design Pattern. In the paper, I argue that the player's pawn can be described by a state machine like the following.

In yesterday's meeting with my CS315 Game Programming students, I was inspired to create a UE4 tutorial video about using state machines for state management. In particular, I wanted to show how an enumeration can be used to define the possible states, and then behavior can be switched depending on the current state. This is not, of course, the State Design Pattern, but it's a heck of a lot better than an ad hoc collection of Boolean variables. It also follows from Michael Allar's Gamemakin' UE4 Style Guide that is mandatory for my students, which says:

Do not use booleans to represent complex and/or dependent states. This makes state adding and removing complex and no longer easily readable. Use an enumeration instead.

To prepare for the video, I have spent a few hours the last two days building a simple playable framework similar to Every Extend. Over a decade ago, I did a similar exercise in Java to produce EEClone. This time, I did it in UE4. (A bit of an aside: Every Extend is like my kata for new game programming frameworks. I use the essential gameplay of it as a case study to understand how game engine functions. Of course, then, when I first started learning UE4, I tried making an Every Extend clone, but it went quite poorly. After a few years of working with UE4, it's become a lot easier to get the game up and running!)

With my gameplay framework in place, I started working on the diagram shown above to use in my series of game programming tutorial videos. However, as I looked at the diagram, I realized I had a rather fundamental problem: these are not the states of the player bomb. It's true that these are, abstractly, states of the game, but they are not the states of a single stateful object. Instead, what I had programmed was leading into the following state model:

That is, a player bomb is created, which has it play a spawning animation. When that is completed, then the player becomes vulnerable to collisions but can also press Space to explode. However, unlike in the initial analysis, there are no other states after this. If the player presses Space or is hit by an obstacle, the player's bomb is destroyed: that actor—that object—no longer exists. It is the game mode that either counts down to respawn or waits until a chain of explosions is complete and then counts down to respawn. These states are not part of the player pawn.

This observation shoots a hole in my plan to make a video using the Every Extend example. I still want to make a video about state analysis, but I need to find a better example—one where a single actor's state is interesting enough to be modeled but simple enough to fit into a tutorial video. Please feel free to share your suggestions in the comments.

Rewriting departmental P&T documents to follow Scholarship Assessed

At the end of February, I gave a presentation at Ball State's Engaging Community series on the topic of publishing community-engaged scholarship. I was glad to be invited, although I don't generally describe myself as a "community-engaged scholar." I see community engagement as a necessary component of the authentic game studio experience: if my teams were not engaged with the community, we would not be doing authentic work. In any case, I took the opportunity to talk about the variety of research questions my colleagues and I have investigated, and the corresponding variety of venues where this work has been published. I have posted the slides, if you want to take a look, though as usual they may not make sense without the stories surrounding them.

Before me in the session were two respected colleagues, Chadwick Menning and Mellisa Holzman, who organize the Elemental sexual assault protection program. One of their main talking points was how to contextualize community-engaged scholarship for the promotion and tenure process. Given that a program such as Elemental is not conventional scholarship, some within the university perceived it as being solely a teaching experience (because it arose from a VBC seminar) or as service (because it involves helping people, I guess). Mellisa's sage advice was to contextualize and justify the work within your vita—a tactical maneuver that is facilitated by conceiving of the vita as a personal expression rather than a bureaucratic formula.

Inspired by their story, I began my talk with some extemporaneous remarks about my own P&T adventures. I told the audience that I didn't really have any trouble defending my work as scholarship because I had rewritten my department's P&T document to make it easier. Back in 2008, I chaired my department's P&T committee, and I had just recently read both Scholarship Assessed and Scholarship Reconsidered. With a favorable committee and authority from the department, I was the primary author of the changes that moved us from a laundry list of recognized activities toward a definition of what we mean by "scholarship". Here is the relevant section:

All works of scholarship will be judged according to the following standards (adapted from Glassick, et al., Scholarship Assessed: Evaluation of the Professoriate, John Wiley & Sons and the Carnegie Foundation for the Advancement of Teaching, 1997, p36). It is the responsibility of the faculty member to clearly demonstrate in writing that the standards listed below are satisfied by the body of work presented to document his or her bid for promotion or tenure.

1. Clear goals. The scholar clearly articulates the basic purposes of the project, defines objectives that are realistic and achievable, and identifies the most important questions in the field.
2. Adequate preparation. The scholar shows an understanding of existing scholarship in the field, brings the necessary skills to the project, and brings together the resources necessary to move the project forward.
3. Appropriate methods. The scholar uses methods appropriate to the stated goals, applies effectively the methods selected, and appropriately modifies procedures in response to changing circumstances.
4. Significant results. The scholar achieves the stated goals. The results of the project add consequentially to the field and open additional areas for further exploration.
5. Effective presentation. The scholar uses a suitable style and effective organization to present the results of the project, uses appropriate forums for communicating these results to their intended audiences, and presents his or her message with clarity and integrity.
6. Reflective critique. The scholar critically evaluates her or his own work, brings an appropriate breadth of evidence to her or his critique, and uses this evaluation to improve the quality of future work.

I should note that this section is followed almost immediately by a more conventional section describing what categories of evidence are recognized. However, whereas this was previously an exhaustive list, it is now a list of items that simply don't require much extra contextualization.

The following categories of evidence of scholarship are recognized by the department. However, support materials not fitting clearly into any of these categories may still be submitted. All evidence of scholarship should be documented according to the standards in [the section above].

• Publishing refereed articles in journals or conference proceedings;
• Publishing book articles, research monographs, or textbooks;
• Writing reviews for national journals;
• Presenting papers at professional meetings;
• Conducting seminars, institutes, or workshops on computer science topics;
• Evidence of success in developing, experimenting with, and implementing new teaching procedures and techniques;
• Receiving grants or fellowships;
• Adoption outside the department of software or electronically distributed materials that you have developed;
• Serving as a professional consultant on matters of computer science to organizations or individuals outside the department.

There was no trouble with my department's approving these changes, and these policies have been in place ever since. When I have been considered for tenure, promotion, graduate faculty status, or merit pay, I simply provide a paragraph or two that establishes any unconventional work as scholarship given the six criteria.

This change did not instigate a visible cultural shift within my department: my colleagues are generally engaged in conventional disciplinary research. However, as far as I can tell, no one has pushed back against these changes either. Although I may be something of a black sheep in terms of how I conduct my research program, I don't sense anyone questioning the scholarly nature of it. I am sure it helps that, in addition to my more progressive work, I regularly produce traditional forms of scholarly output as well.

Although I am happy with this departmental policy, I also believe it does not go far enough in embracing Boyer's philosophy. Our document is patterned after a college document that uses a traditional three-category taxonomy of professorial activity: teaching, research, and service. That is, it misses the point that Boyer made, that everything a professor does should be scholarly—that the way of the scholar is scholarship. Worse yet, the college document reuses the word "scholarship" in place of the more traditional "research," which only further confuses the matter. Boyer's four scholarships simply don't fit into a model that separates scholarship from teaching and service. While I would like to see my department's document align with Boyer's model more explicitly, it's not a battle I choose to fight. Regardless, it was an epiphany for me when, as a young scholar, when I realized I could stop thinking about teaching, research, and service, and instead think about doing all things as a scholar.

(Erin Moore helped organize the university's session on Publishing Community-Engaged Scholarship, and she told me that she has received a few requests for more information about my department's P&T policies. I hope this post helps provide a bit more context than I could in some off-the-cuff remarks. As always, comments are welcome.)