Bloom's Taxonomy, Teaching, and LLMs

Recent discussions of LLMs in the classroom have me reflecting on Bloom's Taxonomy of the Cognitive Domain. Here's a nice visual summary of its revised version.

Blooms Taxonomy of the Cognitive Domain
(By Tidema - Own work, CC BY 4.0, https://commons.wikimedia.org/w/index.php?curid=152872571)

Bloom's Taxonomy, as it is called, is a standard reference model among teachers. The idea behind it is that a learner starts from the bottom and works their way upward. As far as I know, it has not been empirically validated: it's more of a thought piece than science. This is reflected in the many, many variations I've seen in the poster sessions of games conferences, where some young scholar proposes a play-based inversion that moves some piece into a different position on the trajectory. All that is to say, take it with a grain of salt. The fact remains that this model has had arguably outsized influence on the teaching profession. (Incidentally, I prefer the SOLO taxonomy.)

There's been a constant refrain the past few decades among a significant number of educators and pundits that technology has made obsolete the remember stage. Why memorize this table of values when I can look them up? Why remember how this word is spelled? Spellcheck will fix it for me. My skepticism of the concept has only increased as I have worked with more and more students who use digital technology as a crutch rather than a precision instrument.

LLM-generated code comes up in almost every conversation I have among teachers and practitioners in software development. There are ongoing studies into the short- and long-term implications of using these tools. My observations are more anecdotal, but it's no exaggeration to say that every professional developer and almost every educator has landed in the same place: LLMs can generate useful code, but knowing what to do with it requires prior knowledge. That is, the errors within the LLM-generated code are often subtle and require knowledge of both software engineering and the problem domain. 

From the perspective of Bloom's taxonomy, a developer with a code-generating LLM is evaluating its output. They come to their evaluation by building upon the richness of cognitive domain skills that undergird it. At the very fundamental level, they bring to bear a vast amount of facts about the praxis of software development that they have remembered and understood.

If Bloom is right, then among the worst things we could do in software development education is throw students at LLMs before they have the capacity for viable evaluation. Indeed, before LLMs, the discussion around the water cooler was often about how to stop students from just searching Stack Overflow for answers and submitting those. Before Stack Overflow, it was that students were searching the web for definitions rather than remembering them. My hypothesis for learning software development then is something like this:

• Google search eliminates the affordance for learning to remember.
• Stack Overflow eliminates the affordance for learning to understand.
• LLMs eliminate the affordance for learning to apply.

This hypothesis frames the quip that I share when an interlocutor discovers that I am a professor and, inevitably, asks what I think about students using ChatGPT. My answer is that I'm considering banning spellcheck.

Walking away from a November game project: A reflection on NoGaDeMon 2024, Dart, Flutter, and Bloc

I would hate to make this a tradition, but it seems that I once again entered NoGaDeMon. National Game Design Month (NaGaDeMon) is November, and for several years, I created interesting little projects during the month. Last year, I was not able to pull a project together, and I'm afraid that's the case this year as well. However, I was able to learn a bit through the attempt, so I want to capture some of it here before it slips away.

Before November, I had been tinkering with an intersection of ideas related to posts in the last few months: interactive narrative games like my The Endless Storm of Dagger Mountain, which drew from the Powered by the Apocalypse tabletop RPG space, built around some concepts from Blades in the Dark and Scum & Villainy. I figured that, for November, I would try building a very small slice of the idea. For various reasons, I also wanted to try building and releasing a game using Dart and Flutter. I dug in and started making reasonable progress for a side project.

A few days into November, John Harper released Deep Cuts, a campaign and rules expansion for Blades in the Dark. I bought a copy and was quite surprised at the rules changes. I had expected little tweaks and balancing maneuvers, but Deep Cuts actually provides a complete overhaul of the most fundamental Blades action resolution system. This was too cool not to play with, so I rehashed my planned NaGaDeMon project, essentially starting from scratch to support some of the Deep Cuts ideas.

Before last week, I was able to get a very small version of the game working, letting the player experience a single, badly written game scene. The user-interface was just awful, so in order for the game to come together would have required adding a ton of content and a complete player experience design and implementation. Both of those would be tedious efforts, especially the latter, since I am not very fast with Flutter UI development. Part of the inspiration for choosing Flutter was to gain more practice with engaging UIs. 

About two weeks ago, the work of one of my committees exploded into taking most of my unassigned work hours, and this was not altogether unexpected. We also just got the good news that we will be hosting family for several days around Thanksgiving. This will be wonderful, although it also means these won't be hobby-project days. The result is that I've decided to put this project to rest. I did learn quite a bit going this far into the project, and that is the topic for the remainder of this post.

First of all, the obvious lesson is that if I wanted to really focus on learning to make a top-notch interactive Flutter UI, I should have chosen something with zero other design risks. I knew that the best I could do in one month was to make something just functional, yet I am not sure I was honest with myself about how ugly that would likely end up. Maybe I will find a game jam that will let me get a better handle on combining turn-based game timing with implicit animations.

Prior to November, I had been tinkering with some of these design inspirations in Godot Engine, which is of course the engine I used to build The Endless Storm of Dagger Mountain. I was using a rather conventional mutable-state object-oriented architecture. I found myself frequently frustrated by the lack of good refactoring tools for GDScript. This is a significant hindrance to evolving an appropriate design. This is part of what made me switch over to Dart, which is a joy to work with in part because of the excellent tooling support from Android Studio. 

A few summers ago, I spent a great deal of time studying Filip Hracek's egamebook repository. Nothing shippable came out of my efforts—I don't think I ever even blogged about it—but I did learn a lot. I was struck by how Hracek separated the layers of his architecture, and it was the first time I spent a lot of time in a game that used immutable data models. At the time, I had looked into the Bloc architecture and struggled to make sense out of it.

Approaching this November's project, I decided to dig deeper into Bloc. I spent a lot of time with the official tutorials and puzzling over this seemingly simple diagram:

The simple tutorials are simple, which is convenient, but the more robust ones separate the "data" component into a data provider and repository. It seemed clear that the game state could be conceived of as data, but I struggled to conceptualize where the game rules should live. The game rules can be considered part of the domain model, and as such, should be separated from the bloc. This would mean that a response from the domain model may be the modified game state, which then is echoed back through the bloc to the UI with a bloc state change. However, it's also reasonable to conceive of the game state itself as the data layer and the "business logic" as being the transformations of that state. Indeed, this seems to be the difference between the simple and more complex tutorials: the simple ones deal with simple in-memory state, and the more complex ones draw data from different sources and transform them in the data layer. 

Of course, there is no silver bullet. Given the tight time constraints on the project, I simply considered the immutable game state to be my data layer, and I put the game logic in a bloc. I also simply passed the game state along to the UI, but in a more robust solution, I would have had clearer separation between layers. Including a dependency between the UI and the data layers was a matter of expedience and the intentional incurring of technical debt.

My first pass at the implementation had me writing my game states and bloc states by hand. The Equatable package meant that I didn't have to fret over writing some of the boilerplate that's necessary to do state comparisons, and it was easy to integrate this in Android Studio using Felix Angelov's Bloc plugin. When scouring the Web for help with Bloc, one quickly also comes across discussions of Freezed, which library is also integrated into Angelov's plugin. I had tinkered with Freezed in my egamebook-inspired explorations, but I have not shipped anything that uses it. After having built up my understanding of Bloc using Equatable, Freezed was an obvious next step. Next time, I would jump right into using it for cases like this.

Writing a functional Flutter user-interface was straightforward using BlocBuilder. I found this to be a convenient way to conceptualize the game, especially since it had very clear states. For example, in my original explorations (before Deep Cuts), I had the player choosing an action from a list, then customizing the action with various options from Blades in the Dark, such as pushing yourself to trade stress for dice. After rolling the dice, the player is now in a different state of the game in which they are responding to the result, such as by resisting its consequences. This was elegant to express in the code, and I am confident that with enough effort, I could make a compelling user experience out of it. By contrast, Dagger Mountain used an architecture inspired by MVP but that depended too heavily on the undocumented, unenforceable behavior of coroutines. Both of these are "only jam projects," but they are helping me to conceive of how I would approach something more significant in this problem domain. The aforementioned coroutines were my solution to synchronizing the model and view states (for example, to finish an animation before continuing to the next step of the narrative); I'm fairly certain I understand how I can do that with bloc's events and states, but since the November project will remain unfinished, there is risk.

All this exploratory coding meant that I did not follow a test-driven process. I ended up not getting into the testing libraries specifically for bloc. It's possible that this would have helped me better to conceptualize the business logic versus the domain layer, but that remains future work. 

There are still a lot of questions about the game design itself. Indeed, this entire exploration is inspired by design questions around the adaptation of Blades in the Dark tabletop gameplay into a digital experience. Citizen Sleeper is the only project I know of that has worked in this space, and it's a fantastic interpretation. I only became aware of Citizen Sleeper after I started doodling my own ideas, and it's interesting to see where they converge and where they diverge. I hope to dive back into this design space later, but for now, my attention must go toward wrapping up this semester, planning for next semester, and enjoying the upcoming Thanksgiving break.

What people believe you need to do to be an independent game developer

Aspiring game developers are starving for advice. I recently attended a meetup of game developers where an individual gave a formal presentation about how to become an indie. The presentation was thoughtfully crafted and well delivered, and it was entirely structured around imperatives—the things that you, the audience member, need to do if you want to be a successful independent game developer. The audience ate it up and asked for more. They were looking for the golden key that would unlock paradise.

There are two problems here, one overt and one subtle. The overt one is that there is no golden key. There is no set of practices that, if followed, will yield success. I imagine most of the audience knew this and were sifting for gold flakes. However, it was also clearly a mixed crowd, some weathered from years of experience and some fresh-faced hopefuls. I hope the latter were not misled.

The subtler problem was made manifest during the question and answer period when it became clear that the speaker was not actually a successful indie game developer at all. Their singular title had been in development for three years and had just entered beta. They had no actual experience from which to determine if the advice was reasonable or not. The speaker seemed to wholeheartedly believe the advice they were giving despite not being in a position to draw conclusions about their efficacy.

Once I saw the thrust of the presentation, I started taking notes about the kinds of advice the speaker was sharing. 

•  Document everything, and specifically create:
• Story and themes document
• Art and design document
• MDA document
• Have a strong creative vision
• Be a role model for the work environment you want
• Consider these pro tips for hiring staff:
• Use a report card to score your candidates
• Look for ways to get to know what it would be like to work with them
• Try collaborating with them as part of the interview
• Always have a back-up candidate, not a top candidate but someone you know you could work with
• Being their best friend does not mean you should work with them
• Thank people for their contributions and efforts
• Use custom tools to help you work better
• Use the Asset Store in Unity
• Use tools to help you test
• Automate as much as you can to save you time
• Learn to prompt so you can use generative AI
• It allows an artist to be a developer by removing coding barriers
• LLMs can replace tedious use of YouTube, Google, Reddit, etc.
• When pitching to publishers, have two versions of your slide deck:
• pitch slides: the version you send
• pitch presentation: the version you present
• Take budgeting seriously
• Budget for specific deadlines
• Don't spend your own money if you can get money from someone else (e.g. publisher)
• Get a job so that you can support yourself until you can get funding from someone else for the game project
• Quoting one of his professors: "To make money, you need to spend money, and to spend money, you need money."
• Don't get distracted by others (e.g. on social media)

These aren't the things you need to do to be an indie game developer. These are the things that an audience believed you need to do to be an indie game developer or the things that someone with a modicum of experience thought would be worth telling indie hopefuls. It seems to me that this is the advice you would get if you spent an afternoon collecting advice by searching the Internet. It's helpful for me to have a list of what people are likely to believe from consuming popular advice. Sometimes advice is popular because it is accurate; sometimes people tell you to make your game state global.

Three other things jumped out at me about the presentation. First was the unspoken assumption that one would be using Unity. There was no indication from the speaker that this was even a choice, and none of the questions reflected on it. Second, the speaker acknowledged the importance of automation and automated testing, which was great to see. Third, no one pushed back regarding the use of CoPilot or other LLMs to help with coding, whereas I suspect there would have been a riot had he suggested using the same tech to generate artwork. There's a study in there.

Serendipity

As mentioned in yesterday's post, I was at Meaningful Play 2024 a few weeks ago, and I'm finally processing the many pages of notes that I took there. 

Sabrina Culyba gave the morning keynote on that last day of the conference. She spoke about serendipity in game design, sharing a compelling story about the development of Diatoms. The talk was brilliantly prepared and executed. She summarized research findings around serendipity that shows that the following factors can affect its likelihood:

• Having a prepared mind
• Openness
• Being connection-prone
• Belief in serendipity

These are really interesting, and if I didn't have a pile of other research projects in the hopper, I'd be curious to dive into the literature here. The first item sounds like a variation on the maxim, "Luck favors the prepared." The second sounds to me like the eponymous Big Five personality trait that tracks with creativity.

I don't have much else to contribute to the discussion, but it's a neat idea that I don't want to waste away in my notebook.

Fantasy heartbreakers

 I am currently reading William White's Tabletop RPG Design in Theory and Practice at the Forge: 2001-2012 after having met the author at MeaningfulPlay. This excerpt from Chapter 3 made me shout with delight at having a name for a phenomenon.

A fantasy heartbreaker was [Ron Edwards'] term for an independent game that contained interesting innovations, usually without realizing that they were in fact innovative, but whose designers had failed to fully examine their underlying design assumptions—thus producing games that were highly derivative of D&D, whether or not that was actually a design goal of the game—and who were either naïve or overambitious in their expectations for success in the marketplace. (p.93)

Ron Edwards' original post on the topic is cited, but I haven't made the time to read the source yet. White's summary was enough to excite me and want to share it here.