On the applications for Large Language Models in Gaming

For the purposes of this article I am using the term AI to refer to generative AI, and in particular, I am focusing on LLM’s, or “Large Language Models”. The scope of AI in games obviously extends beyond this, but this will be the focus of this piece.

Introduction

I am currently working on integrating support for Large Language Models into my project, Spiritus Astrum.

While working on the technical aspects of this, I have also been thinking about the greater applications for LLM’s in gaming, and of the developing (And deepening) controversy over the same.

Steam has recently clarified their policy on Generative AI. It is now acceptable to submit a game using generative AI on their platform provided that the developer declares that the game uses generative AI, and that the AI contains “guardrails” that prevent it from generating illegal or offensive content at runtime.

However, this declaration by steam has led to some controversy by gamers: Some gamers (Like some artists) believe that generative AI is “stealing” and that it should be banned.

Others believe that allowing Generative AI will lead to a flood of poor-quality AI generated games that will harm steams reputation as a distribution platform, and even harm the gaming industry as a whole.

I try to avoid politics with my blog and the projects that I work on, but it seems that with the ongoing AI debate I may be unintentionally entering a politically charged sphere.

I understand the concerns over generative AI.

There is no doubt that AI has the potential to do harm. In addition to the risks of “Shovelware” titles as I discussed above, there is also the potential that generative AI models could produce, (either intentionally or accidentally), inappropriate or offensive content. Even a safe, “Censored” AI model can be “Jailbroken” in various ways to produce content that it should not be able to produce.

However, despite this, I am still, generally Pro-AI. I would draw a comparision to something like the internet: There may be negative aspects to the technology (Possibly strongly negative aspects) but its usefulness and benefits have made it such a ubiquitous part of our lives that, for better or worse, it is here to stay.

Potential Applications for Generative AI in gaming

I believe quite strongly that Generative AI has a tremendous potential to enrich games in ways that never would have been possible before, to create a whole new era of dynamic, AI-driven content where every world is rich and engaging, and every play experience is unique.

By harnessing the power of LLM’s, we could see the end of boring NPC’s with canned, repetitive phrases. How many times, for example, have we heard Lydia from Skyrim saying “I am sworn to carry your burdens?” Or the famous “Pam param, pan pan param” song hummed by Witcher 3 NPC’s?

Instead, each NPC could have unique dialog, a unique personality, goals, and a strong presence and sense of agency in the world. This could provide for an incredibly rich experience, making the world feel truly lived in. The player would be a part of a greater world, and not the only reason for that world existing.

It is possible to add some randomness to current NPC dialog using a rule based system, where the AI will respond differently based on the current weather, current location, etc (Kingdom Come Deliverance has different NPC dialog depending on whether the players clothing and armour are clean or not!) but this dialog will still get very repetitive very quickly.

In addition, LLM’s can be used to supplement the pre-written dialog for main characters by providing additional dialog between quests, or after a quest has completed.

Even big budget RPG’s have limits to how much dialog has been written, and when you are in between quests, some of the most dynamic and engaging characters revert to a random selection of canned phrases.

LLM’s can also provide post-game content in the same way. Even when the main story is complete, the player can continue to “live” in the world, experiencing new “content” in a way that was never possible before.

Finally, LLM’s could be wired into the games quest system, enabling a true RPG-like experience.

Generative AI as “Dungeon Master”

This point is where I think the true potential of LLM’s lies. The vast majority of modern RPG’s are inspired in some way from the Dungeons and Dragons style of tabletop RPG’s (And, by extension, the old MUD’s, or “Multi-user Dungeons” from the early days of the internet). However, graphical RPG’s face a hurdle that none have yet to overcome: They all allow you only to select and customise a character, not to truly create one.

True, some games provide extensive character customisation options (Skyrim, Cyberpunk 2077, etc), but the characters backstory is usually based on some variation of class selection (Rogue, Mage, Paladin, etc) and not actually creating a character.

Imagine an RPG where you can truly “create” a character, with a backstory to go with them, using a text prompt? Where the game functions as a “Dungeon Master” for your very own adventure?

A world that dynamically adapts to challenge and encourage the player, a game where every playthough is completely unique?

This is the power that LLM’s have the potential to provide to gaming, and it is a power that cannot be ignored.

Experiments, Examples, and Analysis with Generative AI

I have recently conducted some very simple tests to explore the potential of LLM’s within my game, Spiritus Astrum.

All of the below tests were run with an 8b model, most with Llama 3.1 8B at a Q4 quant. This is, obviously, far smaller than the 70b and 120b models that I have been running on my new AI test rig. However, even though I have added the ability to connect to a KoboldCPP server using unity, very few players have access to this kind of hardware, so I am instead using a model that would be small enough to run on an average gaming computer.

For these tests, the information that the AI was given in the prompt was:

You are a Wasteland wanderer, surviving on a desolate, war-torn, ruined world.

Short Description of the AI’s character.

You are talking to a North Star Starship captain, a combat veteran.

Short Description of the players character. This could be entered by the player during character creation. It could be created completely by the player.

They are carrying a 10mm pistol, a medical kit, and food ration pack.

A list of the items in the players inventory, this can easily be obtained by simply converting the name of each item to a text string.

It is currently daytime.

It is hot and humid.

Time of day, weather, and similar information can be obtained from the games variables.

Nearby there is an old, ruined house, a piece of starship wreckage, and a supply crate. In the distance there is an old radio tower and a military base.

This information can be obtained by doing a radius search and then taking the name of any objects found and adding it to the prompt.

In this way, the prompt can be generated and updated quite easily.

So, lets take a look at a few sample outputs. I still have a lot more testing to do in this area, this is just the beginning.

AI:
Greetings, traveler. I’m doing well, considering our circumstances. It’s not every day one gets to explore the ruins of civilization alongside a seasoned adventurer like yourself. That 10mm pistol looks like it’s seen some action; what stories does it hold? And how about that medical kit? Have you had to use it recently? By the way, would you like something to drink from my ration pack? It’s not much, but it might help quench your thirst in this sweltering heat. What do you say?

This is an excellent response. Not only does it make sense, but the AI has also correctly “realised” that the player is a “seasoned adventurer” and that the Pistol in their inventory has “seen some use” likely because the prompt indicated that the player was a “combat veteran”.

It would be possible to use conventional techniques to implement something like this, but this would, first of all, lead to a LOT of rules being necessary, and second of all if would still lead to generic, repetitive, monotonous phrases.

Furthermore, If I change the player part of the prompt to:

You are talking to an inexperienced starship crew member. This is their first field mission. They are carrying a 10mm pistol, a medical kit, and food ration pack

Then the AI has a quite different response:

I’m doing alright, considering the circumstances. It’s not every day you get to explore a post-apocalyptic wasteland. What about you? First time out in the field, huh? You look a bit overwhelmed. Don’t worry, I’ve been around these parts for a while. If you need any advice or help, just let me know. We can start by checking out that supply crate over there. It might have some useful gear inside. And be careful with that pistol – it’s not as reliable as you might think. Now, what do you say we take a look?

The AI has responded correctly based on the players inexperience, and even warned them not to allow their pistol to discharge accidentally! This is a subtly different interaction than before.

This type of variation would not be possible with a rule based system, even a complex one.

A rule based system (or any other “conventional” AI dialog system would also not allow the AI to alter their response based on other factors, such as weather (Sweltering heat), enemies being present or not present, or any of the multitude of other factors and variables that would be present in a computer game.

The only thing that the AI got wrong was in relation to the ration pack. The AI is offering the player a chance to drink from their (The AI’s) ration pack, even though it is actually the player that is carrying the item.

Another example:

AI:
Hello, How are you? I’m doing alright, considering the circumstances. It’s not every day you get to meet someone from the stars. What brings you to this godforsaken place? And what’s with all the gear? Are you planning on sticking around for a while?

USER:
I’m hoping to find a ship, to get off this planet, do you know anywhere I can find one?

AI:
A ship, huh? That’s a tall order. Most of them got destroyed in the Great War or scavenged for parts by now. But I might have some info that could help you out. There’s a rumor going around about a hidden bunker beneath the old military base over there. Supposedly, it’s filled with tech and supplies, including maybe even a functional spacecraft. But be warned, the base is crawling with raiders and mutants. You’ll need to be careful if you’re thinking of checking it out. As for me, I’ve been surviving here for years, so I guess you could say I’m sticking around for a while. What about you? What’s your story? How did you end up here?

This is a another very good response. Not only is quality of the AI’s dialog quite good (despite being completely procedurally generated!) but the AI “knows” that there is a military base nearby, and is recommending that the player go there to explore!

Further Potential Applications

Even these few examples should be enough to illustrate just how powerful Generative AI can be for the gaming industry. Imagine a world where the interactions with NPCs and characters subtly shifts based on the players actions.

Instead of simply having a “moral choice” system which grants the player a “Good Ending” or a “Bad ending”, the players actions could influence the way that the entire game world reacts to the player.

The player could choose who they are and how they want to interact with the world completely at their whim, and the world would respond to that, in totally unique ways every time.

I have not even mentioned the potential for Multimodal models with “Vision” support, such as (LLAva: Large Language and Vision Assistant).

These models have the ability to “see” and image and answer questions about it, describe it, provide a caption, etc.

Within games, it could be possible to send an image of the scene to the LLM at runtime and allow the AI to respond based on what is going on in the world, even without any variables or prompts at all.

I have not experimented much with vision based models, but the potential is certainly there.

Generative AI as the future of Gaming

It used to be the case that graphical improvements drove the gaming industry forward: Every few months it seemed gaming would surpass itself with graphical fidelity.

While this is still happening, I believe the pace has slowed considerably. If you look at games released in the early 2000’s even a period of a year or two would result in noticeable improvements in graphical quality, today, most games made in the last 5 years or so show only modest incremental improvements year on year.

I believe that if the gaming industry is going to continue to innovate, that innovation is going to have to come from somewhere other than graphics, and that place could very well be AI.

AI could be the breath of fresh air that the industry needs to reach entirely new heights. In a few short years we may see an entirely new dawn, offering truly immersive, player-centric stories that are as unique as the players themselves.

The Fatal Flaw

There is, however, a serious problem here, which is an inherent problem with LLM’s: The Model is producing content that is inconsistent with the game world.

In the examples above, The NPC talks about a “hidden bunker” beneath the old military base, but even though the military base does exist in the world (And is nearby) it doesn’t have a bunker beneath it. The NPC also mentions “mutants” when there are currently no mutants in the game.

These issues could be partially resolved using a strong prompt (Instructing the AI to only mention named enemy types, etc), but due to the nature of LLM’s the AI is quite likely to produce inconsistent content quite a lot of the time.

Not only that, but even when the content is consistent there will be difficulties in tying it back into the game itself.

The LLM response is a simple text string, how could it be reliably parsed and used to call a function, or modify the game state?

It could be possible to instruct the AI to use certain named tokens in its response, which could then be used to call functions, but this would be unreliable, especially for smaller models (or higher quantisations).

To my knowledge, there are currently no established solutions or best practices for integrating LLM’s into games, and this is certainly an area that I want to experiment with further.

Integration with Spiritus Astrum

Currently, I am looking at three options for integrating LLMs into my project, Spiritus Astrum. These solutions could be used in combination.

The first is to train my own AI model, which would be specifically designed for my game. This would likely be a 7 or an 8b model, or possibly, at most, a 13b. This would probably be the best solution, since it would allow me to train the AI on game-specific data: Characters, Backstory, Enemies, Factions, etc.

I could also simply allow the user to download their own custom model from, for example, hugging face, and connect that locally to Spiritus Astrum. This would allow the user to use any model they wanted, depending on their hardware (But this could cause issues if the user chose an “Uncensored” model, etc).

Finally, The user could download a model and run it from a local server, connecting to it from Spiritus Astrum. Of course, very few players will have access to such a server, this option would be mainly for AI enthusiasts or developers, etc, but since I am working on adding this feature for my own use, there is no reason why I can’t expose it for games to use too.

I could also, potentially, add support for cloud-based AI services, such as chatGPT, but I am unsure of the legal status of using these in a game. In addition, these services could become unavailable if their terms of service change, which is quite likely in the current rapidly changing climate.

At the moment, due to the politicised nature of the AI debate, if I do include Generative AI in my project (Which I may not), it will be turned off by default (Ie, Opt-in, rather than Opt-out).

I will also probably not be shipping any models with the Game itself.

This means that any player who wants to use LLM support will need to, first of all, manually enable the feature in the options, and second of all, manually download a model to use.

The game will be fully playable and functional without LLM support, the LLM feature will simply be an extra feature that can be used to enhance immersion and role playing.

Conclusion

I believe, just like social media, the internet, and all great technological advancements, that AI will become a part of our lives, for better or worse.

Those gamers that boycott AI-driven games may, in years to come, find themselves with very few games to play.

I remember a time when a company having a social media or even an online presence was a novelty, today, it is essentially required.

Generative AI has already proven itself to be frighteningly power in its capabilities, and I am certain that we are only seeing the very tip of the iceberg.

However, despite the uncertainty that an AI-driven world will create, the potential for enhancing and enriching the gaming space is, in my opinion, an exciting and positive one.

–