I have recently been working with WAN 2.2 for AI video generation in comfyui.

One of the workflows that I experimented with was the Phantom Wan 14B Workflow that is included in comfyui as a default template.

However, I had two issues with this workflow.

Firstly, when the “base_precision” is set to the default of “fp16_fast” I get the following error:

WanVideoModelLoader

torch.backends.cuda.matmul.allow_fp16_accumulation is not available in this version of torch, requires torch 2.7.0.dev2025 02 26 nightly minimum currently

This can be easily fixed by setting the base_precision to “fp16”

However, I then ran into another issue, which took several hours to solve. This issue was coming from the WanVideo Sampler:

WanVideoSampler

must be called with a dataclass type or instance

After updating comfyui UI, Cuda, and verifying that my pytorch versions were compatible, I was eventually (With the help of chatGPT) able to figure out that the issue was with the “WanVideo Torch Compile Settings” node included in the default workflow.

When I bypassed this, the Phantom 14B Workflow worked fine.

I am making this post in the hopes that this might help save someone else some time, I couldn’t find any users with a similar issue online.

The post WAN Video And Phantom 14B in ComfyUI Issues first appeared on Phoenix Game Development.

I have created another extension for SillyTavern.

This extension uses the “OpenMeteo” open source weather API.

I have previously tried the “Accuweather” plugin for SillyTavern, but this didn’t work for me, it did not return any results. I did manage to get the WebSearch plugin working, and this can be used to generate weather reports/forecasts in SillyTavern, but the formatting is inconsistent, and I found this to be a poor solution.

My solution using OpenMeteo works well and produces easily understood and correctly formatted output, in the following form:

The following content is weather data starting from today:

Coordinates: <REMOVED>
Timezone: Europe/London GMT+1

12/10/25 Overcast
13/10/25 Light Drizzle
14/10/25 Overcast
15/10/25 Overcast
16/10/25 Overcast
17/10/25 Overcast
18/10/25 Light Drizzle
19/10/25 Light Drizzle
20/10/25 Overcast
21/10/25 Light Drizzle
22/10/25 Light Drizzle
23/10/25 Light Drizzle
24/10/25 Overcast
25/10/25 Moderate Drizzle
26/10/25 Overcast
27/10/25 Moderate Drizzle
[WEATHERDATA]

Currently, the extension only displays a 16 day weather forecast with a human-readable weather code, as shown above, however it is very easy to improve this to display any desired data. OpenMeteo has a very large amount of data available, including rainfall, temperature, wind speed, sunset/sunrise times, etc, etc.

I haven’t added this extension to github yet, so I will just post the code here. This code is based on my earlier Email Checker extension.

You will need to do an NPM install for openmeteo to make this code work.

You will also need to enter your latitude and longitude in the “const params” section of the server script below.

In addition,

On the server, add this at the top of index.js:

const { fetchWeatherApi } = require('openmeteo');

Then, to access the API:

async function fetchweather(ws)
{

const params = {
	"latitude": <ENTER LATITUDE HERE>
	"longitude": <ENTER LONGITUDE HERE>
	"daily": "weather_code",
	"timezone": "Europe/London",
	"forecast_days": 16,
};
const url = "https://api.open-meteo.com/v1/forecast";
const responses = await fetchWeatherApi(url, params);
  
// Process first location. Add a for-loop for multiple locations or weather models
const response = responses[0];

// Attributes for timezone and location
const latitude = response.latitude();
const longitude = response.longitude();
const elevation = response.elevation();
const timezone = response.timezone();
const timezoneAbbreviation = response.timezoneAbbreviation();
const utcOffsetSeconds = response.utcOffsetSeconds();

var retdta = "";

console.log(
	`\nCoordinates: ${latitude}°N ${longitude}°E`,
	`\nElevation: ${elevation}m asl`,
	`\nTimezone: ${timezone} ${timezoneAbbreviation}`,
	`\nTimezone difference to GMT+0: ${utcOffsetSeconds}s`,
);

retdta += 
	`\nCoordinates: ${latitude}°N ${longitude}°E` +	
	`\nTimezone: ${timezone} ${timezoneAbbreviation}` + "\n\n";
	
const daily = response.daily();

// Setup
const startTime = Number(daily.time());
const endTime = Number(daily.timeEnd());
const interval = daily.interval();
const weatherCodes = daily.variables(0).valuesArray();

const timeArray = [];

for (let i = 0, timestamp = startTime; timestamp < endTime; i++, timestamp += interval) {
	const date = new Date((timestamp + utcOffsetSeconds) * 1000);

	// Format as DD/MM/YY
	const day = String(date.getDate()).padStart(2, '0');
	const month = String(date.getMonth() + 1).padStart(2, '0'); // Months are 0-indexed
	const year = String(date.getFullYear()).slice(-2); // Get last two digits
	
	const weatherCode = parseInt(weatherCodes[i], 10);
	const weathercodetext = getweathercodetext(weatherCode);

	// Combine with weather code
	console.log(day + "/" + month + "/" + year + " " + weatherCode + " " + weathercodetext);
	retdta += day + "/" + month + "/" + year + " " + weathercodetext + "\n";
	timeArray.push(`${day}/${month}/${year} - ${weatherCode}`)
}

retdta+= "[WEATHERDATA]";
ws.send(retdta);

const weatherData = {
	daily: {
		time: timeArray,
		weather_code: weatherCodes,
	},
};

}

function getweathercodetext(weathercode)
{
var ret = "";
 switch(weathercode) {
        case 0:
           ret = "Fair";//"Clear Sky";
            break;

        case 1:
           ret = "Mainly Clear";
            break;

                case 2:
           ret = "Partly Cloudy";
            break;  
 
                case 3:
           ret = "Overcast";
            break;  

                case 45:
           ret = "Fog";
            break;  

                case 48:
           ret = "Depositing Rime Fog";
            break;  

               case 51:
           ret = "Light Drizzle";
            break;  

              case 53:
           ret = "Moderate Drizzle";
            break;  

       case 55:
           ret = "Dense Drizzle";
            break;  

       case 56:
           ret = "Light Freezing Drizzle";
            break;  

       case 57:
           ret = "Dense Freezing Drizzle";
            break;  

     case 61:
           ret = "Light Rain";
            break;  

     case 63:
           ret = "Moderate Rain";
            break;  

    case 65:
           ret = "Heavy Rain";
            break; 

    case 66:
           ret = "Light Freezing Rain";
            break; 

    case 67:
           ret = "Heavy Freezing Rain";
            break; 

    case 71:
           ret = "Slight Snow Fall";
            break; 

    case 73:
           ret = "Moderate Snow Fall";
            break; 

    case 75:
           ret = "Heavy Snow Fall";
            break; 

    case 77:
           ret = "Snow Grains";
            break; 

    case 80:
           ret = "Slight Rain Showers";
            break; 

   case 81:
           ret = "Moderate Rain Showers";
            break; 

   case 82:
           ret = "Violent Rain Showers";
            break; 

   case 85:
           ret = "Slight Snow Showers";
            break; 

   case 86:
           ret = "Heavy Rain Showers";
            break; 

   case 95:
           ret = "Slight or Moderate Thunderstorm";
            break; 

   case 96:
           ret = "Thunderstorm with Slight Hail";
            break; 

   case 99:
           ret = "Thunderstorm with Heavy Hail";
            break; 

        default:
        ret = "INVALID WEATHER CODE: " + weathercode;
           break;
    }
    console.log("getweathercodetext: " + weathercode + " "  + ret);
return ret;
}

Then, on the client, use this function to display the weather data:

async function printweatherdata(txt){
	   const context = getContext();
	
	   if(txt == "STARTUPMESSAGE")
	  	 return;	   

   console.log("printweatherdata:" + txt);
   context.executeSlashCommands('/reasoning-set The following content is weather data starting from today:\n ' + txt + ' ').pipe
}

The post SillyTavern Extension: Weather Checker first appeared on Phoenix Game Development.

I have completed the first release of my SillyTaven Extension.

The purpose of the extension is simply to connect to a users email inbox and retrieve the most recent number of emails specified by the user.

I am not hugely experienced in nodeJS/javascript, or with SillyTavern extensions, so the code might be a little rough, but I didn’t see any other extension out there that does this, and I needed this functionality for my own purposes, so I thought I would release it to the community as well.

I intend to add more features in the future, such as the ability to search for specific emails, (By sender, date, search string, etc).

The extension implements a simple websocket server, and uses imapflow to get the emails.

Due to how github works, I had to upload the client and server folders separately.

The server folder is HERE, and the client folder is HERE.

The readme is below (The readme is also on the github page).

–

ST-EmailChecker is a simple client/server extension for SillyTavern that allows an LLM to connect to a user’s inbox and download the most recent emails to the LLM.

The emails are stripped of HTML and hyperlinks (Only the message body is returned) and the emails are added to the LLM as “reasoning” (Using reasoning-set). This prevents large blocks of text from spamming the chat window.

The purpose of this extension is to allow an LLM to function as a digital assistant/secretary, checking for important emails, drafting replies, summarising, etc, etc.

In the future, I may add the ability to search for specific strings in emails, or for emails from specific people, search for only unread emails, etc.

If the emails appear to be truncated in SillyTavern, make sure that your “Response (Tokens)” is set high enough in AI response configuration.

Usage:

Download the code (You will need both the SERVER and CLIENT repos).

You should have two folders:

emailcheckerserver_git
emailcheckerclient_git

Place BOTH of these folders in your SillyTavern/data/default-user/extensions/ directory.

Open the folder “emailcheckerserver_git”.

At the top of the script, you will need to set the hostname, username, and password for your email account.

Now, run:

Node index.js in a command window.

You may need to run:

npm install imapflow

When successful, you should see:

EmailReader: Server Module Loaded Server started on port 3000

Finally, in “emailcheckerclient_git”, set the IP address of your server at the top of the script.

If you are running sillytavenr locally, you do not need to do anything.

However, if you are connecting to a LAN server, enter the IP of the server in this script (Without the port).

The client will show up as an extension in sillytavern as “Email Checker”.

To use, simply click on the down arrow next to “Email Checker”, set the number of emails you wish to return, and click “Run Email Check”.

For any questions or comments, I can be contacted at:

contact@NOSPAM.phoenixgamedevelopment.com

The post SillyTavern Extension: Email Checker first appeared on Phoenix Game Development.

I have spent about a day or so working on writing code with DeepSeek-V3-0324 (I am running the model locally. For information on how I setup this model, check out my post HERE).

This is actually my first time experimenting with AI generated code, and the results were, in a word, spectacular.

With very minimal prompting, the AI was able to output quite complex and complete code, with very few issues or bugs.

I used the AI to generate a C# Forms application to display the Mandlebrot set, display a rotating 3D cube on the screen (A Tesseract), display a demo of Conway’s “Game Of Life” and it’s “Brians Brain” modification, as well as write a Base64-to-image converter.

All of these programs worked very well, and where issues occurred, I was able to quickly resolve them using plain english.

This is an example of the output for the first program, to display the Mandlebrot set. The query was:

“I would like you to write me a program in C# that displays the mandlebrot fractal set. The program should be as concise as possible, and should ideally use no external libraries, however it should display the fractal in colour on a gui.”

Output:

“I understand the requirements – a concise Mandelbrot fractal generator in C# using only built-in libraries while still producing a colored GUI output. This is an interesting challenge given the constraints.”

“Here’s a solution using Windows Forms (System.Drawing) which comes bundled with .NET, requiring no external dependencies:”

using System;
using System.Drawing;
using System.Windows.Forms;

class Mandelbrot : Form
{
    const int MaxIterations = 1000;
    const double Zoom = 300;

    public Mandelbrot()
    {
        Text = "Mandelbrot Set";
        ClientSize = new Size(800, 600);
        DoubleBuffered = true;

        Paint += (sender, e) =>
        {
            for (int y = 0; y < Height; y++)
            {
                for (int x = 0; x < Width; x++)
                {
                    double zx = 0, zy = 0;
                    double cx = (x - Width/2) / Zoom;
                    double cy = (y - Height/2) / Zoom;
                    int iter = MaxIterations;

                    while (zx*zx + zy*zy < 4 && iter > 0)
                    {
                        double tmp = zx*zx - zy*zy + cx;
                        zy = 2*zx*zy + cy;
                        zx = tmp;
                        iter--;
                    }

                    Color c = iter == 0 ? Color.Black : Color.FromArgb(
                        iter % 8 * 32,
                        iter % 16 * 16,
                        iter % 32 * 8);
                    e.Graphics.FillRectangle(new SolidBrush(c), x, y, 1, 1);
                }
            }
        };
    }

    static void Main()
    {
        Application.Run(new Mandelbrot());
    }
}

“This implementation:

1. Uses Windows Forms for the GUI
2. Implements the Mandelbrot algorithm directly
3. Colors points based on iteration count
4. Is completely self-contained (no NuGet packages needed)
5. Compiles to ~100KB executable
6. The zoom factor can be adjusted for different views”

“Would you like me to explain any particular part in more detail? Or perhaps suggest optimizations or modifications?” 

I then asked the AI for detailed instructions for how to run the code:

“Excellent, thank you. How can I run this code, do I just create a new blank project in visual studio?”

And the response that I get was equally intelligent:

“Yes! You have the right idea. Here’s a precise step-by-step guide:”

1. Open Visual Studio
2. Create new project → “Windows Forms App (.NET Framework)” (Important: not .NET Core for this example)
3. Replace ALL code in Form1.cs with the Mandelbrot code I provided
4. Click the Start/Run button (or press F5)

“Key technical notes:

• The code uses .NET Framework’s System.Drawing which isn’t available in .NET Core by default
• The project template must be Windows Forms, not Console or WPF
• No NuGet packages need to be installed
• Compilation happens automatically when you run”

“Would you like me to modify it for .NET Core if that’s your preference? Though we’d need to add a NuGet package for System.Drawing in that case. Or perhaps you’d enjoy seeing a real-time zoomable version?”

I needed to make some changes to the code in order to get it to run properly, and when I posted my finished, working, version of the code back to the AI, it was even able to suggest improvements:

“I see what you’ve done! You’ve:

1. Created a proper Windows Forms project structure
2. Wrapped my Mandelbrot class inside the Form1 container
3. Commented out InitializeComponent() since we don’t need designer-generated code
4. Initiated the Mandelbrot form directly from Form1’s constructor”

“Your changes are quite logical. Though may I suggest two slight optimizations to your version?”

// Instead of Application.Run inside Form1's constructor,
// modify Program.cs like this:
static class Program
{
    static void Main()
    {
        Application.Run(new Mandelbrot()); // Run Mandelbrot directly
    }
}

// And make Form1 empty or remove it entirely

“This would:

1. Eliminate the unnecessary Form1 middleman
2. Follow Windows Forms conventions more closely
3. Prevent potential double-form issues
4. Make the project structure cleaner”

“By the way, I notice you kept all the default using directives – would you like me to strip those down to just what we actually need? We could reduce it to just System, System.Drawing and System.Windows.Forms for better clarity. Three simple namespaces would suffice!”

This is very useful feedback, and the AI even successfully caught the unneeded “Using” directives, I have a having it leaving those in while debugging, having the AI suggest removing them is exactly what a human programmer would do!

The AI also suggested changes to the programs, such as the “Brians Brain” variant of Conway’s Game of Life, which I hadn’t even heard of! It then either modified the program with those changes in mind.

In one case, the changes became so numerous that the AI offered to completely rewrite the program to avoid confusion, which it duly did!

I am confident that I could have written the above programs myself, just relying on internet research and my own knowledge to help me, however, it would certainly have taken far longer. Working with the AI was a substantial time saver, it really was like having another person there to assist me with questions, even complex techical questions.

I am currently experimenting with using the AI to write a much more complex program, (I want to take the system audio and display an aesthetically pleasing visualisation of it on the screen) and this is going well, but it is a lot slower. I feel that with the more complex program the advantages of using AI to code are not as strong, I find that I can probably fix mistakes faster than waiting on the AI.

I think that this is a very important point to make regarding the use of generative AI in the workplace, and in society. Simple tasks (such as simple programs) can be easily offloaded to the AI, however when it comes to more complex tasks, much more human interaction is needed.

So, in the future, as AI becomes normalised in society, we could simply use AI for the menial tasks, freeing up valuable Human Resources for creative and complex tasks. The idea would be that instead of AI “replacing” humans, it is instead enabling humans to do what they do best: Create, Understand, and Produce, while the AI simple handles the simple boring time-wasting tasks.

I have uploaded some of the examples of the AI-Generated scripts to my github. To access the scripts, click on the links below. The Scripts are written in C# (.net).

Displaying the Mandlebrot Set.

Displaying a rotating Tesseract.

Displaying Conways “Game of Life” (With speed controls and randomisation).

Displaying the “Brian’s Brain” variant of Game of Life (With speed controls and randomisation).

Displaying an image encoded into a Base64 string to the screen.

The post Adventures in AI coding first appeared on Phoenix Game Development.

After hearing good things about this model, I tried, and succeeded, in getting it to run locally.

For those unaware, DeepSeek V3-0324 isn LLM developed by the Chinese Based Company DeepSeek.

The V3 variant is a chat model, as opposed to, for example, R1, which is intended more for reasoning.

The GGUF Q8 version of this model is around 700 GB’s in size, much larger than the 120B (Q8) models that I have been running up to now. However, DeepSeek is a MOE, a Mixture of Experts model, and only has 37 Billion active parameters, meaning that it can be run on systems with less VRAM.

Crudely speaking, MOE’s trade space for compute. IE, MOE’s tend to be much larger than “Dense” models, however, if you can fit them into RAM, they run much faster.

My AI Rig currently has an A6000 Ampere and two 3090s, for a total of 96 GB’s of VRAM, and it has 256 GB’s of DDR4 RAM in 8 Channel mode, and a Threadripper 3975WX CPU.

I am running Unsloths GGUF models, available from HERE.

This gives me a total of 352 GB’s of RAM, which is just enough to run the Q3_K_XL quant of this model.

Q3 is not particularly high, but for a very large model like this, it should be usable.

What follows is my experiences with setting up and evaluating the model.

Firstly, I ensured that both KoboldCPP and SillyTavern were both updated to their latest release (1.93 for Kobold, and 1.13 for SillyTavern). The latest versions of KoboldCPP, in particular, are needed to load MOE models.

The model itself is loaded in the same way as any GGUF model.

The Unsloth GGUF that I downloaded has 62 layers, I was able to offload 17 of them to the GPU’s VRAM, using a tensor split of 0.9,2.0,1.0.

Please not that in later versions of KoboldCPP, the GPU order has changed to the PCIE Bus ID, which, for me, was different to the previous load order (Please see the issue I opened HERE on Github). This meant that I had to change the order of the tensor split.

Once I did this, I noticed that the model was loading successfully, and KoboldCPP was starting, but it was failing with a “CUDA error: out of memory” whenever I would start inference. This was caused, I believe, to the context being too long.

I don’t know if MOE models in general require more RAM for context, or just DeepSeek.

After several days of tinkering, I managed to come up with some settings that worked well for my system.

The main issue for me was the “BLAS Batch Size” under the hardward tab in KoboldCPP. This was set to 512, I changed it to 64.

The BLAS Batch Size has a huge impact on the speed of prompt processing, and this makes and even bigger difference for MOE models due to how they work. A Batch size of 64 makes for VERY slow prompt processing, but once the prompt processing is done, the generation time is very fast.

I wanted to have at least 32k of context, since this is the minimum that I am used to dealing with. I believe deepseek supports up to 128k context, which is huge, but I haven’t verified this.

I also had to set “Quantize KV Cache” to “8-Bit” (Default was FP16) to prevent OOM errors.

I also set the Context to 32k in SillyTavern as well, to match KoboldCpp.

My settings now look like this:

As a result of all of this, I have managed to get DeepSeek V3 0324 Q3_K_XL running on my system with 32k context.

Almost all of my VRAM and System RAM is used up with these settings!

Prompt processing is very slow with the 64 bit batch size, and I am currently looking into reducing the layers being offloaded to the GPU to free up memory, so that I can increase the Batch size to at least 128.

However, the token generation is much faster!

With context set to 2k I was getting 8.51 tokens per second!

2k is obviously too low to be useful, and as I increased the context size, and the context filled up, token generation rates droped to about 4-5 tokens per second, and then, at 32k with some of the context filled, I am getting around 1.5 tokens per second at least, and this is including the slow prompt processing times.

This is not ideal, but it is still faster than what I would get with a dense model.

So, how good is DeepSeek compared to the 120b Dense models that I have been using?

Subjectively, the quality of the writing seems far superior. The Model seems to follow the prompt better, and characters in creative writing exercises seem to have a lot more personality, and feel more unique than other models.

Even though prompt processing is slower with my settings, token generation is much faster,.

I am limited by my available system RAM at the moment (I should have used a Q2 quant, not a Q3 one), and I am considering a RAM upgrade to 512 GB’s.

This presents an interesting question. 512 GB’s of DDR4 RAM (At 3200 MHZ) would cost about the same as anothe Power supply and another 3090 for my current rig, so, which would be better?

For running dense models, the extra 3090 would be much more useful. With 120GB’s of VRAM I could run 120b models at Q8 almost entirely in VRAM (Maybe with a little spilling over into system RAM at high context sizes) at very high speeds. However, for running MOE models, I would be far better with the 512 GB’s of system RAM (I could probably run a Q5 of DeepSeek, with much higher context and without quantising the KV cache. I could also use the GPU’s just for prompt processing, further increasing speed).

So, the question is: Are MOE models generally better than Dense models?

I don’t think there is a hard and fast answer to this.

There are some that argue that MOE models are inferior to dense models in terms of quality, and that they are only popular due to the fact that they can be run on cheaper hardware (RAM and compute is a lot cheaper than VRAM), and that if you can run a large, dense, model, that would be the better option.

However, there are others that might argue that dense models will, or have, hit a wall, and that Mixture of Experts models are a necessary step to continue to improve the quality of models.

I have come across a formula on reddit that seems to provide a rough estimate of the performance of a MOE model compared to a dense model. It is:

sqrt(active parameters * total parameters)

This would seem to indicate that DeepSeek with its 700 Billion total parameters, and 37 Billion active, would be generally equivalent to a 160B dense model. Based on the few days that I have spent working with it, this would seem accurate.

The model feels powerful, and, subjectively, it feels superior to 120b models, but not six times better.

Time will tell where MOE type models become the future or not, but even if they dominate the high end LLM space, Dense models (Including distills of MOE’s) will likely still have a place in the sub-70b space for a long time to come.

The post Taming the Beast: Running DeepSeek V3-0324 Locally first appeared on Phoenix Game Development.

This week was one of the most frustrating that I have had since I first learned to program a computer at the age of 17.

The reason was that this week I discovered the true nature of the Linux Experience: Dependency Hell.

I had heard of this before but did not truly appreciate the nature of this beast.

In short, unlike Windows, which uses compiled executable programs, Linux largely expects users to download source code and compile their own binaries. This is likely due to the sheer number of Linux distros out there, releasing a binary for all of them would be impractical. The problem with this is Linux make extensive use of common libraries, or dependencies, that are shared between various programs, and are, unfortunately, often incompatible.

So, you could have one program which requires at least version 2.0 of a library, and another program that will not run on anything later than 1.0. If you want both programs to run, you are in “Dependency hell”.

Unfortunately, in the AI world, there are a LOT of common libraries (CUDA, PyTorch, Torchvision/audio, xFormers, etc), leading to the issues that I had this week.

It should be noted before I continue that when I built my Linux-Powered AI rig I went into it with significant optimism, I loved the idea of Linux. I can’t stand the bloatware on windows, the tracking, the constant forced updates and restarts, the ads, and the lack of control over the operating system. When I bought my current windows 11 laptop (They didn’t have any with windows 10) I clicked on the start button and saw a new article about Prince Charles’ Coronation. It’s bad enough that I am bombarded with ads on the internet, I do NOT want to see ads when I click on the start menu of my own computer!

The rule is EITHER paid OR ad supported, not both! If your OS costs money, you can’t have ads too! Microsoft haven’t learned this.

I even installed Linux on my current laptop briefly, but the drivers for the trackpad didn’t work, so I reluctantly installed Win11 again.

However, the difficulty that I have had this week just getting programs to run has seriously soured me on the Linux experience.

The issues all started when I tried to install xTTS (A Text to speech extension) for SillyTavern.

I did a Pip Install, I had to update some packages, but everything seemed ok.

Then I realised that when I updated the dependences to allow xTTS to work, I had broken several of my other programs, including comfyUI!

Reinstalling those programs, of course, broke xTTS, because the versions of the software used by each were incompatible.

I eventually learned about Conda and Virtual environments, which solved some of my issues, but not all (Never in my time using windows have I had to install a virtual environment just to get a program to run!).

However, even with Conda/Venv, I was STILL having serious problems just fixing my existing programs, let alone installing new ones. At one point, at least 3-4 of my main AI programs were in a broken state due to dependencies, and I had completely lost faith in the system. It got to the point where I was genuinely ecstatic just to start up a program and have it actually work!

What I learned eventually is that when you run pip install inside a conda environment it installs the packages to the location where pip was installed, which is your systems pip directory. IE, you are installing the packages globally, potentially overwriting other packages.

Using conda install fixes this problem, but most of the packages I was using didn’t have a conda version, so I was forced to spend many hours pip installing dozens of packages to attempt to find a solution.

At one point, I restarted my system, and found that I couldn’t log into the gui! Apparently my hard drive was full, which, for some reason, prevented login?

It turns out that I had used up 75 GIGABYTES of storage page just downloading difference packages! CUDA, for example, is about 2 GB’s. Torch vision is about 1 GB, so every time I do a pip install that’s another 2 GB’s of data.

I purged the pip cache and moved some data from the main drive to the backup drive (Luckily remote desktop still worked, even though the local login didn’t?) and I was able to log in again, but I still had the dependency issues to deal with.

Eventually, after several days of misery, I discovered that you can install pip locally to a conda environment, so that if you then run pip install it will install the packages to the local version of pip, not the system version, allowing you to use pip install without overwriting your system packages.

So far, this seems to work.

Without bragging, I have nearly 25 years experience working with computers. I am new to Linux, yes, but I have extensive experience in programming, the command line, libraries, and the complexities of operating systems. If it takes me three days just to get a program running without breaking another program, then there is a problem with the operating system. It simply should not be that difficult.

Just to clarify, all I am trying to do here is install a program, that’s all. I’m not trying to write code, just run it!

ALL of the above could have been accomplished on windows by just double-clicking on an exe!

There was one program (A web based gui for training voices for Realtime Voice Conversion, or RVC) that I spend several hours trying to resolve the dependencies for. Everything I pip installed something, it broke something else. I had to reinstall comfyui at least seven or either times!

Eventually, I gave up and downloaded a fork of the program that worked on windows. I ran into ONE error (My firewall was blocking the clone of the git repo. I simply disabled it temporarily until the download was completed) and the program was up and running in less than three minutes.

This has consistently been my experience: Fighting with Linux to resolve dependencies that would require just a few clicks on windows.

I know that I have extensive experience on Windows which I lack in Linux, but we are just talking about basic functionality here, you shouldn’t need to be a Linux guru just to get a program to run!

The other discovery that I made is that the AI world doesn’t appear to be quite as Linux-centric as I initially thought. Before I built my AI rig I read that the vast majority of AI-related programs and libraries are designed for Linux, and I would have difficulty running them on windows.

In actual fact, in my experience, the only thing that I have found that was easier on Linux than windows was installing Sage-Attention 2. Sage attention 2 can be installed with just a few lines of code on Linux, it is, apparently, a lot harder on Windows.

Apparently Deepspeed is not properly supported on Windows either.

Everything else either had a windows version, or there was a similar program that worked on windows. In some cases in fact I found more easily accessible information for windows than Linux (Such as Lora training for Hunyuan Video).

It seems to me that a lot of the people singing the praises of Linux are people who are card-carrying members of the Linux club, and either fail to see it’s substantial shortcomings, or obstinately refuse to acknowledge them.

Windows isn’t perfect, I mentioned the issues with bloatware, privacy, etc, but honestly, after dealing with Linux “dependency hell” for the last 3-4 days, I will happily allow Bill Gates to come to my house personally and watch me take a shower.

Running a debloat script on windows 11 is far, far, easier than dealing with Linux’s issues.

Linux is for people who want to work on Linux, not for people who want to work on anything else.

Windows, despite its issues, just works, and when you’re a professional who values their time, you want something that works.

I cannot understand how anyone gets any work done on the OS when you are losing so much time resolving dependencies.

There are some things that are better on Linux, yes, and it has extensive support within the server space and academic/scientific computer space, but you have to balance that against the sheer man-hours wasted doing things that can be done with two clicks on windows.

I was hoping that Linux would eventually become a competitor to Windows and MacOS. That it would be the OS that you could recommend to your mom or pop, an OS free of cost, free of bloat, free of ads and spyware, with all the features that one would need from a modern operating system.

I believed that user-friendly distros like popOS or its parent Ubuntu, or Windows-Like Distros like Mint, would eventually evolve to the point that they can enter the consumer space.

However, having gained some insight into the Linux experience, and the way that Linux functions “under the hood”, I now believe that it is destined to always remain what it is: A niche OS used only by technically minded people who use it either because they have to, or because they have the time or inclination to spend on it.

The post Dependency Hell: The Linux Experience first appeared on Phoenix Game Development.

I have spent most of this week working on video generation with Hunyuan video.

This has been a fascinating exploration into a new and rapidly developing sub-field of the AI industry.

AI video generation looks set to take the world by storm in the coming years!

Hunyuan Video allows for the generation of short clips (Usually less than 10 seconds) from a plain text prompt.

From my testing, longer clips are possible, but the quality drops massively. I have tested generation of clips up to 60 seconds (At 320×240).

My current setup is posted HERE, but in short, I have 2 3090 GPU’s installed.

Hunyuan Video at bf16 is 25.6 GB’s.

Unlike with LLM’s (Large Lanuage Models), Video and Image generation does not have the same ability to split the Model across GPUs, so it is not possible to run Hunyuan at bf 16 on a 24 gb card, so I have been using is at FP8.

What is possible, as I discovered, it to offload the CLIP and VAE data from the first GPU to either the CPU or another GPU, this saves some VRAM, allowing for higher resolutions and/or more frames, but it is not “splitting” the model, just offloading some data.

It is also possible to use GGUF format instead of .safetensors. GGUF files for Hunyuan are available HERE.

Running Hunyuan at BF16 would require 45 GB’s RAM on one card.

I am using ComfyUI to run Hunyuan, and having tried many workflows, my current workflow uses GGUFDisTorchMultiGPU nodes as well as DualCLIPLoaderGGUFMultiGPU and VAELoaderMultiGPU to offload as much data as possible to other cards.

The quality I am getting is quite good.

With my setup, it seems that I can generate 201 frames video at 928×640. At 24 Frames per seconds, that’s about 8 seconds of video. This takes about 22 minutes to generate, with Sage attention 1 and TEACache optimisation enabled.

An example of the quality of content (A short, 3 second video) that I can produce is here:

The prompt for the above was:

“A male warrior with long hair and a battle-scarred face wearing a bulletproof vest, holding a large rifle, stands in a war-torn futuristic city. The camera pans around him, as explosions and gunfire rage.
Masterpiece, best quality, dynamic scene.”

What is worthy of note, other than the good quality, is that Hunyuan video seems to have no issue creating realistic looking firearms. I have not used any other video-based AI models before, but I have used Stable Diffusion to generate images, and many of them are unable to create firearms, likely due to censorship, etc. Without getting too far into the realms of politics, it is interesting that a model created by Tencent, A Chinese company, is actually less censored than many models available in the west.

Here is another example, another 3 second Video:

Another example, this one an 8 second video, using the same prompt, is:

It seems that the longer clip has a slightly lower quality. It is hard to tell, but it seems that the background, especially, on larger clips seems to become less clear and more blurry and indistinct.

If I again take the same prompt, but this time increase the length to 60 seconds (Reducing the quality to 320×240 to allow it to fit in Memory), I get this:

Clearly, the quality here is incredibly low, but I don’t think the reason is just because of the lower resolution, I think the main reason is the clip length. For example, if I generate a 3-second clip (using the same prompt as above) at 320×640, I get this:

Which, despite it’s low resolution, is a much better looking video. It seems then, that regardless of resolution, the length is limited to about 8-10 seconds.

While I have tried various quantisations and workflows, in addition to experimenting with different options, there is still a lot of experimentation to do.

In particular, I would like to experiment with the guidance and steps, in addition to the various VAE Decode settings (Tile size, etc).

From what I have read, running the model at Q8 does not substantially impair the quality.

–

The post Experiments with Hunyuan Video first appeared on Phoenix Game Development.

This week, I have been mostly working on the concept for the new region system in Spiritus Astrum.

This system basically divides worlds into large regions, each with their own tech levels, from primitive, to industrial, to modern, to futuristic.

I have enough artwork at the moment to make this system work, although I intend to expand it with a lot more artwork later.

The region system is basically working now, although it is not 100% done.

The system is quite complex, I can add each ground cover object to multiple regions, with a probability for each. So, an item might be common in an industrial region, and uncommon in a modern region, and not appear at all in other regions.

I have yet to fully test this system, but so far, it looks promising.

Crucially, the region system works persistently, using the seed of the terrainchunk to make sure that the same objects appear every time (and in the same location, etc).

Additonally, I have added LLM support to Spiritus Astrum. This, again, is just a concept so far, but I think there is huge potential here that I intend to look at more in the future, when I get more time.

The LLM support works with both LAN servers, local Kobold CPP servers, and loading models directly within unity (Although this seems much slower for some reason). I am using UNDREAM for the LLM support, as mentioned in my other posts on AI.

I have also started doing some more testing and bug fixes. Some issues have crept into the project with all of the improvements that I have been making. There is an issue with shaders breaking when in combat with an AI that I am having difficulty solving.

The post Spiritus Astrum [P0-B9]: p1474: Towards Early Access first appeared on Phoenix Game Development.

For the purposes of this article I am using the term AI to refer to generative AI, and in particular, I am focusing on LLM’s, or “Large Language Models”. The scope of AI in games obviously extends beyond this, but this will be the focus of this piece.

Introduction

I am currently working on integrating support for Large Language Models into my project, Spiritus Astrum.

While working on the technical aspects of this, I have also been thinking about the greater applications for LLM’s in gaming, and of the developing (And deepening) controversy over the same.

Steam has recently clarified their policy on Generative AI. It is now acceptable to submit a game using generative AI on their platform provided that the developer declares that the game uses generative AI, and that the AI contains “guardrails” that prevent it from generating illegal or offensive content at runtime.

However, this declaration by steam has led to some controversy by gamers: Some gamers (Like some artists) believe that generative AI is “stealing” and that it should be banned.

Others believe that allowing Generative AI will lead to a flood of poor-quality AI generated games that will harm steams reputation as a distribution platform, and even harm the gaming industry as a whole.

I try to avoid politics with my blog and the projects that I work on, but it seems that with the ongoing AI debate I may be unintentionally entering a politically charged sphere.

I understand the concerns over generative AI.

There is no doubt that AI has the potential to do harm. In addition to the risks of “Shovelware” titles as I discussed above, there is also the potential that generative AI models could produce, (either intentionally or accidentally), inappropriate or offensive content. Even a safe, “Censored” AI model can be “Jailbroken” in various ways to produce content that it should not be able to produce.

However, despite this, I am still, generally Pro-AI. I would draw a comparision to something like the internet: There may be negative aspects to the technology (Possibly strongly negative aspects) but its usefulness and benefits have made it such a ubiquitous part of our lives that, for better or worse, it is here to stay.

Potential Applications for Generative AI in gaming

I believe quite strongly that Generative AI has a tremendous potential to enrich games in ways that never would have been possible before, to create a whole new era of dynamic, AI-driven content where every world is rich and engaging, and every play experience is unique.

By harnessing the power of LLM’s, we could see the end of boring NPC’s with canned, repetitive phrases. How many times, for example, have we heard Lydia from Skyrim saying “I am sworn to carry your burdens?” Or the famous “Pam param, pan pan param” song hummed by Witcher 3 NPC’s?

Instead, each NPC could have unique dialog, a unique personality, goals, and a strong presence and sense of agency in the world. This could provide for an incredibly rich experience, making the world feel truly lived in. The player would be a part of a greater world, and not the only reason for that world existing.

It is possible to add some randomness to current NPC dialog using a rule based system, where the AI will respond differently based on the current weather, current location, etc (Kingdom Come Deliverance has different NPC dialog depending on whether the players clothing and armour are clean or not!) but this dialog will still get very repetitive very quickly.

In addition, LLM’s can be used to supplement the pre-written dialog for main characters by providing additional dialog between quests, or after a quest has completed.

Even big budget RPG’s have limits to how much dialog has been written, and when you are in between quests, some of the most dynamic and engaging characters revert to a random selection of canned phrases.

LLM’s can also provide post-game content in the same way. Even when the main story is complete, the player can continue to “live” in the world, experiencing new “content” in a way that was never possible before.

Finally, LLM’s could be wired into the games quest system, enabling a true RPG-like experience.

Generative AI as “Dungeon Master”

This point is where I think the true potential of LLM’s lies. The vast majority of modern RPG’s are inspired in some way from the Dungeons and Dragons style of tabletop RPG’s (And, by extension, the old MUD’s, or “Multi-user Dungeons” from the early days of the internet). However, graphical RPG’s face a hurdle that none have yet to overcome: They all allow you only to select and customise a character, not to truly create one.

True, some games provide extensive character customisation options (Skyrim, Cyberpunk 2077, etc), but the characters backstory is usually based on some variation of class selection (Rogue, Mage, Paladin, etc) and not actually creating a character.

Imagine an RPG where you can truly “create” a character, with a backstory to go with them, using a text prompt? Where the game functions as a “Dungeon Master” for your very own adventure?

A world that dynamically adapts to challenge and encourage the player, a game where every playthough is completely unique?

This is the power that LLM’s have the potential to provide to gaming, and it is a power that cannot be ignored.

Experiments, Examples, and Analysis with Generative AI

I have recently conducted some very simple tests to explore the potential of LLM’s within my game, Spiritus Astrum.

All of the below tests were run with an 8b model, most with Llama 3.1 8B at a Q4 quant. This is, obviously, far smaller than the 70b and 120b models that I have been running on my new AI test rig. However, even though I have added the ability to connect to a KoboldCPP server using unity, very few players have access to this kind of hardware, so I am instead using a model that would be small enough to run on an average gaming computer.

For these tests, the information that the AI was given in the prompt was:

You are a Wasteland wanderer, surviving on a desolate, war-torn, ruined world.

Short Description of the AI’s character.

You are talking to a North Star Starship captain, a combat veteran.

Short Description of the players character. This could be entered by the player during character creation. It could be created completely by the player.

They are carrying a 10mm pistol, a medical kit, and food ration pack.

A list of the items in the players inventory, this can easily be obtained by simply converting the name of each item to a text string.

It is currently daytime.

It is hot and humid.

Time of day, weather, and similar information can be obtained from the games variables.

Nearby there is an old, ruined house, a piece of starship wreckage, and a supply crate. In the distance there is an old radio tower and a military base.

This information can be obtained by doing a radius search and then taking the name of any objects found and adding it to the prompt.

In this way, the prompt can be generated and updated quite easily.

So, lets take a look at a few sample outputs. I still have a lot more testing to do in this area, this is just the beginning.

AI:
Greetings, traveler. I’m doing well, considering our circumstances. It’s not every day one gets to explore the ruins of civilization alongside a seasoned adventurer like yourself. That 10mm pistol looks like it’s seen some action; what stories does it hold? And how about that medical kit? Have you had to use it recently? By the way, would you like something to drink from my ration pack? It’s not much, but it might help quench your thirst in this sweltering heat. What do you say?

This is an excellent response. Not only does it make sense, but the AI has also correctly “realised” that the player is a “seasoned adventurer” and that the Pistol in their inventory has “seen some use” likely because the prompt indicated that the player was a “combat veteran”.

It would be possible to use conventional techniques to implement something like this, but this would, first of all, lead to a LOT of rules being necessary, and second of all if would still lead to generic, repetitive, monotonous phrases.

Furthermore, If I change the player part of the prompt to:

You are talking to an inexperienced starship crew member. This is their first field mission. They are carrying a 10mm pistol, a medical kit, and food ration pack

Then the AI has a quite different response:

I’m doing alright, considering the circumstances. It’s not every day you get to explore a post-apocalyptic wasteland. What about you? First time out in the field, huh? You look a bit overwhelmed. Don’t worry, I’ve been around these parts for a while. If you need any advice or help, just let me know. We can start by checking out that supply crate over there. It might have some useful gear inside. And be careful with that pistol – it’s not as reliable as you might think. Now, what do you say we take a look?

The AI has responded correctly based on the players inexperience, and even warned them not to allow their pistol to discharge accidentally! This is a subtly different interaction than before.

This type of variation would not be possible with a rule based system, even a complex one.

A rule based system (or any other “conventional” AI dialog system would also not allow the AI to alter their response based on other factors, such as weather (Sweltering heat), enemies being present or not present, or any of the multitude of other factors and variables that would be present in a computer game.

The only thing that the AI got wrong was in relation to the ration pack. The AI is offering the player a chance to drink from their (The AI’s) ration pack, even though it is actually the player that is carrying the item.

Another example:

AI:
Hello, How are you? I’m doing alright, considering the circumstances. It’s not every day you get to meet someone from the stars. What brings you to this godforsaken place? And what’s with all the gear? Are you planning on sticking around for a while?

USER:
I’m hoping to find a ship, to get off this planet, do you know anywhere I can find one?

AI:
A ship, huh? That’s a tall order. Most of them got destroyed in the Great War or scavenged for parts by now. But I might have some info that could help you out. There’s a rumor going around about a hidden bunker beneath the old military base over there. Supposedly, it’s filled with tech and supplies, including maybe even a functional spacecraft. But be warned, the base is crawling with raiders and mutants. You’ll need to be careful if you’re thinking of checking it out. As for me, I’ve been surviving here for years, so I guess you could say I’m sticking around for a while. What about you? What’s your story? How did you end up here?

This is a another very good response. Not only is quality of the AI’s dialog quite good (despite being completely procedurally generated!) but the AI “knows” that there is a military base nearby, and is recommending that the player go there to explore!

Further Potential Applications

Even these few examples should be enough to illustrate just how powerful Generative AI can be for the gaming industry. Imagine a world where the interactions with NPCs and characters subtly shifts based on the players actions.

Instead of simply having a “moral choice” system which grants the player a “Good Ending” or a “Bad ending”, the players actions could influence the way that the entire game world reacts to the player.

The player could choose who they are and how they want to interact with the world completely at their whim, and the world would respond to that, in totally unique ways every time.

I have not even mentioned the potential for Multimodal models with “Vision” support, such as (LLAva: Large Language and Vision Assistant).

These models have the ability to “see” and image and answer questions about it, describe it, provide a caption, etc.

Within games, it could be possible to send an image of the scene to the LLM at runtime and allow the AI to respond based on what is going on in the world, even without any variables or prompts at all.

I have not experimented much with vision based models, but the potential is certainly there.

Generative AI as the future of Gaming

It used to be the case that graphical improvements drove the gaming industry forward: Every few months it seemed gaming would surpass itself with graphical fidelity.

While this is still happening, I believe the pace has slowed considerably. If you look at games released in the early 2000’s even a period of a year or two would result in noticeable improvements in graphical quality, today, most games made in the last 5 years or so show only modest incremental improvements year on year.

I believe that if the gaming industry is going to continue to innovate, that innovation is going to have to come from somewhere other than graphics, and that place could very well be AI.

AI could be the breath of fresh air that the industry needs to reach entirely new heights. In a few short years we may see an entirely new dawn, offering truly immersive, player-centric stories that are as unique as the players themselves.

The Fatal Flaw

There is, however, a serious problem here, which is an inherent problem with LLM’s: The Model is producing content that is inconsistent with the game world.

In the examples above, The NPC talks about a “hidden bunker” beneath the old military base, but even though the military base does exist in the world (And is nearby) it doesn’t have a bunker beneath it. The NPC also mentions “mutants” when there are currently no mutants in the game.

These issues could be partially resolved using a strong prompt (Instructing the AI to only mention named enemy types, etc), but due to the nature of LLM’s the AI is quite likely to produce inconsistent content quite a lot of the time.

Not only that, but even when the content is consistent there will be difficulties in tying it back into the game itself.

The LLM response is a simple text string, how could it be reliably parsed and used to call a function, or modify the game state?

It could be possible to instruct the AI to use certain named tokens in its response, which could then be used to call functions, but this would be unreliable, especially for smaller models (or higher quantisations).

To my knowledge, there are currently no established solutions or best practices for integrating LLM’s into games, and this is certainly an area that I want to experiment with further.

Integration with Spiritus Astrum

Currently, I am looking at three options for integrating LLMs into my project, Spiritus Astrum. These solutions could be used in combination.

The first is to train my own AI model, which would be specifically designed for my game. This would likely be a 7 or an 8b model, or possibly, at most, a 13b. This would probably be the best solution, since it would allow me to train the AI on game-specific data: Characters, Backstory, Enemies, Factions, etc.

I could also simply allow the user to download their own custom model from, for example, hugging face, and connect that locally to Spiritus Astrum. This would allow the user to use any model they wanted, depending on their hardware (But this could cause issues if the user chose an “Uncensored” model, etc).

Finally, The user could download a model and run it from a local server, connecting to it from Spiritus Astrum. Of course, very few players will have access to such a server, this option would be mainly for AI enthusiasts or developers, etc, but since I am working on adding this feature for my own use, there is no reason why I can’t expose it for games to use too.

I could also, potentially, add support for cloud-based AI services, such as chatGPT, but I am unsure of the legal status of using these in a game. In addition, these services could become unavailable if their terms of service change, which is quite likely in the current rapidly changing climate.

At the moment, due to the politicised nature of the AI debate, if I do include Generative AI in my project (Which I may not), it will be turned off by default (Ie, Opt-in, rather than Opt-out).

I will also probably not be shipping any models with the Game itself.

This means that any player who wants to use LLM support will need to, first of all, manually enable the feature in the options, and second of all, manually download a model to use.

The game will be fully playable and functional without LLM support, the LLM feature will simply be an extra feature that can be used to enhance immersion and role playing.

Conclusion

I believe, just like social media, the internet, and all great technological advancements, that AI will become a part of our lives, for better or worse.

Those gamers that boycott AI-driven games may, in years to come, find themselves with very few games to play.

I remember a time when a company having a social media or even an online presence was a novelty, today, it is essentially required.

Generative AI has already proven itself to be frighteningly power in its capabilities, and I am certain that we are only seeing the very tip of the iceberg.

However, despite the uncertainty that an AI-driven world will create, the potential for enhancing and enriching the gaming space is, in my opinion, an exciting and positive one.

–

The post On the applications for Large Language Models in Gaming first appeared on Phoenix Game Development.

I have been mostly working on creating some new art objects for the game. I will have these done relatively quickly. I then intend to improve the exploration and story elements of the game.

I have also been working on improving how prefabs are stored and organised in the game, which should make it easier to reuse them later with other art objects.

I have also been working on integrating KoboldCPP support into the UDream LLM asset.

I believe I have solved the problem, it was really just a minor formatting issue caused by the data coming from KoboldCPP not being in the same format as what UDream was expecting.

I have also added a chat-style interface to SpiritusAstrum that will be used for the LLM system when it is integrated. It is very basic at the moment, but it allows for scrolling, and multiple messages to be viewed at once, as well as basic colour coding.

I have also integrated it into the existing dialog systems.

I will probably finish the modelling next week. I still have a few objects to go, and I need to adjust the scale, etc, to make sure everything is consistent.

The post Spiritus Astrum [P0-B9]: p1471: Towards Early Access first appeared on Phoenix Game Development.