Auto loaded correct LlamaSharp backend for WPF app

[AshD]

Great work on LlamaSharp. The new version was easy to integrate with and works great with the llama-2-7b-chat.Q4_K_M.gguf model.

I am writing a WPF app and my dev machine has CUDA installed. I used the CUDA backed and it works great, uses the GPU fine.

I added the CPU backend because I am not sure, if the other client computers would have CUDA installed or even have a NVidia GPU. When doing so, the speed dropped and it was not using the GPU.

Is it possible to do CUDA detection and chose the CUDA backend over the CPU backend?
Something like this, with CUDA checking?
#65

An Auto detection Universal Backend package that contains all the DLLs and picks the correct one will work great for clients apps where the developer cannot control the machine specs.

Thanks,
Ash

[martindevans]

The way #65 is set up at the moment is it will preferentially use the CUDA binaries if they exist. This is actually an unresolved problem with that PR - if you have the CUDA binaries but don't have CUDA it will fail at runtime.

I'm not personally using CUDA, so I haven't done mucg investigation into it. But if you know of a reliable way to detect CUDA capability and choose the right binaries I'd be very interested to hear about it and integrate into that PR. As you say a single backend package would be great!

[saddam213]

I think detecting CUDA is going to be a lot more difficualt due to the edgecases that exist

The main problem I see is that just because you have CUDA installed that does not mean you have a GPU or compatable GPU

You can also have both toolsets installed, CUDA 11 & 12, which one do we target

[martindevans]

Yeah that would be my worry, there's a lot of difficult cases to consider here.

Do you think we could sidestep the whole problem by just loading up the CUDA binaries and then if it fails at runtime we unload them and try again with CPU binaries?

[saddam213]

Just wondering if this is even something that can be solved at this level without dragging in some code to call a cuda function, kinda like how we do an empty call to llama.cpp.

Even if we just provide a way to dynamically load the dll for the user, so apps can at least add some sort of selector, e.g: select CPU, CUDA11 or CUDA12 via the App/UI then dynamically load it that way?

This way it will always default to CPU but end users can easily have the option to select GPU?

Edit: we could also fix that static constructer that makes the numa flag impossible to set at the same time :p

Tricky one this

[AshD]

Agreed. The main problem with CUDA, is that not installed by default with the GPU driver installed.
For example, on my friend's new laptop with a 3070, it was not installed and had to be manually installed.
I wish Llama.cpp had a DirectML backed, it would work for my windows app.

One way to check the CUDA version is below, but it would need some way to fallback if something goes wrong as Martin says.

using System;
using System.Linq;

class Program
{
    static void Main()
    {
        string cudaPath = Environment.GetEnvironmentVariable("CUDA_PATH");
        if (!string.IsNullOrEmpty(cudaPath))
        {
            // Split the path into parts
            string[] pathParts = cudaPath.Split('\\');
            
            // Find the part that starts with 'v', which usually indicates the version
            string versionPart = pathParts.FirstOrDefault(part => part.StartsWith("v"));
            
            if (versionPart != null)
            {
                // Remove the 'v' to get the actual version number
                string version = versionPart.Substring(1);
                Console.WriteLine($"CUDA is installed at {cudaPath}.");
                Console.WriteLine($"CUDA version: {version}");
            }
            else
            {
                Console.WriteLine($"CUDA is installed at {cudaPath}, but the version could not be determined.");
            }
        }
        else
        {
            Console.WriteLine("CUDA is not installed.");
        }
    }
}

[saddam213]

there is also the annoying edge case of users who dont want to use GPU the have installed at all.

Is this ultimately an end user choice?

[AshD]

From a Benchmarking perspective for the llama-2-7b-chat.Q4_K_M.gguf model:
CPU: Core i9 - 13 Gen
GPU: RTX 4090

CPU Backend: ~8 tokens per second
GPU Backend: ~39 tokens per second with all 35 layers on the GPU

So, there is a big difference in performance.

[AshD]

there is also the annoying edge case of users who dont want to use GPU the have installed at all.

Is this ultimately an end user choice?

Can you explain this use case? I get annoyed when an app does not use my GPU or CPU to the full extent and make me wait :-)

[saddam213]

Its not really a question of performance, that speaks for itself, it how far do we go on deciding what hardware we are going to use without asking the user

[martindevans]

We could support both user choice and autmatic loading. For example:

LLamaNative.Configure(cuda_enabled = true, numa = false);

NativeApi.Whatever(); // native calls work as normal

Then we'd just have the static constructor in NativeApi inspect the user configuration when loading and choose binaries appropriately.

Edit: And if there's no explicit Configure call we just do whatever seems best (use the highest CUDA version available, fallback to CPU if that fails).

[AshD]

Thanks Martin. That would work for me.

[saddam213]

@martindevans this does seem like the best way to go

[SanftMonster]

It may be a common problem than just on WPF. If one would like to distribute a package dependent on LLamaSharp, he has to deal with both users with CUDA and without CUDA. Therefore a detection seems to be important.

As a trick I think the following steps may be a solution:

1. Place DLL with and without CUDA in a folder of your project.
2. Detect the cuda version of the user's system.
3. Copy the corresponding DLL to the same directory with your executable file.

The following code for example, could detect the cuda version (generated by chatgpt):

using System;
using System.Diagnostics;

class Program
{
    static void Main()
    {
        string command = "nvcc --version";
        string output = RunCommand(command);
        string cudaVersion = ParseCudaVersion(output);
        Console.WriteLine("CUDA Version: " + cudaVersion);
    }

    static string RunCommand(string command)
    {
        Process process = new Process();
        process.StartInfo.FileName = "cmd.exe";
        process.StartInfo.Arguments = "/c " + command;
        process.StartInfo.RedirectStandardOutput = true;
        process.StartInfo.UseShellExecute = false;
        process.StartInfo.CreateNoWindow = true;
        process.Start();
        string output = process.StandardOutput.ReadToEnd();
        process.WaitForExit();
        return output;
    }

    static string ParseCudaVersion(string output)
    {
        string version = "Unknown";
        int index = output.IndexOf("release ");
        if (index >= 0)
        {
            int startIndex = index + 8;
            int endIndex = output.IndexOf(",", startIndex);
            if (endIndex >= startIndex)
            {
                version = output.Substring(startIndex, endIndex - startIndex);
            }
        }
        return version;
    }
}

In practices, the cases will be more complex, including avx2, avx512 and support for other platforms. The code above is only a quick work around. As for the implementation in LLamaSharp, it seems still a long way to go (but at least it's partially supported thanks to Martin's work). 🤗

[AshD]

I modified Martin's work to get our Fusion Quill app that uses LlamaSharp into the Microsoft Windows Store.

I downloaded the code before the Llama 0.6 release but after Martin's TryLoadLibrary code checkin.

I had to modify it to put the cpu dll in a different folder. My Modification is in bold.

    private static IntPtr TryLoadLibrary()
    {

#if NET6_0_OR_GREATER

        if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
        {
            // All of the Windows libraries, in order of preference
            return TryLoad("cu12.1.0/libllama.dll")
                ?? TryLoad("cu11.7.1/libllama.dll")
                ?? TryLoad("**cpu/**libllama.dll")
                ?? IntPtr.Zero;
        }

Also to deploy it in the Microsoft Store, I had to change it to tell it what the absolute path was. Changes are in bold.

static IntPtr? TryLoad(string path, bool supported = true)
{
if (!supported)
return null;

if (NativeLibrary.TryLoad(**Path.Combine(AppDomain.CurrentDomain.BaseDirectory, path),** out var handle))
    return handle;

return null;

}

LlamaSharp made the process of integrating with Llama.cpp real easy. Thanks guys!

Thanks,
Ash