Skip to content

Add LLamaSharp.Backend.Vulkan #2 #514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Add LLamaSharp.Backend.Vulkan #2 #514

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1c4052c
More backends WithVulkan true by default (#7)
moozoo64 Feb 18, 2024
03ccf95
More backends (#8)
moozoo64 Feb 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 85 additions & 3 deletions .github/workflows/compile.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,85 @@ jobs:
./build/libllama.so
# ./build/libclblast.so
name: llama-bin-linux-clblast-x64.so


compile-vulkan:
name: Compile (vulkan) - ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [
ubuntu-22.04,
windows-latest
]
env:
OPENBLAS_VERSION: 0.3.23
OPENCL_VERSION: 2023.04.17
CLBLAST_VERSION: 1.6.0
VULKAN_VERSION: 1.3.261.1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are all of the defines required? It looks like only VULKAN_VERSION is used?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are all of the defines required? It looks like only VULKAN_VERSION is used?

Well no... I copied that from "compile-clblast:"....
It just needs VULKAN_VERSION:
Your compile.yml is inverted compared to ggerganov/llama.cpp build.yml. Which builds a lot of the backends together per os.
I think this way will be a lot more verbose when adding all of the backends. eg kcompute, openBlas

runs-on: ${{ matrix.os }}
steps:
- name: Clone
id: checkout
uses: actions/checkout@v4
with:
repository: ggerganov/llama.cpp

- name: Download dependencies - Linux
if: ${{ matrix.os == 'ubuntu-22.04' }}
run: |
wget -qO- https://packages.lunarg.com/lunarg-signing-key-pub.asc | sudo tee /etc/apt/trusted.gpg.d/lunarg.asc
sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-jammy.list http://packages.lunarg.com/vulkan/lunarg-vulkan-jammy.list
sudo apt update
sudo apt install vulkan-sdk

- name: Download dependencies - Windows
id: get_vulkan
if: ${{ matrix.os == 'windows-latest' }}
run: |
curl.exe -o $env:RUNNER_TEMP/VulkanSDK-Installer.exe -L "https://sdk.lunarg.com/sdk/download/${env:VULKAN_VERSION}/windows/VulkanSDK-${env:VULKAN_VERSION}-Installer.exe"
& "$env:RUNNER_TEMP\VulkanSDK-Installer.exe" --accept-licenses --default-answer --confirm-command install
Add-Content $env:GITHUB_ENV "VULKAN_SDK=C:\VulkanSDK\${env:VULKAN_VERSION}"
Add-Content $env:GITHUB_PATH "C:\VulkanSDK\${env:VULKAN_VERSION}\bin"

- name: Build
id: cmake_build
if: ${{ matrix.os == 'windows-latest' }}
run: |
mkdir build
cd build
cmake .. ${{ env.COMMON_DEFINE }} -DLLAMA_NATIVE=OFF -DLLAMA_BUILD_SERVER=ON -DLLAMA_VULKAN=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_PREFIX_PATH="$env:RUNNER_TEMP/vulkan"
cmake --build . --config Release -j ${env:NUMBER_OF_PROCESSORS}
#copy $env:RUNNER_TEMP/clblast/lib/clblast.dll .\bin\Release\clblast.dll
# # We should probably generate a sha256 sum in a file, and use that.
# echo "78a8c98bcb2efe1a63318d901ab204d9ba96c3b29707b4ce0c4240bdcdc698d6 ./bin/Release/clblast.dll" >> tmp
# sha256sum -c tmp || exit 255
# rm tmp
ls -R
- name: Build
if: ${{ matrix.os == 'ubuntu-22.04' }}
run: |
mkdir build
cd build
cmake .. ${{ env.COMMON_DEFINE }} -DLLAMA_NATIVE=OFF -DLLAMA_BUILD_SERVER=ON -DLLAMA_VULKAN=ON -DBUILD_SHARED_LIBS=ON
cmake --build . --config Release -j ${env:NUMBER_OF_PROCESSORS}
# if we ever want to pull libvulkan.so back into the packages, just uncomment this line, and the one below for the upload
# cp $(ldconfig -p | grep libvulkan.so | tail -n 1 | cut -d ' ' -f 4) ./
ls -R
- name: Upload artifacts (Windows)
if: ${{ matrix.os == 'windows-latest' }}
uses: actions/upload-artifact@v4
with:
path: |
.\build\bin\Release\llama.dll
name: llama-bin-win-vulkan-x64.dll
- name: Upload artifacts (linux)
if: ${{ matrix.os == 'ubuntu-22.04' }}
uses: actions/upload-artifact@v4
with:
path: |
./build/libllama.so
name: llama-bin-linux-vulkan-x64.so

compile-cublas:
name: Compile (cublas)
strategy:
Expand Down Expand Up @@ -277,7 +355,8 @@ jobs:
"compile-macos",
"compile-windows",
"compile-cublas",
"compile-clblast"
"compile-clblast",
"compile-vulkan"
]
steps:
- uses: actions/download-artifact@v4
Expand All @@ -288,7 +367,7 @@ jobs:
- name: Rearrange Files
run: |
# Make all directories at once
mkdir --parents deps/{avx,avx2,avx512,osx-arm64,osx-x64,cu11.7.1,cu12.1.0,clblast}
mkdir --parents deps/{avx,avx2,avx512,osx-arm64,osx-x64,cu11.7.1,cu12.1.0,clblast,vulkan}

cp artifacts/llama-bin-linux-noavx-x64.so/libllama.so deps/libllama.so
cp artifacts/llama-bin-linux-avx-x64.so/libllama.so deps/avx/libllama.so
Expand All @@ -312,6 +391,9 @@ jobs:
cp artifacts/llama-bin-win-clblast-x64.dll/{llama,clblast}.dll deps/clblast/
cp artifacts/llama-bin-linux-clblast-x64.so/libllama.so deps/clblast/

cp artifacts/llama-bin-win-vulkan-x64.dll/llama.dll deps/vulkan/
cp artifacts/llama-bin-linux-vulkan-x64.so/libllama.so deps/vulkan/

- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
Expand Down
8 changes: 8 additions & 0 deletions LLama/LLamaSharp.Runtime.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/win-x64/native/cuda12/llama.dll</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/vulkan/llama.dll">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/win-x64/native/vulkan/llama.dll</Link>
</None>

<None Include="$(MSBuildThisFileDirectory)runtimes/deps/libllama.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
Expand All @@ -53,6 +57,10 @@
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-x64/native/cuda12/libllama.so</Link>
</None>
<None Include="$(MSBuildThisFileDirectory)runtimes/deps/vulkan/libllama.so">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<Link>runtimes/linux-x64/native/vulkan/libllama.so</Link>
</None>

<None Include="$(MSBuildThisFileDirectory)runtimes/deps/osx-arm64/libllama.dylib">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
Expand Down
57 changes: 57 additions & 0 deletions LLama/Native/NativeApi.Load.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text.Json;

Expand Down Expand Up @@ -59,6 +60,51 @@ private static void Log(string message, LogLevel level)
Console.ResetColor();
}

private static string GetVulkanVersion()
{
var apiVersionString = string.Empty;
try
{
ProcessStartInfo start = new()
{
FileName = "vulkaninfo",
Arguments = "--summary",
RedirectStandardOutput = true,
UseShellExecute = false,
CreateNoWindow = true
};

Process process = new()
{
StartInfo = start
};
process.Start();

string output = process.StandardOutput.ReadToEnd();
process.WaitForExit();

var lines = output.Split('\n');
int apiVersionLineIndex = lines.ToList().FindIndex(line => line.Contains("apiVersion"));
if (apiVersionLineIndex >= 0)
{
var apiVersionline = lines[apiVersionLineIndex];
//apiVersionline =" apiVersion = 1.3.260";
//apiVersionline =" apiVersion = 4206830 (1.3.238)";
if(apiVersionline.Contains('=') && apiVersionline.Length > apiVersionline.IndexOf('=')+1)
{
apiVersionString = apiVersionline.Substring(apiVersionline.IndexOf('=')+1).Trim();
if(apiVersionString.Contains('(') && apiVersionString.Contains(')') && apiVersionString.IndexOf(')') > apiVersionString.IndexOf('('))
{
apiVersionString = apiVersionString.Substring(apiVersionString.IndexOf('(')+1,apiVersionString.IndexOf(')')-apiVersionString.IndexOf('(')-1);
}
return apiVersionString;
}
}
}
catch {}
return apiVersionString;
}

private static int GetCudaMajorVersion()
{
string? cudaPath;
Expand Down Expand Up @@ -209,6 +255,17 @@ private static List<string> GetLibraryTryOrder(NativeLibraryConfig.Description c
// otherwise no cuda detected but allow fallback
}

if (configuration.UseVulkan && (platform == OSPlatform.Windows || platform == OSPlatform.Linux)) // no vulkan on macos
{
string vulkanVersion = GetVulkanVersion();
if(!string.IsNullOrEmpty(vulkanVersion))
{
Log($"Detected device supporting vulkan version {vulkanVersion}.", LogLevel.Information);
string vulkanLibraryPath = $"{prefix}vulkan/{libraryNamePrefix}{libraryName}{suffix}";
result.Add(vulkanLibraryPath);
}
}

// use cpu (or mac possibly with metal)
if (!configuration.AllowFallback && platform != OSPlatform.OSX)
{
Expand Down
21 changes: 19 additions & 2 deletions LLama/Native/NativeLibraryConfig.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ public sealed class NativeLibraryConfig

private string _libraryPath = string.Empty;
private bool _useCuda = true;
private bool _useVulkan = true;
private AvxLevel _avxLevel;
private bool _allowFallback = true;
private bool _skipCheck = false;
Expand Down Expand Up @@ -69,6 +70,20 @@ public NativeLibraryConfig WithCuda(bool enable = true)
return this;
}

/// <summary>
/// Configure whether to use vulkan backend if possible.
/// </summary>
/// <param name="enable"></param>
/// <returns></returns>
/// <exception cref="InvalidOperationException">Thrown if `LibraryHasLoaded` is true.</exception>
public NativeLibraryConfig WithVulkan(bool enable = true)
{
ThrowIfLoaded();

_useVulkan = enable;
return this;
}

/// <summary>
/// Configure the prefferred avx support level of the backend.
/// </summary>
Expand Down Expand Up @@ -164,7 +179,8 @@ internal static Description CheckAndGatherDescription()

return new Description(
Instance._libraryPath,
Instance._useCuda,
Instance._useCuda,
Instance._useVulkan,
Instance._avxLevel,
Instance._allowFallback,
Instance._skipCheck,
Expand Down Expand Up @@ -250,7 +266,7 @@ public enum AvxLevel
Avx512,
}

internal record Description(string Path, bool UseCuda, AvxLevel AvxLevel, bool AllowFallback, bool SkipCheck, bool Logging, string[] SearchDirectories)
internal record Description(string Path, bool UseCuda,bool UseVulkan, AvxLevel AvxLevel, bool AllowFallback, bool SkipCheck, bool Logging, string[] SearchDirectories)
{
public override string ToString()
{
Expand All @@ -268,6 +284,7 @@ public override string ToString()
return $"NativeLibraryConfig Description:\n" +
$"- Path: {Path}\n" +
$"- PreferCuda: {UseCuda}\n" +
$"- PreferVulkan: {UseVulkan}\n" +
$"- PreferredAvxLevel: {avxLevelString}\n" +
$"- AllowFallback: {AllowFallback}\n" +
$"- SkipCheck: {SkipCheck}\n" +
Expand Down
24 changes: 24 additions & 0 deletions LLama/runtimes/build/LLamaSharp.Backend.Vulkan.nuspec
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
<?xml version="1.0" encoding="utf-8"?>
<package >
<metadata>
<id>LLamaSharp.Backend.Vulkan</id>
<version>$version$</version>
<title>LLamaSharp.Backend.Vulkan - Vulkan Backend for LLamaSharp</title>
<authors>llama.cpp Authors</authors>
<requireLicenseAcceptance>false</requireLicenseAcceptance>
<license type="expression">MIT</license>
<icon>icon512.png</icon>
<projectUrl>https://github.com/SciSharp/LLamaSharp</projectUrl>
<description>LLamaSharp.Backend.Vulkan is a backend for LLamaSharp to use with Vulkan.</description>
<releaseNotes></releaseNotes>
<copyright>Copyright 2023 The llama.cpp Authors. All rights reserved.</copyright>
<tags>LLamaSharp LLama LLM GPT AI ChatBot SciSharp</tags>
</metadata>

<files>
<file src="LLamaSharpBackend.props" target="build/netstandard2.0/LLamaSharp.Backend.Vulkan.props" />
<file src="runtimes/deps/vulkan/llama.dll" target="runtimes\win-x64\native\vulkan\llama.dll" />
<file src="runtimes/deps/vulkan/libllama.so" target="runtimes\linux-x64\native\vulkan\libllama.so" />
<file src="icon512.png" target="icon512.png" />
</files>
</package>
7 changes: 0 additions & 7 deletions LLama/runtimes/build/temp.csproj
View file
Open in desktop

This file was deleted.

Loading