v0.8.1 -- Unable to utilize CUDA in any way

[Onkitova]

Hello and thank you for the great project!

I am trying to utilize GPU (windows 11 + Nvidia 2060S with the most recent driver and CUDA 12.3) with example console project from the repo. Unfortunately, despite
NativeLibraryConfig.Instance.WithCuda().WithLogs();
it automatically loads just avx2 version:

If I am trying to manually adjust example project code with changing
NativeLibraryConfig.Instance.WithCuda().WithLogs();
to
NativeLibraryConfig.Instance.WithLibrary(@"C:\Users\Username\Desktop\LLamaSharp-master\LLamaSharp-master\LLama.Examples\bin\Debug\net6.0\runtimes\win-x64\native\cuda12\libllama.dll");
I am getting an exception:

System.TypeInitializationException: 'The type initializer for 'LLama.Native.NativeApi' threw an exception.'

RuntimeError: Failed to load the native library [C:\Users\Username\Desktop\LLamaSharp-master\LLamaSharp-master\LLama.Examples\bin\Debug\net6.0\runtimes\win-x64\native\cuda12\libllama.dll] you specified.

Same goes for cuda11 version. Just for the sake of info: if I am trying to manually feed avx2 version of libllama.dll with exact NativeLibraryConfig.Instance.WithLibrary(%PATH%) thing -- everything loads and works fine.

Any ideas in which direction to dig to solve this?

[martindevans]

If loading the binary by passing the explicit path isn't working it's likely you're missing some dependencies.

[Onkitova]

If loading the binary by passing the explicit path isn't working it's likely you're missing some dependencies.

Are there any other dependencies except CUDA Toolkit?

[SanftMonster]

@Onkitova What is the result of nvcc -V and nvidia-smi in your windows powershell?

[Onkitova]

@AsakusaRinne thanks for spotting this and also confirming my hypothesis.

So, the problem was that initially nvcc -V returned:

after a little investigation I found out that a complete clear reinstall of Cuda Toolkit (12.2 or 12.3 as well, can surely confirm BOTH working) in "Express mode" + restart of PC solve the problem.

It was kind of my bad that initially I tried to install CUDA Toolkit in custom mode and select just "CUDA/Runtime" components (which, occasionally, don't install nvcc), while -- in order to install nvcc -- you need to select "CUDA/Development" components as well like that:

Now everything works as expected:

Thank you!

However, I would like to ask 1 more thing before closing this issue: is there a way to somehow embed nvcc -- or libraries it actually refers to -- in order to not require the whole CUDA Toolkit as a prerequisite to running applications using llamasharp (if we expect cuda to work as well)?

When there is only current nvidia driver installed and cuda toolkit isn't -- there is no such thing as nvcc.

[SanftMonster]

is there a way to somehow embed nvcc -- or libraries it actually refers to -- in order to not require the whole CUDA Toolkit as a prerequisite to running applications using llamasharp (if we expect cuda to work as well)?

I'm afraid that I don't know a convinient way to achieve it. Theoretically, what we need to do is to put the files exactly the correct paths of a PC. However the configuration differs on devices and systems, which requires a huge amount of work to take account of all the cases. So I prefer not to include it in the roadmap. If there's an one-click way to install cuda toolkit with nuget in the future, please let us know and we'll discuss about it again. :)

[Onkitova]

Theoretically, what we need to do is to put the files exactly the correct paths of a PC.

One thing I have noticed is that every cuda-utilizing software (including, for example, another LLM UI program -- LM studio, which works with driver only and no cuda toolkit requirement) somewhy embeds these 3 files:

Other software example:

Most of them doesn't require CUDA Toolkit as a prerequisite.

The same 3 files that original llama.cpp repo distributes with every release under this asset package:

I am telling you because you are possibly more able to debug further than average user getting this

System.TypeInitializationException: 'The type initializer for 'LLama.Native.NativeApi' threw an exception.'
RuntimeError: Failed to load the native library [C:\Users\Username\Desktop\LLamaSharp-master\LLamaSharp-master\LLama.Examples\bin\Debug\net6.0\runtimes\win-x64\native\cuda12\libllama.dll] you specified.

Maybe those 3 files are the ones we need to reference in order to mitigate the error without CUDA Toolkit?

I am closing this issue as solved, but for now I suggest you to consider adding something about CUDA Toolkit installation to readme (especially the part about truly necessary components of it: development and runtime components branches).

[SanftMonster]

Thank you a lot for this information! I'll test if this approach will work without requiring a CUDA. It'll be a great enhancement if that works. I've opened an issue for it #350