Access Violation in SafeLlamaContextHandle.Decode

[PaulaScholz]

Looks like the batch.ToNativeBatch function returns an nb object containing no tokens and thus there is an access violation by llama_decode. This happens with the raw example chat program and Semantic Kernel Chat example. Ironically, the web project works fine.

    /// <summary>
    /// </summary>
    /// <param name="batch"></param>
    /// <returns>Positive return values does not mean a fatal error, but rather a warning:<br />
    ///  - 0: success<br />
    ///  - 1: could not find a KV slot for the batch (try reducing the size of the batch or increase the context)<br />
    ///  - &lt; 0: error<br />
    /// </returns>
    public DecodeResult Decode(LLamaBatch batch)
    {
        lock (GlobalInferenceLock)
            using (batch.ToNativeBatch(out var nb))
                return (DecodeResult)llama_decode(this, nb);
    }

Call Stack: [Managed to Native Transition]

LLamaSharp.dll!LLama.Native.SafeLLamaContextHandle.Decode(LLama.Native.LLamaBatch batch) Line 373 C#
LLamaSharp.dll!LLama.Native.SafeLLamaContextHandle.Decode(System.Collections.Generic.List<LLama.Native.LLamaToken> tokens, LLama.Native.LLamaSeqId id, LLama.Native.LLamaBatch batch, ref int n_past) Line 400 C#
LLamaSharp.dll!LLama.StatelessExecutor.InferAsync(string prompt, LLama.Abstractions.IInferenceParams inferenceParams, System.Threading.CancellationToken cancellationToken) Line 98 C#
[Resuming Async Method]
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start<LLama.StatelessExecutor.d__21>(ref LLama.StatelessExecutor.d__21 stateMachine) Unknown
LLamaSharp.dll!LLama.LLamaTransforms.KeywordTextOutputStreamTransform.TransformAsync(System.Collections.Generic.IAsyncEnumerable tokens) Line 232 C#
[Resuming Async Method]
System.Private.CoreLib.dll!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start<LLama.LLamaTransforms.KeywordTextOutputStreamTransform.d__12>(ref LLama.LLamaTransforms.KeywordTextOutputStreamTransform.d__12 stateMachine) Unknown
LLamaSharp.SemanticKernel.dll!LLamaSharp.SemanticKernel.ChatCompletion.LLamaSharpChatCompletion.GetChatMessageContentsAsync(Microsoft.SemanticKernel.ChatCompletion.ChatHistory chatHistory, Microsoft.SemanticKernel.PromptExecutionSettings executionSettings, Microsoft.SemanticKernel.Kernel kernel, System.Threading.CancellationToken cancellationToken) Line 77 C#
Microsoft.SemanticKernel.Abstractions.dll!Microsoft.SemanticKernel.ChatCompletion.ChatCompletionServiceExtensions.GetChatMessageContentAsync(Microsoft.SemanticKernel.ChatCompletion.IChatCompletionService chatCompletionService, Microsoft.SemanticKernel.ChatCompletion.ChatHistory chatHistory, Microsoft.SemanticKernel.PromptExecutionSettings executionSettings, Microsoft.SemanticKernel.Kernel kernel, System.Threading.CancellationToken cancellationToken) Unknown
LlamaSharpExample.dll!LLama.Examples.Examples.SemanticKernelChat.Main() Line 34 C#
LlamaSharpExample.dll!LLama.Examples.Examples.SemanticKernelChat.

() Unknown

[martindevans]

Is this happening all the time, or just in specific situations?

[PaulaScholz]

All the time in a console program.

[PaulaScholz]

If I go back to LlamaSharp 0.10, the issue disappears.

[martindevans]

Can you give us some details on what model you're using, what calls you're making etc?

[PaulaScholz]

Sure, I'm just using the Semantic Kernel chat program sample. As for model, llama-2-13b-chat.Q5_K_M.gguf is the file. It works with 0.10, same hardware.

[martindevans]

It's odd no one else has reported this! It doesn't look like you're doing anything particularly unusual.

If the error is caused by trying to decode an empty batch that is at least easy to avoid.

[martindevans]

I've just pushed up #691, which skips decoding if the batch is empty. Could you pull that branch and try it to see if it fixes your issue?