GetState() returns a 0x0000... (null) handle

[redthing1]

I am using LlamaSharp on Apple Silicon (M2 Max) to run a Llama2-7B model.

I am inputting a document into the model context using the ChatSession::ChatAsync API. Then, I want to save the state and reload it later.

So I tried using llamaContext.GetState to get an opaque handle to the state, but it returns a null pointer. However, using llamaContext.GetStateData does return a byte sequence that allows me to save and load the state as expected.

Why is this the case?

[redthing1]

Is this somehow expected behavior?? When I try to load it, unsurprisingly the program crashes.

[martindevans]

That's definitely a bug (my bug in fact, oops). It should be doing something like this:

// Wrap memory in a "state"
var state = new State(memory);

// set memory to zero, to prevent it being freed in finally block
memory = IntPtr.Zero;

return state;

I'm traveling for the next week and don't have a computer capable of doing LLM things. So if you can test that change and submit it as a PR that would be great!

[redthing1]

Thanks for the quick reply! I'll send in a PR soon.