Switching ChatSessions without writing to file.

[eublefar]

Hi

Could you please confirm that this is correct way to switch between chat sessions without writing them to files (for performance reasons)?
https://github.com/eublefar/LLAMASharpUnityDemo/blob/main/Assets/LLamaSharpTestScript.cs

Do you think it's worth to implement it into some interface at ChatSession level? (similar to how it's done at Executor level)

Also one thing I struggled to find is something like AddMessage but in non-lazy way with context.Decode happening at the same time, to precompute KV cache before generating the first message.

[SanftMonster]

Hi, I think it's a correct way with a rough look at your code.

Do you think it's worth to implement it into some interface at ChatSession level? (similar to how it's done at Executor level)

From my perspective I'd like to see such an API in ChatSession. As you can see, some extra process of the output is performed when switching context, which could be wrapped in an API.

Also one thing I struggled to find is something like AddMessage but in non-lazy way with context.Decode happening at the same time, to precompute KV cache before generating the first message.

It's a good idea for performance improvement. More generally The generation and prefill could be separate parts and exposed to users in a certain way. Currently I think there's also a tricky way to do so. When you want to prefill it without generation, you just need to set the last several words of your prompt as antiprompt (but be sure it's alone).

FYI @martindevans

[eublefar]

Great, thanks for taking time to look at the code.

I think I could make a PR for these changes if you want.

[SanftMonster]

It would be great if you'd like to make it😀