LLama.Native.SafeLLamaContextHandle.Tokenize bug?

[vvdb-architecture]

I've been using Microsoft's Kernel Memory with LLmaSharp and have encountered an (issue)[https://github.com/microsoft/kernel-memory/issues/252].

Basically, a call to

 _context.Tokenize(text).Length;

with text equal to "\n" throws an exception:

LLama.Exceptions.RuntimeError
  HResult=0x80131500
  Message=Error happened during tokenization. It's possibly caused by wrong encoding. Please try to specify the encoding.
  Source=LLamaSharp
  StackTrace:
   at LLama.Native.SafeLLamaContextHandle.Tokenize(String text, Boolean add_bos, Boolean special, Encoding encoding)

and according to the folks there, it seems to be a bug in LLamaSharp. So I've been asked to post the issue here.

I've tested with various models, e.g. kai-7b-instruct.Q5_K_M.gguf and it seems to occur in most of them.

[martindevans]

Thanks for reporting this, I've reproduced it in a test and made some adjustments to the tokenizer which should fix it.

[dluc]

@martindevans I looked at the fix, and if I understand correctly \n is 3 tokens, could you confirm?

Until the bugfix is released, I'll hard code this edge case in Kernel Memory (which is using 0.9.1) to avoid exceptions.

[dluc]

Btw in 0.9.1 the exception occurs also with these strings:

• \t
• \t\t
• \t\t\t
• ......
• \n\n
• \n\n\n
• \n\n\n\n
• ....
• \t\n
• \t\n\t\n\n\n\n\t\t
• ....
• \b
• \v
• \0

[martindevans]

Thanks for those examples, I've just added a unit test covering them (it passes, so it looks like the fix works).

if I understand correctly \n is 3 tokens

It's actually:

• 1: Beginning of string token
• 29871: A space (llama.cpp tokenizer really likes to add a space at the start)
• 13: The actual newline token

[vvdb-architecture]

Another one is \nk.

[martindevans]

All of these should be fixed by the changes currently in master :)

[aletfa]

When a new Nuget package will be released with these fixes?

[martindevans]

Very soon, it's been held up by some other OpenCL related changes. I hope by the end of this week.

[aletfa]

I confirm that the fix resolves all the problems. TY. I'll wait for the next week.

[martindevans]

I jsut released 0.10.0 with these changes :)