Behaviour between slow and fast LLaMa tokenizer not equivalent

[NielsRogge]

System Info

Transformers v4.29.2

Who can help?

@ArthurZucker

Reproduction

For a new model (#23460), I'd like to get equivalent behaviour between the slow and fast LLaMa tokenizers. The code of the slow tokenizer was taken from the original code, and now I'd like to translate this to the fast tokenizer as well.

However, as can be seen below, behaviour is not equivalent:

from transformers import LlamaTokenizer, LlamaTokenizerFast
import torch

tokenizer = LlamaTokenizer.from_pretrained("huggyllama/llama-7b", truncation_side="left")
tokenizer.add_special_tokens({"pad_token": "[PAD]"})
tokenizer.add_special_tokens({"bos_token": "</s>"})
tokenizer.add_special_tokens({"eos_token": "</s>"})
tokenizer.add_special_tokens({"unk_token": "</s>"})

fast_tokenizer = LlamaTokenizerFast.from_pretrained("huggyllama/llama-7b", truncation_side="left")
fast_tokenizer.add_special_tokens({"pad_token": "[PAD]"})
fast_tokenizer.add_special_tokens({"bos_token": "</s>"})
fast_tokenizer.add_special_tokens({"eos_token": "</s>"})
fast_tokenizer.add_special_tokens({"unk_token": "</s>"})

prompt = "What is unusual about this image?"

encoding = tokenizer(prompt, return_tensors="pt")

fast_encoding = fast_tokenizer(prompt, return_tensors="pt")

for k,v in encoding.items():
    assert torch.allclose(fast_encoding[k], v)
=> this assertion fails since the input_ids differ:

tensor([[    2,  1724,   338, 22910,  1048,   445,  1967, 29973]])
tensor([[    1,  1724,   338, 22910,  1048,   445,  1967, 29973]])

Expected behavior

I'd expect that the assertion above passes.

[ArthurZucker]

Thanks for reporting, will have a look

[ArthurZucker]

Okay, what's happening here is that you are adding tokens that are already present in the vocabulary of the model.
</s> is 2.

• fast: When you add the bos_token it is not added as it already exist, but the content is updated with the new value for the fast tokenizer.
• slow: the token id is properly updated, but the post_processor is not. This was fixed in [LlamaTokenizerFast] nit update post_processor on the fly #23855

[ArthurZucker]

Reproduced is still working for the latest version of transformers because you are relying on adding the token, which should be ignored but is not. The content in rust is modified .
Use this:
fast_tokenizer.bos_token = "</s>"

[ArthurZucker]

(this will update the processor)

[NielsRogge]

Thanks for taking a look!

However I'm using the latest version of Transformers, have added fast_tokenizer.bos_token = "</s>", but the assertion still fails for me.

[NielsRogge]

Reproduced on main branch, here's a Colab notebook: https://colab.research.google.com/drive/1KA_mliTsvjnhOCO3SApVJkgVd2HEeVQZ?usp=sharing.

[ArthurZucker]

Actually, with fast tokenizers there is no logic to properly update the template processor if it exists. The default has always been to initialize the model with the correct tokens. Meaning:
fast_tokenizer = LlamaTokenizerFast.from_pretrained("huggyllama/llama-7b", truncation_side="left", bos_token=“”)
Is what you should be using. The template processor gets updated only if you change “add_bos” and “add_eos” otherwise the logic is a bit complicated, we have to overload parent setters to bos_token as well as bos_token_id to update the template processing. Not in favor of that so leaving as is, will improve the doc for changing bos and eos in fast

[NielsRogge]

Hmm ok so there's no way to have an equivalent fast tokenizer that makes the script above pass?

The reason is that for the new InstructBLIP model (#23460), the processor class (InstructBlipProcessor) would normally use the AutoTokenizer class to load files from the hub. And as the AutoTokenizer API uses the fast tokenizer by default, I'm currently not getting equivalent results as when I use the slow one.

[ArthurZucker]

No way no. I am not in favor of introducing a very hacky behaviour while the fix should be in rust in that case.
The following works:

from transformers import LlamaTokenizer, LlamaTokenizerFast
import torch

fast_tokenizer = LlamaTokenizerFast.from_pretrained("huggyllama/llama-7b", truncation_side="left", bos_token = "</s>", unk_token = "</s>")
fast_tokenizer.add_special_tokens({"pad_token": "[PAD]"})


tokenizer = LlamaTokenizer.from_pretrained("huggyllama/llama-7b", truncation_side="left", bos_token = "</s>", unk_token = "</s>")
tokenizer.add_special_tokens({"pad_token": "[PAD]"})
prompt = "What is unusual about this image?"

encoding = tokenizer(prompt, return_tensors="pt")

fast_encoding = fast_tokenizer(prompt, return_tensors="pt")

for k,v in encoding.items():
    assert torch.allclose(fast_encoding[k], v)

[ArthurZucker]

Also once you have a tokenizer ready, you can save it and it should have the correct postProcessor