Cannot save/load tokenizer without special tokens

[jonathanasdf]

System Info

main

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained('stabilityai/FreeWilly2')
tokenizer.pad_token_id = None
tokenizer.eos_token_id = None
tokenizer.bos_token_id = None
tokenizer.unk_token_id = None
tokenizer.save_pretrained('/tmp/tok_test')
transformers.AutoTokenizer.from_pretrained('/tmp/tok_test')

transformers/models/llama/tokenization_llama_fast.py", line 152, in update_post_processor
    single = f"{(bos+':0 ') * self.add_bos_token}$A:0{(' '+eos+':0') if self.add_eos_token else ''}"
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

Expected behavior

This works on transformers==4.33.1

I use a separate tokenizer for user input so it doesn't convert "special tokens" in unsanitized input. Maybe there's a different way to do this than setting all the special tokens to None?

[ArthurZucker]

Thanks for reporting fixing now 😉