issues with accents on convert_ids_to_tokens()

[perezjln]

Hello, the BertTokenizer seems loose accents when convert_ids_to_tokens() is used :

Example:

• original sentence: "great breakfasts in a nice furnished cafè, slightly bohemian."
• corresponding list of token produced : ['great', 'breakfast', '##s', 'in', 'a', 'nice', 'fur', '##nis', '##hed', 'cafe', ',', 'slightly', 'bohemia', '##n', '.']

Here the problem is in "cafe" that loses its accent. I'm using BertTokenizer.from_pretrained('Bert-base-multilingual') as the tokenizer, I also tried with "Bert-base-uncased" and experienced the same issue.

Thanks for this great work!

[elyase]

This is expected behaviour and is how the multilingual and the uncased models were trained. From the original repo:

We are releasing the BERT-Base and BERT-Large models from the paper. Uncased means that the text has been lowercased before WordPiece tokenization, e.g., John Smith becomes john smith. The Uncased model also strips out any accent markers.

[thomwolf]

Yes this is expected.