Skip to content

Fix typo in token_postproc_method names #2261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 1, 2024
Merged

Fix typo in token_postproc_method names #2261

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions cpp/serve/engine.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,8 @@ class EngineImpl : public Engine {
this->tokenizer_ = Tokenizer::FromPath(engine_config->model);
std::string token_table_postproc_method;
if (model_configs[0].count("token_table_postproc_method") == 0) {
// Backward compatibility: use "byte-fallback" by default
token_table_postproc_method = "byte-fallback";
// Backward compatibility: use "byte_fallback" by default
token_table_postproc_method = "byte_fallback";
} else {
token_table_postproc_method =
model_configs[0].at("token_table_postproc_method").get<std::string>();
Expand Down
8 changes: 4 additions & 4 deletions cpp/tokenizers.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,12 +69,12 @@ class Tokenizer : public ObjectRef {
* later processing. E.g. For LLaMA-2, convert "▁of" to " of".
*
* \param token_table The raw token table.
* \param postproc_method The postprocessing method to use. Now we only support "byte-fallback"
* and "byte-level", which refers to the type of the decoder of the tokenizer.
* - "byte-fallback": Use the decoding method in the byte-fallback BPE tokenizer. This is used
* \param postproc_method The postprocessing method to use. Now we only support "byte_fallback"
* and "byte_level", which refers to the type of the decoder of the tokenizer.
* - "byte_fallback": Use the decoding method in the byte-fallback BPE tokenizer. This is used
* by LLaMA-2, Mixtral-7b, etc. This method: 1) transform tokens like <0x1B> to hex char
* byte 1B. (known as the byte-fallback method); 2) transform \\u2581 to space.
* - "byte-level": Use the decoding method in the byte-level BPE tokenizer. This is used by
* - "byte_level": Use the decoding method in the byte-level BPE tokenizer. This is used by
* LLaMA-3, GPT-2, Phi-2, etc. This method inverses the bytes-to-unicode transformation in
* the encoding process as in
* https://github.com/huggingface/transformers/blob/87be06ca77166e6a6215eee5a990ab9f07238a18/src/transformers/models/gpt2/tokenization_gpt2.py#L38-L59
Expand Down