Eval bug: server API endpoint not respecting `n_predict` with `-2` (until context filled)

[henryclw]

Name and Version

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
version: 4844 (d76a86d)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

CUDA

Hardware

RTX 3090

Models

• DeepSeek-R1-Distill-Qwen-1.5B-Q8_0.gguf
• QwQ-32B-Q4_K_M.gguf

Problem description & steps to reproduce

Run the llama.cpp server then pass a chat completion request with n_predict = -2 (until context filled)
curl --location 'http://localhost:8080/v1/chat/completions'
--header 'Content-Type: application/json'
--header 'Authorization: Bearer no-key'
--data '{
"messages": [
{
"role": "user",
"content": "Vancouver is a city located on the northwestern coast of Canada. It is the largest city in the province of British Columbia, flanked by the Pacific Ocean to the west and the Coast Mountains to the east. What else are special about Vancouver?"
}
],
"n_predict": -2
}'

Expected behavior: Keep sampling until context size full
Actual behavior:

{
    "choices": [
        {
            "finish_reason": "length",
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "<think>"
            }
        }
    ],
    "created": 1741406445,
    "model": "gpt-3.5-turbo",
    "system_fingerprint": "b4844-d76a86d9",
    "object": "chat.completion",
    "usage": {
        "completion_tokens": 1,
        "prompt_tokens": 53,
        "total_tokens": 54
    },
    "id": "chatcmpl-bMQP1cnlEjjlHUSYjIwRiQZayJeHSQ1t",
    "timings": {
        "prompt_n": 1,
        "prompt_ms": 203.39,
        "prompt_per_token_ms": 203.39,
        "prompt_per_second": 4.916662569447859,
        "predicted_n": 1,
        "predicted_ms": 0.034,
        "predicted_per_token_ms": 0.034,
        "predicted_per_second": 29411.76470588235
    }
}

It's confirmed that the same setting with n_predict = 32 could set the answer limit to 32 tokens.

First Bad Commit

No response

Relevant log output

main: server is listening on http://0.0.0.0:80 - starting the main loop
srv  update_slots: all slots are idle
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 32768, n_keep = 0, n_prompt_tokens = 53
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 53, n_tokens = 53, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 53, n_tokens = 53
slot      release: id  0 | task 0 | stop processing: n_past = 53, truncated = 0
slot print_timing: id  0 | task 0 | 
prompt eval time =     971.71 ms /    53 tokens (   18.33 ms per token,    54.54 tokens per second)
       eval time =       0.04 ms /     1 tokens (    0.04 ms per token, 25000.00 tokens per second)
      total time =     971.75 ms /    54 tokens
srv  update_slots: all slots are idle

[henryclw]

@ggerganov with this option, it responds with {"error":{"code":500,"message":"context shift is disabled","type":"server_error"}}, in all cases except for when I run it with -n -2 and remove max_tokens from request, in this case it returns 1 token

Referred from @remixer-dec in #12251

[ngxson]

You can use --no-context-shift on server to archive the same behavior

n_predict == -2 is only implemented on main.cpp and infill.cpp

[Martin-Laclaustra]

You can use --no-context-shift on server to archive the same behavior

n_predict == -2 is only implemented on main.cpp and infill.cpp

Please reopen. The behavior is not the same because prediction ends with an error and the client discards the whole response.

With the most recent commit (c95fa36), the following commands result in the same response:

~/opt/llama.cpp/build/bin/llama-server -m deepseek-r1-671b-1.58bit --port 8080 --host 0.0.0.0 --cache-type-k q4_0 --n-gpu-layers 10 --threads 20 --ctx-size 256

~/opt/llama.cpp/build/bin/llama-server -m deepseek-r1-671b-1.58bit --port 8080 --host 0.0.0.0 --cache-type-k q4_0 --n-gpu-layers 10 --threads 20 --ctx-size 256 --no-context-shift

Response:

main: server is listening on http://0.0.0.0:8080 - starting the main loop
srv  update_slots: all slots are idle
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 256, n_keep = 0, n_prompt_tokens = 14
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 14, n_tokens = 14, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 14, n_tokens = 14
slot      release: id  0 | task 0 | stop processing: n_past = 255, truncated = 0
srv    send_error: task id = 0, error: context shift is disabled
srv  update_slots: no tokens to decode
srv  update_slots: all slots are idle
srv  cancel_tasks: cancel task, id_task = 0
srv  log_server_r: request: POST /v1/chat/completions 192.168.1.125 200
srv  update_slots: all slots are idle

(no statistics about the prediction)

Client:
predictions are streamed but when the error occurs, everything is discarded.

However, if --n_predict -2is used with this patch:

--- a/examples/server/server.cpp
+++ b/examples/server/server.cpp
@@ -1335,6 +1335,8 @@ struct server_slot {
 
         if (params.n_predict != -1) {
             n_remaining = params.n_predict - n_decoded;
+        } else if (global_params.n_predict == -2) {
+            n_remaining = n_ctx - n_past - 1;
         } else if (global_params.n_predict != -1) {
             n_remaining = global_params.n_predict - n_decoded;
         }

The command bellow allows for a valid although truncated response:

~/opt/llama.cpp/build/bin/llama-server -m deepseek-r1-671b-1.58bit --port 8080 --host 0.0.0.0 --cache-type-k q4_0 --n-gpu-layers 10 --threads 20 --ctx-size 256 --n_predict -2

Response:

main: server is listening on http://0.0.0.0:8080 - starting the main loop
srv  update_slots: all slots are idle
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 256, n_keep = 0, n_prompt_tokens = 14
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 14, n_tokens = 14, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 14, n_tokens = 14
slot      release: id  0 | task 0 | stop processing: n_past = 255, truncated = 0
slot print_timing: id  0 | task 0 | 
prompt eval time =    1418.02 ms /    14 tokens (  101.29 ms per token,     9.87 tokens per second)
       eval time =   67252.94 ms /   242 tokens (  277.90 ms per token,     3.60 tokens per second)
      total time =   68670.95 ms /   256 tokens
srv  update_slots: all slots are idle
srv  log_server_r: request: POST /v1/chat/completions 192.168.1.125 200

(statistics are calculated and provided)

Client:
predictions are streamed and remain usable, without error.

[ngxson]

Please reopen. The behavior is not the same because prediction ends with an error and the client discards the whole response.

Why don't you simply update the client's code?

IMO it makes no sense to add n_predict == -2 because if you use OAI API, they will also return an error when you reach the limit

[Martin-Laclaustra]

Please reopen. The behavior is not the same because prediction ends with an error and the client discards the whole response.

Why don't you simply update the client's code?

Updating one client's code would fix only that client (i.e. the very same client served by the example server). In contrast, allowing controlling the "stop just before error" at the server level would allow working with any client (even those which can not be modified easily or at all -closed source-).
This would allow more control for serving with "edge" hardware, where many compromises (context size, speed, etc.) may be traded off for flexibility and privacy.

IMO it makes no sense to add n_predict == -2 because if you use OAI API, they will also return an error when you reach the limit

Actually I think that llama.cpp server can aim to be more flexible and adaptable than OAI API.
Using your argument, it also would not make sense to start the server with an n_predict = 100 because you can not start the OAI API with that restriction when setting up the server.
The n_predict == -2 is a supporting config that would set the limit as if for every request we computed in the client exactly X as the remaining tokens in the context and asked only for them. It forces to take all that into account in the client, even if all could be fixed at the server level.

I hope that you consider leaving the code patched even as an undocumented feature (for future improvement or removal). Or as a use-at-your-own-risk feature.

Thank you.