Agent and Plan mode gives 400 status error running local assistant after enabling Tools

[henryfung3a27]

Before submitting your bug report

• I believe this is a bug. I'll try to join the Continue Discord for questions
• I'm not able to find an open issue that reports the same bug
• I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Windows 11
- Continue version: v1.1.59
- IDE version: 1.101.2
- Model: DeepSeek Coder v2 Lite Instruct (vLLM)
- config:
  
name: my-config
version: 0.0.1
schema: v1
models:
  - name: AI Coder
    provider: openai
    model: /Model/Model
    apiBase: https://dev.server/aicoder/v1
    apiKey: 123456
    defaultCompletionOptions:
      contextLength: 128000
    requestOptions:
      verifySsl: false
      timeout: 7200
      caBundlePath: D:\\dev.server.crt
    roles:
      - chat
      - edit
      - apply
      - summarize
    capabilities:
      - tool_use
context:
  - provider: file
  - provider: code
  - provider: codebase
    params:
      nFinal: 10
  - provider: docs
  - provider: diff
  - provider: folder
  - provider: terminal
  - provider: currentFile
  - provider: open
  - provider: search
    params:
      maxResults: 100
  - provider: clipboard
  - provider: tree
  - provider: problems
  - provider: commit
    params:
      # how many commits to load into submenu
      Depth: 50
      # how many recent commits to include
      LastXCommitsDepth: 10
rules:
  - Give concise responses
prompts:
  - name: API route inspection
    description: Analyzes API routes for security issues
    prompt: Review this API route for security vulnerabilities. Ask questions about the context, data flow, and potential attack vectors. Be thorough in your investigation.
  - name: Generate comment
    description: Generate comment for the code
    prompt: Generate comments for the code
  - name: Generate commit message
    description: Generate Git commit message
    prompt: Generate a git commit message

Description

I was trying out the Agent mode and found out that enabling tools will give 400 (no body) error in query.

• Tools disabled using the toggle: OK
• Tools enabled using the toggle but exclude all tools: OK
• Tools enabled using the toggle and only enable one tool (both Ask First and Automatic): Fail

It doesn't matter which tool is enabled. I am pretty sure the model can use tool because I have tried with other extension with the same model and is able to invoke tool use.

To reproduce

1. Open VSCode, download latest Continue extension (v1.1.59 at the time of writing)
2. Config the local assistant using the yaml described
3. Send Hi in Chat mode, gives greeting output from model
4. Select Agent mode, disable Tools and say hi, gives greeting output from model
5. Select Agent mode, enable Tools and say hi, gives 400 (no body) error response

Continue console output

Options tab in the console for the error query after enabling one tool

{
  "contextLength": 128000,
  "model": "/Model/Model",
  "maxTokens": 4096,
  "tools": [
    {
      "type": "function",
      "displayTitle": "Grep Search",
      "wouldLikeTo": "search for \"{{{ query }}}\" in the repository",
      "isCurrently": "getting search results for \"{{{ query }}}\"",
      "hasAlready": "retrieved search results for \"{{{ query }}}\"",
      "readonly": true,
      "isInstant": true,
      "group": "Built-In",
      "function": {
        "name": "grep_search",
        "description": "Perform a search over the repository using ripgrep. Output may be truncated, so use targeted queries",
        "parameters": {
          "type": "object",
          "required": [
            "query"
          ],
          "properties": {
            "query": {
              "type": "string",
              "description": "The search query to use. Must be a valid ripgrep regex expression, escaped where needed"
            }
          }
        }
      }
    }
  ]
}

Log output

> VSCode console


[Extension Host] Error: 400 status code (no body)
    at APIError.generate (c:\Users\User\.vscode\extensions\continue.continue-1.1.59-win32-x64\out\extension.js:109840:18)
    at OpenAI.makeStatusError (c:\Users\User\.vscode\extensions\continue.continue-1.1.59-win32-x64\out\extension.js:110751:25)
    at OpenAI.makeRequest (c:\Users\User\.vscode\extensions\continue.continue-1.1.59-win32-x64\out\extension.js:110795:29)
    at processTicksAndRejections (node:internal/process/task_queues:105:5)
    at OpenAIApi.chatCompletionStream (c:\Users\User\.vscode\extensions\continue.continue-1.1.59-win32-x64\out\extension.js:114858:26)
    at OpenAI2.streamChat (c:\Users\User\.vscode\extensions\continue.continue-1.1.59-win32-x64\out\extension.js:171737:34)
    at llmStreamChat (c:\Users\User\.vscode\extensions\continue.continue-1.1.59-win32-x64\out\extension.js:691642:17)
    at ud.handleMessage [as value] (c:\Users\User\.vscode\extensions\continue.continue-1.1.59-win32-x64\out\extension.js:726099:29)

ERR [Extension Host] Error handling webview message: {
  "msg": {
    "messageId": "059b90e0-eb72-4a2f-a0e1-8e910a99d534",
    "messageType": "llm/streamChat",
    "data": {
      "completionOptions": {
        "tools": [
          {

[henryfung3a27]

BTW I have tried this method to try and see the actual request body. I can see traffics made by VSCode (update servers, marketplace, etc) but none related to Continue.

Then I set the proxy attribute in Continue config.yaml and now the query can't reach the model. It just says Connection Error.

[Patrick-Erichsen]

Thanks for the detailed writeup. I'm not sure what's going on here tbh. Your yaml looks correct to me. Your provider is openai compatible I assume?

If you're able to clone the repo and step through core/llm/llms/OpenAI.ts that's about all I can think of to troubleshoot further unfortunately.

[jacobm-gavin]

Came to say that I am experiencing this exact issue, and have yet to find a solution
continue version 1.1.71

[Patrick-Erichsen]

@uinstinct another one for next week perhaps.

[uinstinct]

hey @henryfung3a27 @jacobm-gavin
I tested this now on a local proxy node server and seemed to be working fine.
Can you please share your exact request and response jsons?

also any chance tweaking proxy settings on vscode help (docs reference)?

[jacobm-gavin]

The Response is a 400 status code with no body.

The request is:

[Extension Host] Error handling webview message: {
  "msg": {
    "messageId": "53c44757-e678-4720-ad2f-ab37abad8263",
    "messageType": "llm/streamChat",
    "data": {
      "completionOptions": {
        "tools": [
          {
            "type": "function",
            "displayTitle": "Read File",
            "wouldLikeTo": "read {{{ filepath }}}",
            "isCurrently": "reading {{{ filepath }}}",
            "hasAlready": "viewed {{{ filepath }}}",
            "readonly": true,
            "isInstant": true,
            "group": "Built-In",
            "function": {
              "name": "read_file",
              "description": "Use this tool if you need to view the contents of an existing file.",
              "parameters": {
                "type": "object",
                "required": [
                  "filepath"
                ],
                "properties": {
                  "filepath": {
                    "type": "string",
                    "description": "The path of the file to read, relative to the root of the workspace (NOT uri or absolute path)"
                  }
                }
              }
            },
            "defaultToolPolicy": "allowedWithoutPermission",
            "systemMessageDescription": "To read a file with a known filepath, use the read_file tool. For example, to read a file located at 'path/to/file.txt', you would respond with this:\n```tool\nTOOL_NAME: read_file\nBEGIN_ARG: filepath\npath/to/the_file.txt\nEND_ARG\n```"
          },
          {
            "type": "function",
            "displayTitle": "Create New File",
            "wouldLikeTo": "create a new file at {{{ filepath }}}",
            "isCurrently": "creating a new file at {{{ filepath }}}",
            "hasAlready": "created a new file at {{{ filepath }}}",
            "group": "Built-In",
            "readonly": false,
            "isInstant": true,
            "function": {
              "name": "create_new_file",
              "description": "Create a new file. Only use this when a file doesn't exist and should be created",
              "parameters": {
                "type": "object",
                "required": [
                  "filepath",
                  "contents"
                ],
                "properties": {
                  "filepath": {
                    "type": "string",
                    "description": "The path where the new file should be created, relative to the root of the workspace"
                  },
                  "contents": {
                    "type": "string",
                    "description": "The contents to write to the new file"
                  }
                }
              }
            },
            "defaultToolPolicy": "allowedWithPermission",
            "systemMessageDescription": "To create a NEW file, use the create_new_file tool with the relative filepath and new contents. For example, to create a file located at 'path/to/file.txt', you would respond with:\n```tool\nTOOL_NAME: create_new_file\nBEGIN_ARG: filepath\npath/to/the_file.txt\nEND_ARG\nBEGIN_ARG: contents\nContents of the file\nEND_ARG\n```"
          },
          {
            "type": "function",
            "displayTitle": "Run Terminal Command",
            "wouldLikeTo": "run the following terminal command:",
            "isCurrently": "running the following terminal command:",
            "hasAlready": "ran the following terminal command:",
            "readonly": false,
            "group": "Built-In",
            "function": {
              "name": "run_terminal_command",
              "description": "Run a terminal command in the current directory.\nThe shell is not stateful and will not remember any previous commands.      When a command is run in the background ALWAYS suggest using shell commands to stop it; NEVER suggest using Ctrl+C.      When suggesting subsequent shell commands ALWAYS format them in shell command blocks.      Do NOT perform actions requiring special/admin privileges.      Choose terminal commands and scripts optimized for win32 and x64 and shell powershell.exe.",
              "parameters": {
                "type": "object",
                "required": [
                  "command"
                ],
                "properties": {
                  "command": {
                    "type": "string",
                    "description": "The command to run. This will be passed directly into the IDE shell."
                  },
                  "waitForCompletion": {
                    "type": "boolean",
                    "description": "Whether to wait for the command to complete before returning. Default is true. Set to false to run the command in the background. Set to true to run the command in the foreground and wait to collect the output."
                  }
                }
              }
            },
            "defaultToolPolicy": "allowedWithPermission",
            "systemMessageDescription": "To run a terminal command, use the run_terminal_command tool\nThe shell is not stateful and will not remember any previous commands.      When a command is run in the background ALWAYS suggest using shell commands to stop it; NEVER suggest using Ctrl+C.      When suggesting subsequent shell commands ALWAYS format them in shell command blocks.      Do NOT perform actions requiring special/admin privileges.      Choose terminal commands and scripts optimized for win32 and x64 and shell powershell.exe.\nYou can also optionally include the waitForCompletion argument set to false to run the command in the background.      \nFor example, to see the git log, you could respond with:\n```tool\nTOOL_NAME: run_terminal_command\nBEGIN_ARG: command\ngit log\nEND_ARG\n```"
          },
          {
            "type": "function",
            "displayTitle": "Glob File Search",
            "wouldLikeTo": "find file matches for \"{{{ pattern }}}\"",
            "isCurrently": "finding file matches for \"{{{ pattern }}}\"",
            "hasAlready": "retrieved file matches for \"{{{ pattern }}}\"",
            "readonly": true,
            "isInstant": true,
            "group": "Built-In",
            "function": {
              "name": "file_glob_search",
              "description": "Search for files recursively in the project using glob patterns. Supports ** for recursive directory search. Output may be truncated; use targeted patterns",
              "parameters": {
                "type": "object",
                "required": [
                  "pattern"
                ],
                "properties": {
                  "pattern": {
                    "type": "string",
                    "description": "Glob pattern for file path matching"
                  }
                }
              }
            },
            "defaultToolPolicy": "allowedWithoutPermission",
            "systemMessageDescription": "To return a list of files based on a glob search pattern, use the file_glob_search tool\n```tool\nTOOL_NAME: file_glob_search\nBEGIN_ARG: pattern\n*.py\nEND_ARG\n```"
          },
          {
            "type": "function",
            "displayTitle": "Search Web",
            "wouldLikeTo": "search the web for \"{{{ query }}}\"",
            "isCurrently": "searching the web for \"{{{ query }}}\"",
            "hasAlready": "searched the web for \"{{{ query }}}\"",
            "readonly": true,
            "group": "Built-In",
            "function": {
              "name": "search_web",
              "description": "Performs a web search, returning top results. Use this tool sparingly - only for questions that require specialized, external, and/or up-to-date knowledege. Common programming questions do not require web search.",
              "parameters": {
                "type": "object",
                "required": [
                  "query"
                ],
                "properties": {
                  "query": {
                    "type": "string",
                    "description": "The natural language search query"
                  }
                }
              }
            },
            "defaultToolPolicy": "allowedWithoutPermission",
            "systemMessageDescription": "To search the web, use the search_web tool with a natural language query. For example, to search for the current weather, you would respond with:\n```tool\nTOOL_NAME: search_web\nBEGIN_ARG: query\nWhat is the current weather in San Francisco?\nEND_ARG\n```"
          },
          {
            "type": "function",
            "displayTitle": "View Diff",
            "wouldLikeTo": "view the git diff",
            "isCurrently": "getting the git diff",
            "hasAlready": "viewed the git diff",
            "readonly": true,
            "isInstant": true,
            "group": "Built-In",
            "function": {
              "name": "view_diff",
              "description": "View the current diff of working changes",
              "parameters": {
                "type": "object",
                "properties": {}
              }
            },
            "defaultToolPolicy": "allowedWithoutPermission",
            "systemMessageDescription": "To view the current git diff, use the view_diff tool. This will show you the changes made in the working directory compared to the last commit.\n```tool\nTOOL_NAME: view_diff\n```"
          },
          {
            "type": "function",
            "displayTitle": "Read Currently Open File",
            "wouldLikeTo": "read the current file",
            "isCurrently": "reading the current file",
            "hasAlready": "viewed the current file",
            "readonly": true,
            "isInstant": true,
            "group": "Built-In",
            "function": {
              "name": "read_currently_open_file",
              "description": "Read the currently open file in the IDE. If the user seems to be referring to a file that you can't see, try using this",
              "parameters": {
                "type": "object",
                "properties": {}
              }
            },
            "defaultToolPolicy": "allowedWithPermission",
            "systemMessageDescription": "To view the user's currently open file, use the read_currently_open_file tool.\nIf the user is asking about a file and you don't see any code, use this to check the current file\n```tool\nTOOL_NAME: read_currently_open_file\n```"
          },
          {
            "type": "function",
            "displayTitle": "ls",
            "wouldLikeTo": "list files and folders in {{{ dirPath }}}",
            "isCurrently": "listing files and folders in {{{ dirPath }}}",
            "hasAlready": "listed files and folders in {{{ dirPath }}}",
            "readonly": true,
            "isInstant": true,
            "group": "Built-In",
            "function": {
              "name": "ls",
              "description": "List files and folders in a given directory",
              "parameters": {
                "type": "object",
                "properties": {
                  "dirPath": {
                    "type": "string",
                    "description": "The directory path relative to the root of the project. Use forward slash paths like '/'. rather than e.g. '.'"
                  },
                  "recursive": {
                    "type": "boolean",
                    "description": "If true, lists files and folders recursively. To prevent unexpected large results, use this sparingly"
                  }
                }
              }
            },
            "defaultToolPolicy": "allowedWithoutPermission",
            "systemMessageDescription": "To list files and folders in a given directory, call the ls tool with \"dirPath\" and \"recursive\". For example:\n```tool\nTOOL_NAME: ls\nBEGIN_ARG: dirPath\npath/to/dir\nEND_ARG\nBEGIN_ARG: recursive\nfalse\nEND_ARG\n```"
          },
          {
            "type": "function",
            "displayTitle": "Create Rule Block",
            "wouldLikeTo": "create a rule block for \"{{{ name }}}\"",
            "isCurrently": "creating a rule block for \"{{{ name }}}\"",
            "hasAlready": "created a rule block for \"{{{ name }}}\"",
            "readonly": false,
            "isInstant": true,
            "group": "Built-In",
            "function": {
              "name": "create_rule_block",
              "description": "Creates a \"rule\" that can be referenced in future conversations. This should be used whenever you want to establish code standards / preferences that should be applied consistently, or when you want to avoid making a mistake again. To modify existing rules, use the edit tool instead.\n\nRule Types:\n- Always: Include only \"rule\" (always included in model context)\n- Auto Attached: Include \"rule\", \"globs\", and/or \"regex\" (included when files match patterns)\n- Agent Requested: Include \"rule\" and \"description\" (AI decides when to apply based on description)\n- Manual: Include only \"rule\" (only included when explicitly mentioned using @ruleName)",
              "parameters": {
                "type": "object",
                "required": [
                  "name",
                  "rule"
                ],
                "properties": {
                  "name": {
                    "type": "string",
                    "description": "Short, descriptive name summarizing the rule's purpose (e.g. 'React Standards', 'Type Hints')"
                  },
                  "rule": {
                    "type": "string",
                    "description": "Clear, imperative instruction for future code generation (e.g. 'Use named exports', 'Add Python type hints'). Each rule should focus on one specific standard."
                  },
                  "description": {
                    "type": "string",
                    "description": "Description of when this rule should be applied. Required for Agent Requested rules (AI decides when to apply). Optional for other types."
                  },
                  "globs": {
                    "type": "string",
                    "description": "Optional file patterns to which this rule applies (e.g. ['**/*.{ts,tsx}'] or ['src/**/*.ts', 'tests/**/*.ts'])"
                  },
                  "regex": {
                    "type": "string",
                    "description": "Optional regex patterns to match against file content. Rule applies only to files whose content matches the pattern (e.g. 'useEffect' for React hooks or '\\bclass\\b' for class definitions)"
                  },
                  "alwaysApply": {
                    "type": "boolean",
                    "description": "Whether this rule should always be applied. Set to false for Agent Requested and Manual rules. Omit or set to true for Always and Auto Attached rules."
                  }
                }
              }
            },
            "defaultToolPolicy": "allowedWithPermission",
            "systemMessageDescription": "Sometimes the user will provide feedback or guidance on your output. If you were not aware of these \"rules\", consider using the create_rule_block tool to persist the rule for future interactions.\nThis tool cannot be used to edit existing rules, but you can search in the \".continue/rules\" folder and use the edit tool to manage rules.\nTo create a rule, respond with a create_rule_block tool call and the following arguments:\n- name: Short, descriptive name summarizing the rule's purpose (e.g. 'React Standards', 'Type Hints')\n- rule: Clear, imperative instruction for future code generation (e.g. 'Use named exports', 'Add Python type hints'). Each rule should focus on one specific standard.\n- description: Description of when this rule should be applied. Required for Agent Requested rules (AI decides when to apply). Optional for other types.\n- globs: Optional file patterns to which this rule applies (e.g. ['**/*.{ts,tsx}'] or ['src/**/*.ts', 'tests/**/*.ts'])\n- alwaysApply: Whether this rule should always be applied. Set to false for Agent Requested and Manual rules. Omit or set to true for Always and Auto Attached rules.\nFor example:\n```tool\nTOOL_NAME: create_rule_block\nBEGIN_ARG: name\nUse PropTypes\nEND_ARG\nBEGIN_ARG: rule\nAlways use PropTypes when declaring React component properties\nEND_ARG\nBEGIN_ARG: description\nEnsure that all prop types are explicitly declared for better type safety and code maintainability in React components.\nEND_ARG\nBEGIN_ARG: globs\n**/*.js\nEND_ARG\nBEGIN_ARG: alwaysApply\nfalse\nEND_ARG\n```"
          },
          {
            "type": "function",
            "displayTitle": "Read URL",
            "wouldLikeTo": "fetch {{{ url }}}",
            "isCurrently": "fetching {{{ url }}}",
            "hasAlready": "viewed {{{ url }}}",
            "readonly": true,
            "isInstant": true,
            "group": "Built-In",
            "function": {
              "name": "fetch_url_content",
              "description": "Can be used to view the contents of a website using a URL. Do NOT use this for files.",
              "parameters": {
                "type": "object",
                "required": [
                  "url"
                ],
                "properties": {
                  "url": {
                    "type": "string",
                    "description": "The URL to read"
                  }
                }
              }
            },
            "defaultToolPolicy": "allowedWithPermission",
            "systemMessageDescription": "To fetch the content of a URL, use the fetch_url_content tool. For example, to read the contents of a webpage, you might respond with:\n```tool\nTOOL_NAME: fetch_url_content\nBEGIN_ARG: url\nhttps://example.com\nEND_ARG\n```"
          },
          {
            "type": "function",
            "displayTitle": "Grep Search",
            "wouldLikeTo": "search for \"{{{ query }}}\" in the repository",
            "isCurrently": "getting search results for \"{{{ query }}}\"",
            "hasAlready": "retrieved search results for \"{{{ query }}}\"",
            "readonly": true,
            "isInstant": true,
            "group": "Built-In",
            "function": {
              "name": "grep_search",
              "description": "Perform a search over the repository using ripgrep. Output may be truncated, so use targeted queries",
              "parameters": {
                "type": "object",
                "required": [
                  "query"
                ],
                "properties": {
                  "query": {
                    "type": "string",
                    "description": "The search query to use. Must be a valid ripgrep regex expression, escaped where needed"
                  }
                }
              }
            },
            "defaultToolPolicy": "allowedWithoutPermission",
            "systemMessageDescription": "To perform a grep search within the project, call the grep_search tool with the query pattern to match. For example:\n```tool\nTOOL_NAME: grep_search\nBEGIN_ARG: query\n.*main_services.*\nEND_ARG\n```"
          },
          {
            "type": "function",
            "displayTitle": "Request Rules",
            "wouldLikeTo": "request rule {{{ name }}}",
            "isCurrently": "reading rule {{{ name }}}",
            "hasAlready": "read rule {{{ name }}}",
            "group": "Built-In",
            "readonly": false,
            "function": {
              "name": "request_rule",
              "description": "Use this tool to retrieve additional 'rules' that contain more context/instructions based on their descriptions. Available rules:\nNo rules available.",
              "parameters": {
                "type": "object",
                "required": [
                  "name"
                ],
                "properties": {
                  "name": {
                    "type": "string",
                    "description": "Name of the rule"
                  }
                }
              }
            },
            "defaultToolPolicy": "disabled",
            "systemMessageDescription": "To retrieve \"rules\" that contain more context/instructions based on their descriptions, use the request_rule tool with the name of the rule. The available rules are:\nNo rules available.\n\nFor example, you might respond with:\n```tool\nTOOL_NAME: request_rule\nBEGIN_ARG: name\nrule_name\nEND_ARG\n```"
          },
          {
            "type": "function",
            "displayTitle": "Edit File",
            "wouldLikeTo": "edit {{{ filepath }}}",
            "isCurrently": "editing {{{ filepath }}}",
            "hasAlready": "edited {{{ filepath }}}",
            "group": "Built-In",
            "readonly": false,
            "isInstant": false,
            "function": {
              "name": "edit_existing_file",
              "description": "Use this tool to edit an existing file. If you don't know the contents of the file, read it first.\n  When addressing code modification requests, present a concise code snippet that\n  emphasizes only the necessary changes and uses abbreviated placeholders for\n  unmodified sections. For example:\n\n  ```language /path/to/file\n  // ... existing code ...\n\n  {{ modified code here }}\n\n  // ... existing code ...\n\n  {{ another modification }}\n\n  // ... rest of code ...\n  ```\n\n  In existing files, you should always restate the function or class that the snippet belongs to:\n\n  ```language /path/to/file\n  // ... existing code ...\n\n  function exampleFunction() {\n    // ... existing code ...\n\n    {{ modified code here }}\n\n    // ... rest of function ...\n  }\n\n  // ... rest of code ...\n  ```\n\n  Since users have access to their complete file, they prefer reading only the\n  relevant modifications. It's perfectly acceptable to omit unmodified portions\n  at the beginning, middle, or end of files using these \"lazy\" comments. Only\n  provide the complete file when explicitly requested. Include a concise explanation\n  of changes unless the user specifically asks for code only.\n\nNote this tool CANNOT be called in parallel.",
              "parameters": {
                "type": "object",
                "required": [
                  "filepath",
                  "changes"
                ],
                "properties": {
                  "filepath": {
                    "type": "string",
                    "description": "The path of the file to edit, relative to the root of the workspace."
                  },
                  "changes": {
                    "type": "string",
                    "description": "Any modifications to the file, showing only needed changes. Do NOT wrap this in a codeblock or write anything besides the code changes. In larger files, use brief language-appropriate placeholders for large unmodified sections, e.g. '// ... existing code ...'"
                  }
                }
              }
            },
            "defaultToolPolicy": "allowedWithPermission",
            "systemMessageDescription": "To edit an EXISTING file, use the edit_existing_file tool with\n- filepath: the relative filepath to the file.\n- changes: Any modifications to the file, showing only needed changes. Do NOT wrap this in a codeblock or write anything besides the code changes. In larger files, use brief language-appropriate placeholders for large unmodified sections, e.g. '// ... existing code ...'\nOnly use this tool if you already know the contents of the file. Otherwise, use the read_file or read_currently_open_file tool to read it first.\nFor example:\n```tool\nTOOL_NAME: edit_existing_file\nBEGIN_ARG: filepath\npath/to/the_file.ts\nEND_ARG\nBEGIN_ARG: changes\n// ... existing code ...\nfunction subtract(a: number, b: number): number {\n  return a - b;\n}\n// ... rest of code ...\nEND_ARG\n```"
          }
        ]
      },
      "title": "devstral",
      "messages": [
        {
          "role": "system",
          "content": "<important_rules>\n  You are in agent mode.\n\n  If you need to use multiple tools, you can call call multiple read only tools simultaneously.\n\n  Always include the language and file name in the info string when you write code blocks.\n  If you are editing \"src/main.py\" for example, your code block should start with '```python src/main.py'\n\n</important_rules>"
        },
        {
          "role": "user",
          "content": "hi"
        }
      ],
      "messageOptions": {
        "precompiled": true
      }
    }
  }
}

Error: 400 status code (no body)

I am running devstral 4bit https://huggingface.co/unsloth/Devstral-Small-2505-bnb-4bit on vllm.
Similarly to the original issue, I am able to send requests to the model when all tools are disabled, but enabling even one tool causes the above behavior

[jacobm-gavin]

@uinstinct @henryfung3a27 Looks like my issue turned out to be from a faulty configuration of vLLM. by adding

                --tool-call-parser mistral \
                --enable-auto-tool-choice \
                --chat-template examples/tool_chat_template_mistral.jinja

to my vLLM startup command, which I did not know were missing, the issue resolved itself and the model can use tools in agent mode as expected. @henryfung3a27 curious to know your issue may have stemmed from something similar

[henryfung3a27]

I tried again with latest version (v1.1.76). My vLLM config has the --tool-call-parser and --enable-auto-tool-choice already. After adding --chat-template examples/tool_chat_template_mistral.jinja, there are finally some response from server.

Prompt: what is the git status now on this project

I expect it to run git status command from terminal.

Model Output:
"
To check the git status of the project, I'll first need to initialize a git repository if it hasn't been initialized yet. Then, I'll run the git status command to see the current status. Let's do that.
"

And that's it. I found errors on vLLM output:

aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365] Error trying to handle streaming tool call.
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365] Traceback (most recent call last):
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/tool_parsers/mistral_tool_parser.py", line 220, in extract_tool_calls_streaming
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]     tool_call_arr: list[dict] = partial_json_parser.loads(
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]   File "/usr/local/lib/python3.12/dist-packages/partial_json_parser/core/api.py", line 15, in parse_json
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]     return parser(ensure_json(json_string, allow_partial, use_fast_fix))
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]   File "/usr/lib/python3.12/json/__init__.py", line 346, in loads
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]     return _default_decoder.decode(s)
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]   File "/usr/lib/python3.12/json/decoder.py", line 338, in decode
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]   File "/usr/lib/python3.12/json/decoder.py", line 356, in raw_decode
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365]     raise JSONDecodeError("Expecting value", s, err.value) from None
aicoder  | ERROR 08-12 19:32:16 [mistral_tool_parser.py:365] json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I also realise that during vLLM startup, there is a warning message that only shows after adding the --chat-template option.

WARNING 08-12 19:27:11 [chat_utils.py:345] 'chat_template' cannot be overridden for mistral tokenizer.

I am using vLLM docker image vllm/vllm-openai:0.9.2.

some digging

Then I removed the --chat-template option and restart the vLLM, the agent mode has response again (?!).

Model output:
"
Let me check the git status for you.
"

Then I assume it will run the command from terminal, but it stopped here. vLLM showed the same error: "Error trying to handle streaming tool call".

I found an issue from vllm repo: vllm-project/vllm#18729 which is closed and I believe was included in v0.9.2.

---

Please let me know if this is no longer related to Continue. I will try to update vLLM to v0.10.0 in near future.