fix context length for lmstudio and ollama (#2462) #4314

thecolorblue · 2025-06-04T05:14:55Z

Related GitHub Issue

Closes: #2462

Description

Fixing the context length viewable in the task header while using ollama and LM Studio models.

For both Ollama and LM Studio the correct context length is now being pulled from each service. No fields are needed in the settings page for setting the context length.

I decided to update the code to treat Ollama and LM Studio as model routers and take advantage of the model results caching already in use. There are some UI updates that could be made to make using Ollama and LM Studio more intuitive (for example, debugging an incorrect port or the service not online is difficult), but I did not include those changes in this PR. I can open up another issue for those changes.

I needed to use the @lmstudio/sdk package to get the proper model details from LM Studio. If LM Studio adds the model context length to their API this can be removed.

Test Procedure

I tested with 'devstral' and 'qwen2.5-coder' with both Ollama and LM Studio. I then shut off Ollama and LM Studio and tried to use Roo normally to see how it handles not connecting.

Type of Change

🐛 Bug Fix: Non-breaking change that fixes an issue.
✨ New Feature: Non-breaking change that adds functionality.
💥 Breaking Change: Fix or feature that would cause existing functionality to not work as expected.
♻️ Refactor: Code change that neither fixes a bug nor adds a feature.
💅 Style: Changes that do not affect the meaning of the code (white-space, formatting, etc.).
📚 Documentation: Updates to documentation files.
⚙️ Build/CI: Changes to the build process or CI configuration.
🧹 Chore: Other changes that don't modify src or test files.

Pre-Submission Checklist

Screenshots / Videos

Documentation Updates

As described in the issue, the documentation here mentions configuring the context size.

(Optional) Configure Model context size in Advanced settings, so Roo Code knows how to manage its sliding window.

That should be removed.

Get in Touch

discord: braddavis1008

Important

Fixes context length display for Ollama and LM Studio models by treating them as model routers and utilizing caching.

Behavior:
- Fixes context length display for Ollama and LM Studio models in task header.
- Retrieves correct context length from services, removing manual configuration in settings.
Refactor:
- Treats Ollama and LM Studio as model routers, utilizing model results caching.
- Updates webviewMessageHandler.ts to handle Ollama and LM Studio model fetching.
Dependencies:
- Adds @lmstudio/sdk to package.json for fetching LM Studio model details.
Validation:
- Updates validateApiConfiguration() in validate.ts to check for model IDs for Ollama and LM Studio.
Misc:
- Removes redundant model fetching functions from ollama.ts and lm-studio.ts.

^{This description was created by}^{for 67812b2923994d58b9b6a9255b8f3396286f7fc8. You can customize this summary. It will automatically update as commits are pushed.}

src/api/providers/fetchers/lmstudio.ts

webview-ui/src/components/ui/hooks/useSelectedModel.ts

daniel-lxs

Hey @thecolorblue Thanks for your contribution!

This is a solid implementation. I've left a few suggestions to help improve the robustness of the code, let me know what you think!

src/api/providers/fetchers/ollama.ts

src/api/providers/fetchers/lmstudio.ts

src/api/providers/fetchers/ollama.ts

thecolorblue · 2025-06-09T05:41:48Z

I am running into failed tests when running pnpm test locally. I was not able to figure out what changes need to be made to fix them.

Here is one example:

roo-cline:test:   ● webviewMessageHandler - requestRouterModels › handles individual provider failures gracefully
roo-cline:test: 
roo-cline:test:     expect(jest.fn()).toHaveBeenCalledWith(...expected)
roo-cline:test: 
roo-cline:test:     Expected: {"routerModels": {"glama": {"model-1": {"contextWindow": 8192, "description": "Test model 1", "maxTokens": 4096, "supportsPromptCache": false}}, "litellm": {}, "openrouter": {"model-1": {"contextWindow": 8192, "description": "Test model 1", "maxTokens": 4096, "supportsPromptCache": false}}, "requesty": {}, "unbound": {}}, "type": "routerModels"}
roo-cline:test:     Received
roo-cline:test:            1
roo-cline:test:               Object {
roo-cline:test:             -   "routerModels": Object {
roo-cline:test:             -     "glama": Object {
roo-cline:test:             -       "model-1": Object {
roo-cline:test:             -         "contextWindow": 8192,
roo-cline:test:             -         "description": "Test model 1",
roo-cline:test:             -         "maxTokens": 4096,
roo-cline:test:             -         "supportsPromptCache": false,
roo-cline:test:             -       },
roo-cline:test:             -     },
roo-cline:test:             -     "litellm": Object {},
roo-cline:test:             -     "openrouter": Object {
roo-cline:test:             -       "model-1": Object {
roo-cline:test:             -         "contextWindow": 8192,
roo-cline:test:             -         "description": "Test model 1",
roo-cline:test:             -         "maxTokens": 4096,
roo-cline:test:             -         "supportsPromptCache": false,
roo-cline:test:             +   "error": "Requesty API error",
roo-cline:test:             +   "success": false,
roo-cline:test:             +   "type": "singleRouterModelFetchResponse",
roo-cline:test:             +   "values": Object {
roo-cline:test:             +     "provider": "requesty",
roo-cline:test:                 },
roo-cline:test:             -     },
roo-cline:test:             -     "requesty": Object {},
roo-cline:test:             -     "unbound": Object {},
roo-cline:test:             -   },
roo-cline:test:             -   "type": "routerModels",
roo-cline:test:               },
roo-cline:test:            2
roo-cline:test:               Object {
roo-cline:test:             -   "routerModels": Object {
roo-cline:test:             -     "glama": Object {
roo-cline:test:             -       "model-1": Object {
roo-cline:test:             -         "contextWindow": 8192,
roo-cline:test:             -         "description": "Test model 1",
roo-cline:test:             -         "maxTokens": 4096,
roo-cline:test:             -         "supportsPromptCache": false,
roo-cline:test:             -       },
roo-cline:test:             -     },
roo-cline:test:             -     "litellm": Object {},
roo-cline:test:             -     "openrouter": Object {
roo-cline:test:             -       "model-1": Object {
roo-cline:test:             -         "contextWindow": 8192,
roo-cline:test:             -         "description": "Test model 1",
roo-cline:test:             -         "maxTokens": 4096,
roo-cline:test:             -         "supportsPromptCache": false,
roo-cline:test:             +   "error": "Unbound API error",
roo-cline:test:             +   "success": false,
roo-cline:test:             +   "type": "singleRouterModelFetchResponse",
roo-cline:test:             +   "values": Object {
roo-cline:test:             +     "provider": "unbound",
roo-cline:test:                 },
roo-cline:test:             -     },
roo-cline:test:             -     "requesty": Object {},
roo-cline:test:             -     "unbound": Object {},
roo-cline:test:             -   },
roo-cline:test:             -   "type": "routerModels",
roo-cline:test:               },
roo-cline:test:            3
roo-cline:test:               Object {
roo-cline:test:             -   "routerModels": Object {
roo-cline:test:             -     "glama": Object {
roo-cline:test:             -       "model-1": Object {
roo-cline:test:             -         "contextWindow": 8192,
roo-cline:test:             -         "description": "Test model 1",
roo-cline:test:             -         "maxTokens": 4096,
roo-cline:test:             -         "supportsPromptCache": false,
roo-cline:test:             -       },
roo-cline:test:             -     },
roo-cline:test:             -     "litellm": Object {},
roo-cline:test:             -     "openrouter": Object {
roo-cline:test:             -       "model-1": Object {
roo-cline:test:             -         "contextWindow": 8192,
roo-cline:test:             -         "description": "Test model 1",
roo-cline:test:             -         "maxTokens": 4096,
roo-cline:test:             -         "supportsPromptCache": false,
roo-cline:test:             +   "error": "LiteLLM connection failed",
roo-cline:test:             +   "success": false,
roo-cline:test:             +   "type": "singleRouterModelFetchResponse",
roo-cline:test:             +   "values": Object {
roo-cline:test:             +     "provider": "ollama",
roo-cline:test:                 },
roo-cline:test:             -     },
roo-cline:test:             -     "requesty": Object {},
roo-cline:test:             -     "unbound": Object {},
roo-cline:test:             -   },
roo-cline:test:             -   "type": "routerModels",
roo-cline:test:               },
roo-cline:test: 
roo-cline:test:     Number of calls: 5
roo-cline:test: 
roo-cline:test:       183 |
roo-cline:test:       184 |             // Verify successful providers are included
roo-cline:test:     > 185 |             expect(mockClineProvider.postMessageToWebview).toHaveBeenCalledWith({
roo-cline:test:           |                                                            ^
roo-cline:test:       186 |                     type: "routerModels",
roo-cline:test:       187 |                     routerModels: {
roo-cline:test:       188 |                             openrouter: mockModels,
roo-cline:test: 
roo-cline:test:       at Object.<anonymous> (core/webview/__tests__/webviewMessageHandler.test.ts:185:50)

It looks like the tests assume there are only 5 providers but it is not clear to me how to add more.

daniel-lxs · 2025-06-09T12:57:39Z

Hi @thecolorblue! I can help you fix the failing test. The issue is that your PR added two new providers (ollama and lmstudio) to the router system, but the test "handles individual provider failures gracefully" is still expecting only the original 5 providers.

Here's how to fix it:

In src/core/webview/__tests__/webviewMessageHandler.test.ts, around line 172-178:

Update the mock setup to handle 7 providers instead of 5:

// Mock some providers to succeed and others to fail
mockGetModels
    .mockResolvedValueOnce(mockModels) // openrouter
    .mockRejectedValueOnce(new Error("Requesty API error")) // requesty
    .mockResolvedValueOnce(mockModels) // glama
    .mockRejectedValueOnce(new Error("Unbound API error")) // unbound
    .mockRejectedValueOnce(new Error("LiteLLM connection failed")) // litellm
    .mockRejectedValueOnce(new Error("Ollama connection failed")) // ollama - ADD THIS
    .mockRejectedValueOnce(new Error("LMStudio connection failed")) // lmstudio - ADD THIS

Around line 187-193, update the expected routerModels:

routerModels: {
    openrouter: mockModels,
    requesty: {},
    glama: mockModels,
    unbound: {},
    litellm: {},
    ollama: {},      // ADD THIS
    lmstudio: {},    // ADD THIS
},

Around line 216, add expectations for the new provider error messages:

expect(mockClineProvider.postMessageToWebview).toHaveBeenCalledWith({
    type: "singleRouterModelFetchResponse",
    success: false,
    error: "Ollama connection failed",
    values: { provider: "ollama" },
})

expect(mockClineProvider.postMessageToWebview).toHaveBeenCalledWith({
    type: "singleRouterModelFetchResponse",
    success: false,
    error: "LMStudio connection failed",
    values: { provider: "lmstudio" },
})

The test was failing because it expected 5 provider calls but received 7. With these changes, the test should pass!

thecolorblue · 2025-06-09T19:56:40Z

@daniel-lxs Thank you for walking me through it. I'll try again tonight and let you know how it goes.

thecolorblue · 2025-06-10T03:44:43Z

@daniel-lxs looks like that worked. Let me know if there is anything else I can do.

thecolorblue · 2025-06-14T02:19:52Z

@daniel-lxs checking in on this one... I see there are some conflicts. Should I take care of those now? Is there anything else I can do to keep this moving?

daniel-lxs · 2025-06-14T02:33:07Z

Hey @thecolorblue , sorry I'll take a look soon, I can fix the conflicts if necessary, don't worry

…odel `contextLength`

thecolorblue · 2025-06-19T16:34:37Z

@mrubens I believe all of the conflicts have been resolved. Let me know if you see anything else that needs to be addressed.

daniel-lxs · 2025-06-19T16:49:37Z

Thank you @thecolorblue, I'll be taking a look soon!

daniel-lxs

Thank you @thecolorblue for addressing the review so quickly!

Looks good to me

mrubens

Looks good aside from @hannesrudolph's feedback on the fallback. Hannes want to make the call on whether to merge this or try to handle that case better first?

daniel-lxs · 2025-06-20T23:00:23Z

@mrubens @hannesrudolph

I added a check for models:

It doesn’t prevent the user from saving, but it does let them know if the model doesn’t exist.
The color you see is from my theme, but the error message should show up red-ish in most themes.

ptempier · 2025-06-22T08:00:40Z

Note sure if it was expected, but i access ollama via litellm, and the bug fix doesn t seems to work in this situation, i still get 200k context displayed , when it s in fact 24k. So roo won t condense, and context get truncated, functions forgotten, etc ...

…Inc#4314) Co-authored-by: Daniel Riccio <[email protected]>

thecolorblue · 2025-06-23T12:54:08Z

@ptempier are you running the ollama server locally? Did you select 'litellm' as the API provider?

ptempier · 2025-06-24T18:51:59Z

Hello
There's an ollama instance, which is accessed via litellm, which is accessed by roo code.
litellm is selected as api provided.
Ollama and littellm are remote on local network, but not local host.

Co-authored-by: Daniel Riccio <[email protected]>

thecolorblue requested review from cte and mrubens as code owners June 4, 2025 05:14

github-project-automation bot added this to Roo Code Roadmap Jun 4, 2025

github-project-automation bot moved this to Triage in Roo Code Roadmap Jun 4, 2025

github-project-automation bot added this to Roo Code Roadmap Jun 4, 2025

github-project-automation bot moved this to New in Roo Code Roadmap Jun 4, 2025

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working documentation Improvements or additions to documentation labels Jun 4, 2025

ellipsis-dev bot reviewed Jun 4, 2025

View reviewed changes

src/api/providers/fetchers/lmstudio.ts Outdated Show resolved Hide resolved

webview-ui/src/components/ui/hooks/useSelectedModel.ts Outdated Show resolved Hide resolved

webview-ui/src/components/ui/hooks/useSelectedModel.ts Outdated Show resolved Hide resolved

thecolorblue force-pushed the 2462-ollama-context branch from 67812b2 to d823b3d Compare June 4, 2025 12:53

daniel-lxs moved this from Triage to PR [Needs Prelim Review] in Roo Code Roadmap Jun 4, 2025

daniel-lxs added the Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. label Jun 4, 2025

hannesrudolph added PR - Needs Preliminary Review and removed Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. labels Jun 5, 2025

daniel-lxs requested changes Jun 7, 2025

View reviewed changes

src/api/providers/fetchers/ollama.ts Outdated Show resolved Hide resolved

src/api/providers/fetchers/lmstudio.ts Outdated Show resolved Hide resolved

src/api/providers/fetchers/ollama.ts Outdated Show resolved Hide resolved

daniel-lxs moved this from PR [Needs Prelim Review] to PR [Changes Requested] in Roo Code Roadmap Jun 7, 2025

daniel-lxs added PR - Changes Requested and removed PR - Needs Preliminary Review labels Jun 7, 2025

thecolorblue requested a review from jr as a code owner June 9, 2025 05:38

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jun 9, 2025

daniel-lxs moved this from PR [Changes Requested] to PR [Needs Prelim Review] in Roo Code Roadmap Jun 14, 2025

hannesrudolph removed the PR - Changes Requested label Jun 14, 2025

thecolorblue added 2 commits June 19, 2025 10:32

pull lm studio models from the list of loaded models; use lm studio m…

bde21d0

…odel `contextLength`

updated lm studio model description

12c8084

daniel-lxs moved this from PR [Needs Review] to PR [Needs Prelim Review] in Roo Code Roadmap Jun 19, 2025

hannesrudolph added PR - Needs Preliminary Review and removed PR - Needs Review labels Jun 19, 2025

thecolorblue added 2 commits June 19, 2025 12:00

updated testing for lm studio loaded models

0f042b9

better type checks on ollama context length

9b38f80

thecolorblue requested review from hannesrudolph and hoonsubin June 19, 2025 16:23

hoonsubin approved these changes Jun 19, 2025

View reviewed changes

daniel-lxs approved these changes Jun 19, 2025

View reviewed changes

daniel-lxs moved this from PR [Needs Prelim Review] to PR [Needs Review] in Roo Code Roadmap Jun 19, 2025

hannesrudolph added PR - Needs Review and removed PR - Needs Preliminary Review labels Jun 19, 2025

mrubens approved these changes Jun 20, 2025

View reviewed changes

feat: add model availability checks for LMStudio and Ollama components

efd8357

mrubens merged commit 37ed013 into RooCodeInc:main Jun 21, 2025
17 of 18 checks passed

github-project-automation bot moved this from PR [Needs Review] to Done in Roo Code Roadmap Jun 21, 2025

github-project-automation bot moved this from New to Done in Roo Code Roadmap Jun 21, 2025

Alorse pushed a commit to Alorse/Roo-Code that referenced this pull request Jun 22, 2025

fix context length for lmstudio and ollama (RooCodeInc#2462) (RooCode…

f2d4ae0

…Inc#4314) Co-authored-by: Daniel Riccio <[email protected]>

daniel-lxs mentioned this pull request Jun 23, 2025

fix: improve LM Studio model detection to show all downloaded models #5047

Merged

cte pushed a commit that referenced this pull request Jun 24, 2025

fix context length for lmstudio and ollama (#2462) (#4314)

20478b9

Co-authored-by: Daniel Riccio <[email protected]>

chrarnoldus mentioned this pull request Jun 30, 2025

Local Models Fail to Invoke Tools & Mismanage Context Across Calls Kilo-Org/kilocode#927

Closed

ItsOnlyBinary mentioned this pull request Sep 4, 2025

Task display of max context window broken with ollama #7674

Closed

fix context length for lmstudio and ollama (#2462) #4314

fix context length for lmstudio and ollama (#2462) #4314

Uh oh!

Conversation

thecolorblue commented Jun 4, 2025 • edited by ellipsis-dev bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related GitHub Issue

Description

Test Procedure

Type of Change

Pre-Submission Checklist

Screenshots / Videos

Documentation Updates

Get in Touch

Uh oh!

Uh oh!

Uh oh!

Uh oh!

daniel-lxs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thecolorblue commented Jun 9, 2025

Uh oh!

daniel-lxs commented Jun 9, 2025

Uh oh!

thecolorblue commented Jun 9, 2025

Uh oh!

thecolorblue commented Jun 10, 2025

Uh oh!

thecolorblue commented Jun 14, 2025

Uh oh!

daniel-lxs commented Jun 14, 2025

Uh oh!

thecolorblue commented Jun 19, 2025

Uh oh!

daniel-lxs commented Jun 19, 2025

Uh oh!

daniel-lxs left a comment

Choose a reason for hiding this comment

Uh oh!

mrubens left a comment

Choose a reason for hiding this comment

Uh oh!

daniel-lxs commented Jun 20, 2025

Uh oh!

Uh oh!

ptempier commented Jun 22, 2025

Uh oh!

thecolorblue commented Jun 23, 2025

Uh oh!

ptempier commented Jun 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

thecolorblue commented Jun 4, 2025 •

edited by ellipsis-dev bot

Loading