[Feature] MCP bridge support

[James4Ever0]

The Feature

Act as a MCP bridge to utilize tools and provide enhanced response.

LiteLLM will read a MCP server config and act as a middle man between the MCP server, the LLM server and the LLM client.

There is an existing project for bridging MCP servers to OpenAI compatible clients.

Motivation, pitch

MCP is a general protocol for LLM Agents. By doing so MCP can be integrated into existing chat systems and have wider use cases.

Are you a ML Ops Team?

Yes

[turian]

I would love this

[ishaan-jaff]

added to March 2025 roadmap 🔥

[ishaan-jaff]

Hi @James4Ever0, @turian and everyone else on this thread. Initial implementation is done, is this what you wanted? #9426

I'd love your feedback on this as it's in Beta/ You can centrally define all MCP tools on the litellm config and your MCP clients can call list + call tool on litellm

(h/t @wagnerjt )

[forpr1093]

Hi @ishaan-jaff, thanks for your contribution to this. However, I think what @James4Ever0 is looking for is to have LiteLLM act as a bridge between the clients and the MCP Server, utilizing the tools from the existing MCP Server (similar to SecretiveShell/MCP-Bridge), rather than having LiteLLM function as an MCP Server itself. That said, this approach might still be useful in certain use cases.

[wagnerjt]

Thanks @ishaan-jaff for this. I will be testing it when I get a moment!

I just wanted to share with @forpr1093 and @James4Ever0 that a MCP-bridge solution would be really great for a single ease of use point for the various deployed MCP servers. The problem is there are a number of things that MCP provides (Resources, Tools, and Prompts) as well as how quickly the MCP specification is changing around the transport and the authentication that it is probably best to start small.

While this first PR is only a MCP server on top of the litellm proxy, another win would be to simply incorporate the MCP client for tools into the litellm sdk so it can bring in tools from various MCP-backed servers

[ishaan-jaff]

What do you think about this interface @wagnerjt @forpr1093. This is an OpenAI MCP bridge

LiteLLM Python SDK with MCP Tools

import asyncio
from litellm import experimental_createMCPClient, completion
from litellm.mcp_stdio import Experimental_StdioMCPTransport
from litellm import openai

async def main():
   client_one = None

   try:
       # Initialize an MCP client to connect to a `stdio` MCP server:
       transport = Experimental_StdioMCPTransport(
           command='node',
           args=['src/stdio/dist/server.js']
       )
       client_one = await experimental_createMCPClient(
           transport=transport
       )

      
       tool_set_one = await client_one.list_tools()
       tools = tool_set_one
       response = await litellm.completion(
           model="gpt-4o",
           tools=tools,
           messages=[
               {
                   "role": "user",
                   "content": "Find products under $100"
               }
           ]
       )

       print(response.text)
   except Exception as error:
       print(error)
   finally:
       await asyncio.gather(
           client_one.close() if client_one else asyncio.sleep(0),
       )

if __name__ == "__main__":
   asyncio.run(main())

LiteLLM Proxy with MCP Tools

import asyncio
from openai import OpenAI
from litellm import experimental_createMCPClient
from litellm.mcp_stdio import Experimental_StdioMCPTransport

async def main():
   client_one = None
   try:
       # Initialize an MCP client to connect to a `stdio` MCP server:
       transport = Experimental_StdioMCPTransport(
           command='node',
           args=['src/stdio/dist/server.js']
       )
       client_one = await experimental_createMCPClient(
           transport=transport
       )
       
       # Get tools from MCP client
       tool_set_one = await client_one.list_tools()
       tools = tool_set_one
       
       # Initialize OpenAI client with custom base URL
       openai_client = OpenAI(
           base_url="http://localhost:4000",
           api_key="your-api-key"
       )
       
       # Create completion with the tools
       response = openai_client.chat.completions.create(
           model="gpt-4o",
           tools=tools,
           messages=[
               {
                   "role": "user",
                   "content": "Find products under $100"
               }
           ]
       )
       
       print(response.choices[0].message.content)
   except Exception as error:
       print(error)
   finally:
       await asyncio.gather(
           client_one.close() if client_one else asyncio.sleep(0),
       )

if __name__ == "__main__":
   asyncio.run(main())

[rawwerks]

fwiw, vercel just added mcp to ai sdk, might be a good place to look for clues => https://github.com/search?q=repo%3Avercel%2Fai%20mcp&type=code

[ishaan-jaff]

Hi everyone here's our initial implementation of a MCP bridge with litellm python SDK. Is this what you wanted ? @wagnerjt @forpr1093 @turian @James4Ever0 ? #9436

Overview

LiteLLM acts as a MCP bridge to utilize MCP tools with all LiteLLM supported models. LiteLLM offers the following features for using MCP

• List Available MCP Tools: OpenAI clients can view all available MCP tools
  • litellm.experimental_mcp_client.load_mcp_tools to list all available MCP tools
• Call MCP Tools: OpenAI clients can call MCP tools
  • litellm.experimental_mcp_client.call_openai_tool to call an OpenAI tool on an MCP server

Usage

1. List Available MCP Tools

In this example we'll use litellm.experimental_mcp_client.load_mcp_tools to list all available MCP tools on any MCP server. This method can be used in two ways:

• format="mcp" - (default) Return MCP tools
  • Returns: mcp.types.Tool
• format="openai" - Return MCP tools converted to OpenAI API compatible tools. Allows using with OpenAI endpoints.
  • Returns: openai.types.chat.ChatCompletionToolParam

# Create server parameters for stdio connection
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
import os
import litellm
from litellm import experimental_mcp_client


server_params = StdioServerParameters(
    command="python3",
    # Make sure to update to the full absolute path to your math_server.py file
    args=["./mcp_server.py"],
)

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        # Initialize the connection
        await session.initialize()

        # Get tools
        tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")
        print("MCP TOOLS: ", tools)

        messages = [{"role": "user", "content": "what's (3 + 5)"}]
        llm_response = await litellm.acompletion(
            model="gpt-4o",
            api_key=os.getenv("OPENAI_API_KEY"),
            messages=messages,
            tools=tools,
        )
        print("LLM RESPONSE: ", json.dumps(llm_response, indent=4, default=str))

2. List and Call MCP Tools

In this example we'll use

• litellm.experimental_mcp_client.load_mcp_tools to list all available MCP tools on any MCP server
• litellm.experimental_mcp_client.call_openai_tool to call an OpenAI tool on an MCP server

The first llm response returns a list of OpenAI tools. We take the first tool call from the LLM response and pass it to litellm.experimental_mcp_client.call_openai_tool to call the tool on the MCP server.

How litellm.experimental_mcp_client.call_openai_tool works

• Accepts an OpenAI Tool Call from the LLM response
• Converts the OpenAI Tool Call to an MCP Tool
• Calls the MCP Tool on the MCP server
• Returns the result of the MCP Tool call

# Create server parameters for stdio connection
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
import os
import litellm
from litellm import experimental_mcp_client


server_params = StdioServerParameters(
    command="python3",
    # Make sure to update to the full absolute path to your math_server.py file
    args=["./mcp_server.py"],
)

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        # Initialize the connection
        await session.initialize()

        # Get tools
        tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")
        print("MCP TOOLS: ", tools)

        messages = [{"role": "user", "content": "what's (3 + 5)"}]
        llm_response = await litellm.acompletion(
            model="gpt-4o",
            api_key=os.getenv("OPENAI_API_KEY"),
            messages=messages,
            tools=tools,
        )
        print("LLM RESPONSE: ", json.dumps(llm_response, indent=4, default=str))

        openai_tool = llm_response["choices"][0]["message"]["tool_calls"][0]
        # Call the tool using MCP client
        call_result = await experimental_mcp_client.call_openai_tool(
            session=session,
            openai_tool=openai_tool,
        )
        print("MCP TOOL CALL RESULT: ", call_result)

        # send the tool result to the LLM
        messages.append(llm_response["choices"][0]["message"])
        messages.append(
            {
                "role": "tool",
                "content": str(call_result.content[0].text),
                "tool_call_id": openai_tool["id"],
            }
        )
        print("final messages with tool result: ", messages)
        llm_response = await litellm.acompletion(
            model="gpt-4o",
            api_key=os.getenv("OPENAI_API_KEY"),
            messages=messages,
            tools=tools,
        )
        print(
            "FINAL LLM RESPONSE: ", json.dumps(llm_response, indent=4, default=str)
        )

[ishaan-jaff]

docs here: https://docs.litellm.ai/docs/mcp

I'd love feedback on this from the litellm community !

Bonus - Contributor issue - Can we get help with this ?

Task: LiteLLM should maintain a json of all known MCP Servers, can we get help with a script that scrapes all servers here: https://github.com/modelcontextprotocol/servers/tree/main/src and stores as a json and add it here: https://github.com/BerriAI/litellm/blob/main/mcp_servers.json

The benefit of this is we can then allow litellm users to easily reference well known MCP servers

thoughts @wagnerjt @rawwerks @James4Ever0 ?

Each server can be stored as the following

{
    "brave-search": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "-e",
        "BRAVE_API_KEY",
        "mcp/brave-search"
      ],
      "env": {
        "BRAVE_API_KEY": "YOUR_API_KEY_HERE"
      }
    }
  }

[wagnerjt]

Hey @ishaan-jaff. I want to start to say that I have spent some time today to experiment with the mcp client so far. I will test out the other parts I was fumbling with and the proxy aspect next week when I have more time.

First, I really like that you went with making the MCP client config and passing in the session as an argument. This allows us to have the flexibility on the various transport layers (sse vs stdio) and security measure that will come in the future. Very nice 👏!

I tested a go based MCP server over sse with the mcp client and was able to load_mcp_tools but wasn't able to execute the call_openai_tool just yet. I can update the docs next week to show off how to connect via the sse transport.

All code with examples can be found here.

Side note for the script to maintain the MCP servers that are out there, maybe it is sufficient to add a link to https://mcp-get.com/ to the docs until the official mcp-registry comes through.

[wagnerjt]

Wanted to bump the thread that the new spec came out today! I'm still reviewing it myself but wanted to highlight the SSE -> http as well as the authentication. Once I get my hands on it more, I'll come with additional feedback and proposals

https://spec.modelcontextprotocol.io/specification/2025-03-26/

[ishaan-jaff]

Thanks for sharing the new spec @wagnerjt. If I'm not mistaken we'll need to wait for the mcp python SDK to support the new HTTP transport too