MCP embedded resource metadata is not passed to model

[tradeqvest]

Question

Hello!

When returning an embedded resource from an MCP Tool using the MCPServerStreamableHTTP, pydantic ai currently only returns the text due to this code.

However, information besides the text may be interesting for further processing.

embedded_resource = EmbeddedResource(
                    type="resource",
                    resource=TextResourceContents(
                        uri=f"resource://{to_uri(result.id)}",
                        text=result.document.text,
                        mimeType="text/plain",
                        _meta=result.document.metadata or {},
                    ),
                    _meta={
                        "relevance_score": round(result.score, 3),
                        "document_last_modified": document.metadata.get(
                            "modifiedDate", None
                        ),
                    },
                )

In this example, only the text would be returned, which may be insufficient. Is this behavior intended for embedded resources? Other clients from fastmcp return all available content. I feel that at least having control over this behavior would be very beneficial.

Thank you in advance for your support.

Additional Context

No response

[DouweM]

@tradeqvest Do I understand correctly that you'd like the relevance_score and document_last_modified to be passed to the model alongside the resource text? Or are you intending to use this data in code?

[tradeqvest]

Yes, that would be one aspect. Even more important would be all the content in the resource itself, e.g. uri and metadata. Right now, I do not see a way of providing this information to the model nor obtaining it for usage in code.

[DouweM]

@tradeqvest OK, so the entire EmbeddedResource should just be serialized as JSON and sent to the model like that? With the full object available in result.all_messages()?

What do you think we should do when the MCP server returns a ResourceLink, which we're currently handling similarly by looking up the resource and sending just that text/binary to the model, losing any metadata:

pydantic-ai/pydantic_ai_slim/pydantic_ai/mcp.py

Lines 286 to 295 in fc6a2b2

	elif isinstance(part, mcp_types.EmbeddedResource):
	resource = part.resource
	return self._get_content(resource)
	elif isinstance(part, mcp_types.ResourceLink):
	resource_result: mcp_types.ReadResourceResult = await self._client.read_resource(part.uri)
	return (
	self._get_content(resource_result.contents[0])
	if len(resource_result.contents) == 1
	else [self._get_content(resource) for resource in resource_result.contents]
	)

In that case, the model wouldn't know what to do with the the resource URI unless it has a read_resource tool (which is not provided by default -- should it be?).

The handling of directly returned TextContent, ImageContent, and AudioContent also currently loses the metadata, as we pass it to the model as BinaryContent:

pydantic-ai/pydantic_ai_slim/pydantic_ai/mcp.py

Lines 270 to 285 in fc6a2b2

	if isinstance(part, mcp_types.TextContent):
	text = part.text
	if text.startswith(('[', '{')):
	try:
	return pydantic_core.from_json(text)
	except ValueError:
	pass
	return text
	elif isinstance(part, mcp_types.ImageContent):
	return messages.BinaryContent(data=base64.b64decode(part.data), media_type=part.mimeType)
	elif isinstance(part, mcp_types.AudioContent):
	# NOTE: The FastMCP server doesn't support audio content.
	# See <https://github.com/modelcontextprotocol/python-sdk/issues/952> for more details.
	return messages.BinaryContent(
	data=base64.b64decode(part.data), media_type=part.mimeType
	) # pragma: no cover

If we wanted to include the metadata there, we'd need to pass it in a separate "user" message.

Do you know how other AI libraries that act as MCP clients handle this? Unfortunately LLM APIs don't have a native way of providing metadata alongside a file.

[ivo-1]

Hi @DouweM,

@tradeqvest and I are working together :) Just recently this became relevant again, so I wanted to reply.

I don't have answers to all of your questions but at least some more info.

Do you know how other AI libraries that act as MCP clients handle this?

I checked FastMCP's implementation. They just return list[ContentBlock] for unstructured content and dict for structured content. See here.

Maybe there could be a flag to disable _map_tool_result() when initializing an MCPServer?

pydantic-ai/pydantic_ai_slim/pydantic_ai/mcp.py

Line 304 in 5968775

async def _map_tool_result_part(

I think that could be a great addition for anyone who wants to deal with the "raw" CallToolResult as it comes from the MCP Python SDK itself but still wants to use pydantic-ai's MCP Client implementation. For example, in our case we rely on pydantic-ai's MCP Client implementation because we are using both PreparedToolSet and ProcessToolCallback.

I'm also happy to contribute the above functionality.

[DouweM]

I checked FastMCP's implementation. They just return list[ContentBlock] for unstructured content and dict for structured content. See here.

@ivo-1 That's FastMCP's server which translates a tool function return value into an MCP callTool response, so it makes sense that it's just passing it through, leaving it up to the client to do any processing. Their client also just passes the value through, as they don't concern themselves with translating it to something the LLM can consume. That's what we have to do and where it gets tricky, as the model won't understand a plain JSON dump but expects files in a specific format, and wouldn't be able to follow a ResourceLink until it also has a way to look up the resource.

Are you using Pydantic AI MCPServer standalone, similar to the FastMCP Client, by directly consuming the tool call results in Python? Or are you using it with an agent and expecting the results to be sent on the model automatically?

If it's the former, a flag on MCPServer.direct_call_tool to skip the result mapping would make sense, and you're welcome to contribute it. But when passing the result on to the model, I don't think the unprocessed data would be of much use to it.

[ivo-1]

@DouweM thanks for elaborating. I think I understand better now how the concerns of FastMCP's client and Pydantic AI MCP Client differ.

To your question: We are using it with agents.

I still think that if the MCP Server builder explicitly configures the return of a given tool to be EmbeddedResource with _meta fields explicitly set that there could be an expectation that the agent sees that information. Similar to how it works for when defining the output with pydantic BaseModel which result in the tool returning structured_content with a bunch of fields where the agent will also see all of them. But I think it's something we should just wait and see what people's expectations are over time.

For now, we will try to simply adjust the tool output as in our case we have control over that. Thanks again!

[DouweM]

I still think that if the MCP Server builder explicitly configures the return of a given tool to be EmbeddedResource with _meta fields explicitly set that there could be an expectation that the agent sees that information.

@ivo-1 I agree. Unfortunately LLM APIs don't have a native way of providing metadata alongside a file.

We could have an opt-in flag on MCPServer to send the metadata in a user message after the actual tool response (which itself may already be broken up into a "See file X" tool response followed by This is file X: + the multi-modal content as LLM APIs don't support tools directly returning files).

That could be done by having MCPServer.call_tool return a ToolReturn object that can contain content parts alongside the actual return value: https://ai.pydantic.dev/tools/#advanced-tool-returns.

I don't know when I'd get to that though, a contribution would be appreciated :)

[ivo-1]

@DouweM thanks for elaborating! We have fixed the issue for now by changing around the tool output on MCP side. We actually also noticed that we weren't using the Resource primitive correctly, i.e. the URI was meant as a link to retrieve the full document of a chunk but according to the spec the URI has to be of the resource itself!

So for now it's not urgent nor blocking on our side.