JSON schema concerns: annotations and descriptions sometimes ignored

[gvanrossum-ms]

Initial Checks

• I confirm that I'm using the latest version of Pydantic AI
• I confirm that I searched for my issue in https://github.com/pydantic/pydantic-ai/issues before opening this issue

Description

I have a rather complex schema with many optional parts. The schema represents "knowledge" to be extracted from natural language, with many options, e.g. entities, facets, actions, dates and times, and more. I have this schema working pretty well with TypeChat but for various reasons I'm looking to replace that with Pydantic AI, if I can get it to work well.

While trying to make this work, using Azure OpenAI, I found two issues:

• The descriptions and docstrings I put in the schema using Field(description="..."_, although I see they are passed along quite deep in the code, don't appear to affect the model's behavior. I need to repeat the same info in the prompt text. Is this expected?
• Sometimes the model ignores the type annotations. E.g. a field annotated with DateTime (a struct defined in the same file) was given a string value (a date/time formatted using the ISO standard) -- causing your validation to fail (I'm glad the validation worked, it would have been harder to debug in my own code :-).

Are these just known issues with the models? Is there something that could be done about it? Does Pydantic AI attempt to retry the request with an additional section in the prompt indicating the problem? (LLMs can often improve their behavior -- at the cost of extra tokens, for sure.)

I'm totally happy to update my prompts if this is the way it's supposed to be, but both of these surprised me and appear to violate the promise of using type annotations in the schema. (Maybe I didn't read the docs end to end -- but who does. :-)

Example Code

Python, Pydantic AI & LLM client version

Python 3.12.11, pydantic_ai_slim 0.4.9, pydantic_core 2.33.2, openai 1.95.1.

[DouweM]

@gvanrossum-ms Hi Guido! Gaaf dat je Pydantic AI aan het uitproberen bent :)

The descriptions and docstrings I put in the schema using Field(description="..."_, although I see they are passed along quite deep in the code, don't appear to affect the model's behavior. I need to repeat the same info in the prompt text. Is this expected?

That's not expected, but not completely surprising either if the schema is very large, as models can miss little details buried among tons of JSON schema.

Note that we only pass along Pydantic BaseModel-subclass class-level docstrings and Field(description=...) descriptions, not field-level docstrings. (Although that's something I personally think we should probably do as well.)

The easiest way to verify that the descriptions are making it to the API (and thus presumably the model), and to see what's going on behind the scenes in general, would be to add our observability tool Pydantic Logfire to the project as per these docs, instrument Pydantic AI to see the agent runs and tool calls, and instrument httpx to see the raw HTTP requests and responses:

pip install logfire[httpx]
logfire auth

import logfire

logfire.configure()
logfire.instrument_pydantic_ai()
logfire.instrument_httpx(capture_al=True)

(You'll be asked to create a free account.)

If you're registering your Pydantic model using Agent(output_type=...) or using it in a tool function argument, you should see it among "Tools available" in the chat <model name> span. You can expand that to see the full definition, which should include the description.

If you're using the Native Output mode mode instead of the default Tool Output mode, it'll be under the same chat <model name> span -> Details -> Arguments > model_request_parameters -> output_object.

You can double check what they look like in the actual API request as well.

If the descriptions not showing up there, it would be good to see some code that helps us reproduce the issue.

Sometimes the model ignores the type annotations. E.g. a field annotated with DateTime (a struct defined in the same file) was given a string value (a date/time formatted using the ISO standard) -- causing your validation to fail (I'm glad the validation worked, it would have been harder to debug in my own code :-).

This usually means that some part of the JSON schema generated for your Pydantic model was not compatible with OpenAI's strict mode (if it uses any features not listed under their supported schemas), which Pydantic AI will detect and handle by leaving that flag False by default. Without strict mode, JSON schema adherence is up to the model and not strictly enforced.

You'll be able to see the strict key in Logfire as well, under the chat <model name> span > Details > Arguments > model_request_parameters > output_tools.

Looking into this with a plain timestamp: datetime.datetime field (assuming that's similar enough to your DataTime struct) led me to find two bugs that you were likely also running into:

• we were treating the format key in the resulting "timestamp": { "format": "date-time", "type": "string" } schema as strict-incompatible, but it looks like that is (now) supported in strict mode per their "Supported schemas" docs
  • Fix: Allow string format, pattern and others in OpenAI strict JSON mode #2420
• we were treating an entire BaseModel as strict-incompatible if it didn't explicitly specify model_config = ConfigDict(extra='forbid'), as strict mode doesn't work with optional fields, even though it's unlikely that you really wanted extra fields
  • Fix: Let more BaseModels use OpenAI strict JSON mode by defaulting to additionalProperties=False #2419

I also found a third bug:

• Include default values in tool arguments JSON schema #2418

I plan to get all three merged and released today.

With those fixes, I'd expect your example to fare better. If not, it would be useful to see the DateTime struct definition and the resulting JSON schema (from Logfire).

Note that even if the schema is determined to be strict-incompatible, you can force strict mode to be used and have Pydantic AI automatically modify the JSON schema to drop anything unsupported, by using Agent(output_type=ToolOutput(..., strict=True)) (or NativeOutput(..., strict=True) if you're using that).

Does Pydantic AI attempt to retry the request with an additional section in the prompt indicating the problem? (LLMs can often improve their behavior -- at the cost of extra tokens, for sure.)

Yep, the validation error will be sent back to the model, for invalid tool calls and output alike: https://ai.pydantic.dev/tools/#tool-retries

Note that this does not apply when you're using Agent.run_stream, as that will start streaming as soon as it sees something that could be the final output, and will never go back to the model once it's done that. You can use Agent.iter instead: https://ai.pydantic.dev/agents/#iterating-over-an-agents-graph

[gvanrossum-ms]

Thanks for the long response and fixing those bugs!!

I'm still figuring how to make the best use of logfire, but I found one annoying thing: the model supports some output that contains the phrase 'legal secretary', and this causes the entire output to be scrubbed by default because it contains the substring 'secret'. Is that really necessary?

[DouweM]

@gvanrossum-ms Not necessary, just out of an abundance of caution. You can disable scrubbing entirely or provide your own scrub callback: https://logfire.pydantic.dev/docs/how-to-guides/scrubbing/

Let me know how that goes, any additional feedback on Pydantic AI and Logfire is much appreciated!

[Viicos]

Note that we only pass along Pydantic BaseModel-subclass class-level docstrings and Field(description=...) descriptions, not field-level docstrings. (Although that's something I personally think we should probably do as well.)

It is possible to do so using the use_attribute_docstrings configuration, although I believe this has to be set by end users explicitly on their models due to configuration boundaries.

Regarding scrubbing, I think we brought this internally not long ago. With LLM outputs, it is going to be way more common to match the default Logfire scrubbing settings, so maybe this should be tweaked.

[samuelcolvin]

@gvanrossum-ms thanks so much for trying Pydantic AI and Logfire!

Just to put together what @DouweM and @Viicos said above, the following code should give you what you want.

Here is a public trace showing what I see running this example.

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "logfire[httpx]",
#     "pydantic-ai-slim[openai]",
# ]
# ///
from datetime import date

import logfire
from pydantic import BaseModel
from pydantic_ai import Agent

logfire.configure(scrubbing=False)
logfire.instrument_pydantic_ai()
logfire.instrument_httpx(capture_all=True)


class Person(BaseModel, use_attribute_docstrings=True):
    name: str
    """The name of the person."""
    dob: date
    """The date of birth of the person."""
    city: str
    """The city where the person lives."""
    secret: str
    """The secret information about the person."""


person_agent = Agent(
    'openai:gpt-4.1',
    output_type=Person,
    instructions='Extract information about the person',
)
result = person_agent.run_sync("Samuel lived in London and was born on Jan 28th '87 his password is 'sausage'")
print(repr(result.output))

[gvanrossum]

Wow! Thanks for all the help. I have been in near-continuous meetings since Douwe's first reply, but I am now ready to try many of the suggestions.

One question to @samuelcolvin about that example: just out of curiosity, how does Pydantic extract those "field docstrings"? It's a very nice notation, but IIRC the bytecode won't retain them. Do you use inspect.get_source()?

[Viicos]

Do you use inspect.get_source()?

That is exactly it, and this is why we document that it requires the source code to be accessible for this feature to work. You can check how it is implemented here (we are also leveraging the __firstlineno__ class attribute since Python 3.13, making it possible to differentiate classes with identical names within the same file).

[gvanrossum-ms]

FYI Here is a link to my schemas (and the rest of the code -- though most of the code in the same subpackage is unused, except for generic_query.py and prompts.py): https://github.com/gvanrossum-ms/TypeAgent/blob/2babf7a7447997821b99e08fe8e2d6d7cb3a2566/python/ta/typeagent/pydai/search_query_schema.py

I got ToolOutput(..., strict=True) to work, but not NativeOutput -- still that same error. I tried removing the initializations and replacing DateTime with str in DateRange to no avail.

My next project is adding the Field(description=...) and class docstrings back and see how that goes.

UPDATE: The code that actually drives this is in `../knowpro/searchlang.py, at the very end.

[DouweM]

@gvanrossum-ms Thanks for sharing the code, that helps a lot.

This test script (pieced together from a couple places in that repo) uses NativeOutput(SearchQuery, strict=True) and seems to work well:

from typing import Literal

import logfire
from pydantic import BaseModel

from pydantic_ai import Agent, NativeOutput
from pydantic_ai.models.openai import OpenAIModel

logfire.configure()
logfire.instrument_pydantic_ai()
logfire.instrument_httpx(capture_all=True)


class DateVal(BaseModel):
    day: int
    month: int
    year: int


class TimeVal(BaseModel):
    hour: int
    minute: int
    seconds: int


class DateTime(BaseModel):
    date: DateVal
    time: TimeVal | None = None


class DateTimeRange(BaseModel):
    start_date: DateTime
    stop_date: DateTime | None = None


class FacetTerm(BaseModel):
    facet_name: str
    facet_value: str


class EntityTerm(BaseModel):
    name: str
    is_name_pronoun: bool | None
    type: list[str] | None = None
    facets: list[FacetTerm] | None = None


class VerbsTerm(BaseModel):
    words: list[str]
    tense: Literal['Past', 'Present', 'Future']


class ActionTerm(BaseModel):
    action_verbs: VerbsTerm | None = None
    actor_entities: list[EntityTerm] | Literal['*'] = '*'
    target_entities: list[EntityTerm] | None = None
    additional_entities: list[EntityTerm] | None = None
    is_informational: bool = False


class SearchFilter(BaseModel):
    action_search_term: ActionTerm | None = None
    entity_search_terms: list[EntityTerm] | None = None
    search_terms: list[str] | None = None
    time_range: DateTimeRange | None = None


class SearchExpr(BaseModel):
    rewritten_query: str
    filters: list[SearchFilter]


class SearchQuery(BaseModel):
    search_expressions: list[SearchExpr]


BIG_PROMPT = """\
Convert the user's natural language question into a structured SearchQuery.
Follow the schema documentation exactly.

CRITICAL RULES:
1. Set rewritten_query to the original question with minor corrections (fix typos, remove "please")
   but NEVER omit meaningful phrases or time references!
2. For time ranges like "first 15 minutes", ALWAYS include BOTH start_date AND stop_date!

REQUIRED STRUCTURES:
- EntityTerm: {"name": "...", "is_name_pronoun": false, "type": ["book"], "facets": null}
- VerbsTerm: {"words": ["mention"], "tense": "Past"}
- ActionTerm: {"action_verbs": {"words": ["referenced"], "tense": "Past"},
              "actor_entities": "*", "target_entities": [entity], "is_informational": false}
- time_range: {"start_date": {"date": {"day": 1, "month": 4, "year": 2020},
                                "time": {"hour": 9, "minute": 0, "seconds": 0}},
                 "stop_date": {"date": {"day": 1, "month": 4, "year": 2020},
                                "time": {"hour": 9, "minute": 30, "seconds": 0}}}

FIELD USAGE:
- entity_search_terms: tangible things (people, places, books, etc.)
- search_terms: abstract concepts not covered by entities/actions
- action_search_term: when asking about actions/interactions
- facets: for entity properties like [{{"facet_name": "publication_year", "facet_value": "2008"}}]

ENTITY NAME RULES:
- Use name="*" when asking for "all books", "all movies", etc. to match any entity of that type
- Use specific names like "Asimov", "Harry Potter" for particular entities

ACTION PATTERNS:
- "What did X say about Y?" → action_verbs: ["say"], actor_entities: [X], target_entities: [Y]
- "Summarize X's thoughts to Y?" → action_verbs: ["summarize", "explain", "say"], actor_entities: [X], target_entities: [Y]
- "Who was that X we talked about?" → entity_search_terms: [EntityTerm(name="*", type=["person"])], search_terms: ["X"]
- "How did X get referenced?" → action_verbs: ["referenced"], actor_entities: "*", target_entities: [X]
- "Who mentioned X?" → action_verbs: ["mentioned"], actor_entities: "*", target_entities: [X]

IMPORTANT: Questions like "How did X get [verb]?" should use action_search_term with the verb!

TIME RANGE RULE:
time_range is ONLY for message timestamps, NOT content attributes!
- For relative times like "first 15 minutes", use the start of the CONVERSATION TIME RANGE as the reference.
- USE: "first 15 minutes", "messages between 2-3pm"
- DON'T USE: "published in 2008" (use entity facets instead)
"""

model = OpenAIModel('gpt-4o')
agent = Agent(model, instructions=BIG_PROMPT, output_type=NativeOutput(SearchQuery, strict=True), retries=2)

print(agent.run_sync('Please give me the names of books mentioned in the first 15 minutes'))

Without strict=True, it needed 3 tries before it gave valid output as initially it was wrapping the entire response in a properties key, seemingly misinterpreting that key from the JSON schema. Since retries=2 I would expect it to not get to a valid response at all on some runs -- I think that's the error you're referring to? If you're getting some other error even with strict=True, a reproducible example would be great.

In this case, the entire schema is strict-compatible so we should be using strict=True automatically, I've fixed that:

• Automatically use OpenAI strict mode for strict-compatible native output types #2447

[gvanrossum-ms]

Aha! Your example works for me, but if I switch back to our team-specific Azure GPT-4o2 instance it doesn't! So the strictness problem is on our end -- we ought to deploy a newer version of GPT-4. Not sure how easy that is, so I may have to stick to ToolOutput, alas.

[gvanrossum-ms]

I forgot -- is there a way to use those "field docstrings" in BaseModel subclasses used with Pydantic AI? They make schema writing easier and the resulting schema looks more elegant.