See eval reasons in report printed report

[brian-swantide]

Question

I'm running an eval against a dataset with LLMJudge as an evaluator. The table that is printed with this code:

 report = dataset.evaluate_sync(get_mongo_query)
 report.print(include_input=True, include_output=True)

does not include the evaluation reason explaining why the LLMJudge failed. I see in the LLMJudge code that include_reason=True so wondering if I'm missing some configuration or if there's some other way to see this output. I didn't see any documentation about this.

┏━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Case ID     ┃ Inputs             ┃ Outputs           ┃ Assertions ┃ Duration ┃
┡━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
│ simple_case │                    │ ...     │ ✔✗         │    16.1s │

Additional Context

pydantic_evale v0.2.6
python v3.12.3

[dmontagu]

The reasons are present in the EvaluationReport object if you dump it / interact with it programmatically, and are present in the produced spans (for viewing in Logfire or another OTel-compatible observability platform).

I didn't add them (yet 🙂) for console printing because I had a hard time imagining a console output that would include the reasons and also be useful. In particular, the console output today only has columns that include information about all evaluations of a particular variant (i.e., scores, labels, assertions). But on reflection we could probably produce a different console output with one table row per evaluation (rather than per case) where there would be enough room to include the reasons etc., and depending on your usage pattern would still be consumable.

While I've proposed a high-level idea here, it's still largely abstract, and is just my own thinking. One thing that would be helpful is if you could share a description (or better yet concrete rendering) of what you'd like to see in the console.

If you aren't sure how to access the information at all, start by replacing report.print(include_input=True, include_output=True) with print(repr(report)) — that should at least ensure all the information is dumped, though it may be hard to consume. You might also try print(report.model_dump_json(indent=2)) as a way to make the data more visually-parseable.

[brian-swantide]

Thanks for the thorough explanation! When I was thinking about it I was imagining something similar to what you proposed (separate table with a row per evaluation), but possibly filtering that table to only show evaluations that failed. So for every failed evaluation a more detailed view of the reasons.

[github-actions]

This issue is stale, and will be closed in 3 days if no reply is received.

[github-actions]

Closing this issue as it has been inactive for 10 days.

[dmontagu]

Closed by #2163