[Feature] Allow metrics to return multiple scores, and allow optimizers like GEPA to support multi-score optimization

[LakshyAAAgrawal]

What feature would you like to see?

Many tasks have multiple objectives which may be complementary, or a trade-off. How can we allow metrics to return such subscores, and have optimizers optimize them?

This will require thinking about:

1. What is the right interface for the user-defined metric to return multiple scores
2. How can the user specify how different sub-scores should be aggregated? Arithmetic/Geometric mean, weighted avg, etc.
3. How should scores across instances be aggregated?

Would you like to contribute?

• Yes, I'd like to help implement this.
• No, I just want to request it.

Additional Context

Relevant discussion: https://discord.com/channels/1161519468141355160/1161519469319946286/1407399169517420718

Upstream GEPA issue: gepa-ai/gepa#2

[LakshyAAAgrawal]

gepa-ai/gepa#47 and gepa-ai/gepa#2

[MatsErdkamp]

[copy/paste of the Discord response]

Finally got some time to get back around to this. Here's my current thinking.

Every custom score function should be usable.

If we implement something like this, we might as well go all the way: no half-measures where only weighted sums are supported. Any arbitrary formula should be valid.

Score should ideally be adapted to just take a function instead of adding a separate score_fn field.

For optimizers that don’t support evaluating the function we could just calculate the outcome and send that scalar to the optimizer.

Notation

As far as notation goes; I think there are three good sub-options.

a. String-based formulas with declared axes.

score = dspy.metrics.formula(
    axes={"acc": exact_match, "bleu": bleu_like},
    expr="0.7*acc + 0.3*bleu**2"
)

Concise, serializable, config/CLI-friendly.

b. Function-based single score_fn.

def score_fn(ax):
    acc  = ax.acc(exact_match)
    bleu = ax.bleu(bleu_like)
    return 0.7*acc + 0.3*bleu**2

score = dspy.metrics.custom(score_fn)

Pythonic, IDE-friendly, type-safe.

c. Composable builder-style.

score = (
    dspy.metrics.combo()
    .weighted({"acc": 0.7, "bleu": 0.3})
    .clip(min=-1, max=1)
    .penalize("len_ratio", factor=0.1, fn="abs_log")
    .build(axes={
        "acc": exact_match,
        "bleu": bleu_like,
        "len_ratio": length_ratio
    })
) 

We can provide often used score functions as utils.

Here are some (GPT-generated) examples. The implementation of this is dependant of which choice we make for the notation, of course:

score = dspy.metrics.geometric_mean({"acc": exact_match, "bleu": bleu_like})
score = dspy.metrics.harmonic_mean({"p": precision, "r": recall})
score = dspy.metrics.f_beta(precision=precision, recall=recall, beta=2.0)
score = dspy.metrics.minimax({"acc": exact_match, "coverage": coverage})
score = dspy.metrics.piecewise(
    axes={"acc": exact_match, "len_ratio": length_ratio},
    cases=[(sp.log(sp.Symbol("len_ratio")) < 0, -1), (True, sp.Symbol("acc"))]
)

Curious what you guys think. Maybe theres even better options! Please feel free to add them

[MatsErdkamp]

Started working on this here