log: Support structured values

[KodrAus]

Feature Request

Crates

tracing-log

Motivation

Large applications may use libraries or components that use a combination of the log crate and tracing. We can support better diagnostics in these applications by making the most of the shared feature-set between log and tracing. That means retaining as much structure as possible when pushing diagnostic data between them.

In rust-lang/log#328 we're tracking the progress of structured logging support in the log crate. One of its stabilization blockers is that it works nicely with tracing.

Proposal

This is a bit of an open question, we've got flexibility on the log side to tweak things as needed for better integration with tracing. Since tracing already makes some concessions to log support (by handling it explicitly in its event macros) I'm guessing we've got a little flexibility on the tracing side too.

I'm proposing we aim for zero-cost structured integration between log and tracing by leveraging the existing hook in the event macros and using the log::kv APIs to translate between them.

[KodrAus]

OT: @hawkw I've just created this issue and self-assigned it to signal that I'm working on sketching out some more details, but if you have a particular way you like to organize things here let me know and I'll follow that :)

[hawkw]

@KodrAus thanks, this is great! I've wanted to open a ticket for discussing this work for a while.

Since tracing already makes some concessions to log support (by handling it explicitly in its event macros) I'm guessing we've got a little flexibility on the tracing side too.

I'm definitely willing to make changes to tracing in order to get the best possible compatibility story — a big part of why the tracing Value trait and impls are relatively minimal was in order to reserve the ability to make changes to be compatible with log's KV API. I'd prefer to avoid breaking changes whenever possible, but if the best approach requires a breaking change, I'd be willing to consider it.

[CAD97]

#2048 is related, since it makes more things able to be dynamically imported.

The biggest hurdle AIUI for ingesting log's kvs into tracing is that tracing requires static FieldSet information. However... I think it would be possible to allow dynamic field names by something like:

enum FieldImpl {
    Static { fields: &'static [&'static str], index: usize, callsite: &'static dyn Callsite }
    Dynamic { name: &str, callsite: &dyn Callsite } // niches, via nullptr in Static.fields
}

This is necessarily a breaking change to handle non-'static information here, but shouldn't be too difficult of an ask along the other de-'static-ing #2048 is asking for; I'll see if I can add that in the next time I rebase that PR.

[ReDestroyDeR]

Hello! Been looking forward to integrate rust-lang/log with tracing in our new project.

Started digging into the problems of why converting Record.key_values(&self) to tracing Event's is not trivial.

What digging exactly? Tried to do it myself 😢

Then stumbled upon this issue and #2048 PR.

Would this feature be implemented into tracing-log? Or is there any other solution available?

Unfortunately, log crate does not suport spans, and it is a critical functionality for async applications without which the quality of observability is quite lacking. Hope that this issue will get much more deserved attention.