[Feature Request] Log Forwarding Support for python-SDK

[samuelmakarovskiy]

Is your feature request related to a problem? Please describe.

Have noticed an interesting behavior that arose when all instances of a temporal server are brought down after a python worker has connected. The worker infinite retries polling to reconnect which is fine, but the real issue is that the console logs for the underlying rust core are not formatted like the python worker logic ones. See below:

�[2m2023-04-13T14:17:45.292663Z�[0m �[33m WARN�[0m �[1mtemporal_client::retry�[0m�[2m:�[0m gRPC call poll_workflow_task_queue retried 119 times �[3merror�[0m�[2m=�[0mStatus { code: Unavailable, message: "error trying to connect: dns error: failed to lookup address information: Name or service not known", source: Some(tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("dns error", Custom { kind: Uncategorized, error: "failed to lookup address information: Name or service not known" })))) }

Fixing this would allow me to also guarantee any core issues can be detected by monitoring unified logs and not just stdout

Describe the solution you'd like

Couple of things would work

1. Could turn off pretty printing for the rust core logs so that at least I can make my application stdout logs match more reliably
2. The sdk could support passing back core logs to python like it does for typescript (and maybe other sdks). I see it's not supported in the code, but maybe worth clarifying why?

Additional context

[cretz]

Could turn off pretty printing for the rust core logs so that at least I can make my application stdout logs match more reliably

This can be disabled by setting RUST_LOG_COLORS=never env var I believe. Ref https://docs.rs/env_logger/latest/env_logger/#disabling-colors. (can probably also make it clear somehow on the tty that it does not support colors, e.g. TERM=dumb)

I see it's not supported in the code, but maybe worth clarifying why?

The only real reason is you're the first to need it :-) This is something we can add I think. I will leave this issue open on the backlog and we will look into implementing it.

[samuelmakarovskiy]

This can be disabled by setting RUST_LOG_COLORS=never env var I believe. Ref https://docs.rs/env_logger/latest/env_logger/#disabling-colors. (can probably also make it clear somehow on the tty that it does not support colors, e.g. TERM=dumb)

Didn't seem to work, I think the rust core uses tokio-rs/tracing

And from this thread appears the config requires the filter to have set_ansi on it, defaults to true

[cretz]

I see! I opened temporalio/sdk-core#529 to let that be disabled at runtime.

[jmdacruz]

Somewhat related to this topic, it would be nice to have more control on the structure of the Rust logging. One scenario is using structured (JSON) logging on the Python side, which currently gets mixed with the Rust log lines (as @samuelmakarovskiy shared on the description). Structured logging is desired in certain scenarios, like Kubernetes deployments where the infrastructure takes containers stdout/stderr and forwards that to some logging infrastructure.

Edit: @cretz clarified on Slack that "log forwarding" in this context means forwarding to the Python side, which would effectively solve the problem

[cretz]

After further discussion, we are going to wait on temporalio/sdk-core#618 and have buffered, pushed logs be sent to Python logging.