Skip to content

Conversation

@daniel-sanche
Copy link
Contributor

Sometimes, a log may fail as part of a larger batch of logs (eg, due to payload size). These exceptions can be hard to track down, because batches happen asynchronously on a background thread

This PR attempts to help resolve the issue by adding a truncated copy of the failing log to the batch log exception message

  File "router.py", line 86, in pubsub_callback
    found_func(**kwargs)
  File "/app/snippets.py", line 81, in batch_large_log
    batch.commit()
  File "/app/python-logging/google/cloud/logging_v2/logger.py", line 470, in commit
    raise e
  File "/app/python-logging/google/cloud/logging_v2/logger.py", line 465, in commit
    client.logging_api.write_entries(entries, partial_success=True, **kwargs)
  File "/app/python-logging/google/cloud/logging_v2/_gapic.py", line 160, in write_entries
    self._gapic_api.write_log_entries(request=request)
  File "/app/python-logging/google/cloud/logging_v2/services/logging_service_v2/client.py", line 725, in write_log_entries
    metadata=metadata,
  File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 288, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: 400 Log entry with size 263.8K exceeds maximum size of 256.0K: StructEntry(log_name=None, labels=None, insert_id=None, severity='DEFAULT', http_request=None, timestamp=None, resource=Resource(type='global', labels={}), trace=None, span_id=None, trace_sampled=None, source_location=None, operation=None, logger=None, payload={'message': 'simple_log', 'buffer': '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000... [type_url: "type.googleapis.com/google.logging.v2.WriteLogEntriesPartialErrors"
value: "\nA\010\007\022=\010\003\0229Log entry with size 263.8K exceeds maximum size of 256.0K"
, detail: "[ORIGINAL ERROR] generic::invalid_argument: Log entry with size 263.8K exceeds maximum size of 256.0K [google.rpc.error_details_ext] { message: \"Log entry with size 263.8K exceeds maximum size of 256.0K\" details { [type.googleapis.com/google.logging.v2.WriteLogEntriesPartialErrors] { log_entry_errors { key: 7 value { code: 3 message: \"Log entry with size 263.8K exceeds maximum size of 256.0K\" } } } } }"
]

Fixes: #53

@product-auto-label product-auto-label bot added size: s Pull request size is small. api: logging Issues related to the googleapis/python-logging API. labels Oct 12, 2022
Copy link
Contributor

@losalex losalex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets some comments

@product-auto-label product-auto-label bot added size: m Pull request size is medium. and removed size: s Pull request size is small. labels Oct 13, 2022
@daniel-sanche daniel-sanche changed the title [DRAFT] feat: include context on batch log errors feat: include context on batch log errors Oct 17, 2022
@daniel-sanche daniel-sanche marked this pull request as ready for review October 17, 2022 17:38
@daniel-sanche daniel-sanche requested review from a team as code owners October 17, 2022 17:38
@daniel-sanche daniel-sanche added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 17, 2022
@yoshi-kokoro yoshi-kokoro removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Oct 17, 2022
@daniel-sanche daniel-sanche merged commit d08be9a into main Oct 18, 2022
@daniel-sanche daniel-sanche deleted the detailed_large_log_errors branch October 18, 2022 18:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: logging Issues related to the googleapis/python-logging API. size: m Pull request size is medium.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve handling of "Log entry with size X exceeds maximum size"

4 participants