GBQ query getting error of '>' not supported between instances of 'int' and 'object' randomly

[moveingsun]

hi folks, we have a service that run hundreds of thousands GBQ query each day using python client. And recently I've notice the GBQ client fail randomly with error msg "'>' not supported between instances of 'int' and 'object' randomly", which seems to be an issue on client code. And the error just appear randonly. I've opened a support ticket in GCP console, but the conversation there get us nothing and i have to report the issue here, can anyone help me to understand what could cause this issue and how to get it fixed?

Environment details

• OS type and version: Rocky Linux 8
• Python version: 3.9.15
• pip version: 22.2
• google-cloud-bigquery version: 3.10.0 and above

Steps to reproduce

Unable to reproduce, happen randomly in our code.

Code example

try:
    job = self.gbq_client.query(cmd, job_config=job_config, job_id=self.job_id, timeout=30)
    self.session_id = job._properties["jobReference"].get("jobId", self.session_id)
    self._result = job
    job.result()
    failure = job.exception()
    if failure:
        raise QueryExecutionError(failure)
except BadRequest as e:
    raise QueryExecutionError(str(e))
except Exception as e:
    raise e

Stack trace

 File "/pyenv/versions/worker-k8s/lib/python3.9/site-packages/db_core/drivers/gbq_driver.py", line 317, in _execute
    failure = job.exception()
  File "/pyenv/versions/worker-k8s/lib/python3.9/site-packages/google/api_core/future/polling.py", line 282, in exception
    self._blocking_poll(timeout=timeout)
  File "/pyenv/versions/worker-k8s/lib/python3.9/site-packages/google/cloud/bigquery/job/query.py", line 1257, in _blocking_poll
    super(QueryJob, self)._blocking_poll(timeout=timeout, **kwargs)
  File "/pyenv/versions/worker-k8s/lib/python3.9/site-packages/google/api_core/future/polling.py", line 137, in _blocking_poll
    polling(self._done_or_raise)(retry=retry)
  File "/pyenv/versions/worker-k8s/lib/python3.9/site-packages/google/api_core/retry.py", line 349, in retry_wrapped_func
    return retry_target(
  File "/pyenv/versions/worker-k8s/lib/python3.9/site-packages/google/api_core/retry.py", line 191, in retry_target
    return target()
  File "/pyenv/versions/worker-k8s/lib/python3.9/site-packages/google/cloud/bigquery/job/query.py", line 1379, in _done_or_raise
    self._reload_query_results(retry=retry, timeout=transport_timeout)
  File "/pyenv/versions/worker-k8s/lib/python3.9/site-packages/google/cloud/bigquery/job/query.py", line 1360, in _reload_query_results
    self._query_results = self._client._get_query_results(
  File "/pyenv/versions/worker-k8s/lib/python3.9/site-packages/google/cloud/bigquery/client.py", line 1895, in _get_query_results
    timeout = max(timeout, _MIN_GET_QUERY_RESULTS_TIMEOUT)
TypeError: '>' not supported between instances of 'int' and 'object'

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[tswast]

Thanks for the report. This appears to be a duplicate of this issue: #1459 due to a breaking change introduced in google-api-core package.

This should have been fixed in version 3.10 and later of this package, though.

[moveingsun]

actually with version google-cloud-bigquery==3.11.3 and google-api-core-2.11.1. the issue happens more freqnently. for reference, with google-cloud-bigquery==3.11.3 and google-api-core-2.11.1, it happen around 240 times a day, with google-cloud-bigquery==3.10 and google-api-core-2.11.1, our log showed it only happen 2 times a day. we have to downgrade the version to 3.10 to reduce the issue frequency. so it seems a different issue?

[moveingsun]

@tswast can you provide some update?

[phongphung]

for me,
calling job.execution() for job.result() before query completed will cause the error,
work around with checking query_job.running() to wait until job finish then continue work for now

[tswast]

Thanks for checking in with the versions. It sounds like there is some part of the breaking change in google-api-core that hasn't fully been fixed in google-cloud-bigquery.

CC @chalmerlowe

[shollyman]

Yeah, looks like this is another callsite where the new default object representation isn't handled. Another change similar to #1541 is needed for

python-bigquery/google/cloud/bigquery/client.py

Lines 1897 to 1898 in 3e021a4

	if timeout is not None:
	timeout = max(timeout, _MIN_GET_QUERY_RESULTS_TIMEOUT)

[meredithslota]

Googlers, the internal support issue is b/290415719 for additional context. @moveingsun I'm sorry you didn't see a resolution through support, it would have been faster if they had routed us your ticket internally, but that's neither here nor there now (we'll suggest some process changes internally, ha).