multistore_file raises errno.EDEADLK when trying to open credential store

[thobrla]

multistore_file claims to be process and thread safe, but if multiple threads try to refresh credentials simultaneously, it can result in an exception like the stack trace at the bottom of this issue.

multistore_file catches errno.ENOSYS and errno.ENOLCK already. The simple fix would be to add EDEADLK to this list, since multiple threads opening the file can definitely result in this error. Additionally, it might be useful to add an actual Python lock to protect this access, which would help a lot when threads (or, optionally, child processes) contend for this resource on startup.

Encountered exception while copying:

Traceback (most recent call last):
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/gslib/command.py", line 1707, in PerformTask
    results = task.func(cls, task.args, thread_state=self.thread_gsutil_api)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/gslib/copy_helper.py", line 1796, in _PerformSlicedDownloadObjectToFile
    end_byte=args.end_byte))
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/gslib/copy_helper.py", line 2185, in _DownloadObjectToFileResumable
    digesters=digesters, progress_callback=progress_callback)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/gslib/cloud_api_delegator.py", line 225, in GetObjectMedia
    serialization_data=serialization_data, digesters=digesters)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 694, in GetObjectMedia
    serialization_data=serialization_data)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 716, in _PerformResumableDownload
    end_byte=end_byte, serialization_data=serialization_data)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/gslib/gcs_json_api.py", line 772, in _PerformDownload
    use_chunks=False)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 459, in GetRange
    additional_headers=additional_headers)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/transfer.py", line 396, in __GetChunk
    retries=self.num_retries)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 325, in MakeRequest
    http, http_request, e, retry, max_retry_wait))
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 316, in MakeRequest
    check_response_func=check_response_func)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/http_wrapper.py", line 365, in _MakeRequestNoRetry
    redirections=redirections, connection_type=connection_type)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/gslib/gcs_json_media.py", line 377, in NewRequest
    connection_type=connection_type)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/util.py", line 137, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/client.py", line 541, in new_request
    self._refresh(request_orig)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/apitools/apitools/base/py/credentials_lib.py", line 308, in _refresh
    oauth2client.client.OAuth2Credentials._refresh(self, do_request)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/client.py", line 771, in _refresh
    self.store.acquire_lock()
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/multistore_file.py", line 224, in acquire_lock
    self._multistore._lock()
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/multistore_file.py", line 284, in _lock
    self._file.open_and_lock()
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/locked_file.py", line 374, in open_and_lock
    self._opener.open_and_lock(timeout, delay)
  File "/usr/local/share/google/google-cloud-sdk/platform/gsutil/third_party/oauth2client/oauth2client/locked_file.py", line 209, in open_and_lock
    fcntl.lockf(self._fh.fileno(), fcntl.LOCK_EX)
IOError: [Errno 35] Resource deadlock avoided

[nathanielmanistaatgoogle]

Thanks for the report, we would definitely be receptive to an implementation of your suggested fixes, and today I learned that Python's multiprocessing module contains a Lock class. :-)

[thobrla]

The file-locking notion is still useful for dealing with multiple applications executing simultaneously.

Using a multiprocessing lock is feasible, though it would probably need to come from the calling application. A thread lock would be easy, though. This pattern worked well elsewhere in gsutil:

multistore_lock = threading.CreateLock()

def UseProcessSafeLock(multiprocessing_lock):
  global multistore_lock
  multistore_lock = multiprocessing_lock