api_core: Add grpc.Channel stub to grpc_helpers #4705

theacodes · 2018-01-05T18:10:24Z

This is intended to help with testing all grpc-based clients. With this, we should be able to mostly avoid using mock/fakes/stubs and instead use this channel-based approach. This means that every layer of the client is tested and there is less concern about a mock/stub/fake not matching its upstream target. This is akin to mocking at the requests level for HTTP clients.

@dhermes: I considered putting this into a package like api_core.testing.grpc_stub, but I decided against it. I'm happy to reconsider.

api_core/google/api_core/grpc_helpers.py

+        if self.response is not None:
+            return self.response
+
+        raise ValueError(


api_core/google/api_core/grpc_helpers.py

+        callable, it will be invoked with the request protobuf. If it's an
+        exception, the exception will be raised when this is invoked.
+        """
+        self.responses = None


api_core/google/api_core/grpc_helpers.py

+    def __init__(self, method, channel):
+        self._method = method
+        self._channel = channel
+        self.response = None


api_core/google/api_core/grpc_helpers.py

+
+    @property
+    def requests(self):
+        """Sequence[Tuple[str, protobuf.Message]]: Returns a list of all


lukesneeringer

A couple relatively minor things, but in general this looks very good to me.

api_core/google/api_core/grpc_helpers.py

+        exception, the exception will be raised when this is invoked.
+        """
+        self.responses = None
+        """Iterator[protobuf.Message]: A iterator of responses. If specified,


api_core/google/api_core/grpc_helpers.py

+
+
+class _CallableStub(object):
+    """Stub for the grpc callable interface."""


api_core/google/api_core/grpc_helpers.py

+    Returns:
+        str: The simplified name of the method.
+    """
+    return method.rsplit('/', 1).pop()


api_core/google/api_core/grpc_helpers.py

+        request, timeout, metadata, and credentials."""
+
+    def __call__(self, request, timeout=None, metadata=None, credentials=None):
+        self._channel._requests.append((self._method, request))


api_core/google/api_core/grpc_helpers.py

+        self._channel._requests.append((self._method, request))
+        self.requests.append(request)
+        self.calls.append(
+            _ChannelCall(request, timeout, metadata, credentials))


api_core/tests/unit/test_grpc_helpers.py

+        with pytest.raises(ValueError) as exc_info:
+            stub.GetOperation(expected_request)
+
+        assert exc_info.match('GetOperation')


api_core/tests/unit/test_grpc_helpers.py

+
+        response1 = stub.GetOperation(expected_request)
+        response2 = stub.GetOperation(expected_request)
+        response3 = stub.GetOperation(expected_request)


api_core/tests/unit/test_grpc_helpers.py

+
+    def test_subscribe_unsubscribe(self):
+        channel = grpc_helpers.ChannelStub()
+        channel.subscribe(None)


api_core/google/api_core/grpc_helpers.py

+        """Sequence[Tuple[str, protobuf.Message]]: Returns a list of all
+        requests made on this channel in order. The tuple is of method name,
+        request proto."""
+        return self._requests


api_core/google/api_core/grpc_helpers.py

+
+    def __getattr__(self, key):
+        # Ideally this would only return stubs for known methods, however,
+        # grpc doesn't actually create these methods until they're needed.


api_core/google/api_core/grpc_helpers.py

-_ChannelCall = collections.namedtuple(
-    '_ChannelCall', ['request', 'timeout', 'metadata', 'credentials'])
+_MethodCall = collections.namedtuple(
+    '_MethodCall', ['request', 'timeout', 'metadata', 'credentials'])


api_core/google/api_core/grpc_helpers.py

-        if self.responses:
-            self.response = next(self.responses)
+        response = self.response
+        if response is None and self.responses is not None:


api_core/google/api_core/grpc_helpers.py

        channel_stub.GetFoo.responses = iter([
            foo_pb2.Foo(name='bar'),
-            foo_pb2.Foo(name='baz')])
+            foo_pb2.Foo(name='baz')


api_core/google/api_core/grpc_helpers.py

+        try:
+            return self._method_stubs[key]
+        except KeyError:
+            raise AttributeError


api_core/google/api_core/grpc_helpers.py

    def subscribe(self, callback, try_to_connect=False):
        """grpc.Channel.subscribe implementation."""
-        return
+        pass


api_core/tests/unit/test_grpc_helpers.py

-        def on_get_operation(request):
-            return expected_response
+        on_get_operation = mock.Mock(
+            spec=('__call__'), return_value=expected_response)


dhermes

LGTM

theacodes · 2018-01-05T22:56:09Z

Thanks, @dhermes and @lukesneeringer!

@dhermes, @tseaver, @chemelnucfin, @lukesneeringer: Please keep an eye out in code reviews for excessive mocking of gapic/grpc clients - where possible, this should be used instead. I'm happy to help write tests until we develop a good reference corpus.

api_core: Add grpc.Channel stub to grpc_helpers

8979ee0

theacodes added api: core testing labels Jan 5, 2018

theacodes requested review from dhermes and lukesneeringer January 5, 2018 18:10

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jan 5, 2018

Fix inconsistent casing

95c6c9a

lukesneeringer reviewed Jan 5, 2018

View reviewed changes

api_core/google/api_core/grpc_helpers.py

if self.response is not None:

return self.response

raise ValueError(

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

lukesneeringer reviewed Jan 5, 2018

View reviewed changes

api_core/google/api_core/grpc_helpers.py

def __init__(self, method, channel):

self._method = method

self._channel = channel

self.response = None

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

lukesneeringer reviewed Jan 5, 2018

View reviewed changes

api_core/google/api_core/grpc_helpers.py Outdated

@property

def requests(self):

"""Sequence[Tuple[str, protobuf.Message]]: Returns a list of all

This comment was marked as spam.

Sign in to view

This comment was marked as spam.

Sign in to view

lukesneeringer approved these changes Jan 5, 2018

View reviewed changes

dhermes reviewed Jan 5, 2018

View reviewed changes

Jon Wayne Parrott added 3 commits January 5, 2018 11:27

Fail on non-existing methods

a4b7123

Add namedtuple for channel requests

4b84e76

Address review comments

29c8c43

dhermes reviewed Jan 5, 2018

View reviewed changes

Tuples for the tuple god

9158c12

dhermes reviewed Jan 5, 2018

View reviewed changes

More tuples for the tuple god

3d342ae

dhermes approved these changes Jan 5, 2018

View reviewed changes

theacodes merged commit e9a5347 into master Jan 5, 2018

theacodes deleted the channel-mock branch January 5, 2018 22:54

tseaver mentioned this pull request Jan 22, 2018

Update unit tests to use 'api_core.grpc_helpers.ChannelStub' vs. custom mocks #4772

Closed



		class _CallableStub(object):
		"""Stub for the grpc callable interface."""

api_core: Add grpc.Channel stub to grpc_helpers #4705

api_core: Add grpc.Channel stub to grpc_helpers #4705

Uh oh!

Conversation

theacodes commented Jan 5, 2018

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

lukesneeringer left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.