problems implementing overloads on some flexible class methods

[cjw296]

Hi All,

I eventually managed to release a py.typed version of testfixtures, but to do so I had to give up on a bunch of overloads which I thought helped document intended methods of use:

simplistix/testfixtures@70283c2

I've tried to get my head around this a few times over the last few months:

• How do I type an overloaded function's keyword arguments that may not be specified?
• mypy complaining about calls to a datetime factory #1961

I even ended up logging a mypy bug:

• erroneous complaint that overloaded function implementation does not accept all possible arguments of signature

...and as well as several attempts myself, LLMs have also been struggling, even once the underlying typing was sorted out:

• Codex attempt 1
• Codex attempt 2
• Claude attempt 1
• Claude attempt 2

At a high level, I'm essentially looking to revert simplistix/testfixtures@70283c2 but with functioning type annotations, essentially providing these three calling patterns for the factory:

@overload
def mock_datetime(
        tzinfo: TZInfo | None = None,
        delta: float | None = None,
        delta_type: str = 'seconds',
        date_type: type[date] = date,
        strict: bool = False
) -> type[MockDateTime]:
    ...


@overload
def mock_datetime(
        year: int,
        month: int,
        day: int,
        hour: int = ...,
        minute: int = ...,
        second: int = ...,
        microsecond: int = ...,
        tzinfo: TZInfo | None = None,
        delta: float | None = None,
        delta_type: str = 'seconds',
        date_type: type[date] = date,
        strict: bool = False
) -> type[MockDateTime]:
    ...


@overload
def mock_datetime(
        default: datetime,
        tzinfo: TZInfo | None = None,
        delta: float | None = None,
        delta_type: str = 'seconds',
        date_type: type[date] = date,
        strict: bool = False
) -> type[MockDateTime]:
    ...


@overload
def mock_datetime(
        default: None,  # explicit None positional
        tzinfo: TZInfo | None = None,
        delta: float | None = None,
        delta_type: str = 'seconds',
        date_type: type[date] = date,
        strict: bool = False
) -> type[MockDateTime]:
    ...


def mock_datetime(
        *args,
        tzinfo: TZInfo | None = None,

...and then these overload patterns for the methods for manipulating the queue of upcoming mock datetimes:

class MockDateTime(MockedCurrent, datetime):

    @overload
    @classmethod
    def add(
            cls,
            year: int,
            month: int,
            day: int,
            hour: int = ...,
            minute: int = ...,
            second: int = ...,
            microsecond: int = ...,
            tzinfo: TZInfo = ...,
    ) -> None:
        ...

    @overload
    @classmethod
    def add(
            cls,
            instance: datetime,
    ) -> None:
        ...

    @classmethod
    def add(cls, *args, **kw):
        """
@@ -170,29 +147,6 @@ def add(cls, *args, **kw):
        """
        return super().add(*args, **kw)

    @overload
    @classmethod
    def set(
            cls,
            year: int,
            month: int,
            day: int,
            hour: int = ...,
            minute: int = ...,
            second: int = ...,
            microsecond: int = ...,
            tzinfo: TZInfo = ...,
    ) -> None:
        ...

    @overload
    @classmethod
    def set(
            cls,
            instance: datetime,
    ) -> None:
        ...

    @classmethod
    def set(cls, *args, **kw):
        """
@@ -205,28 +159,6 @@ def set(cls, *args, **kw):
        """
        return super().set(*args, **kw)

    @overload
    @classmethod
    def tick(
            cls,
            days: float = ...,
            seconds: float = ...,
            microseconds: float = ...,
            milliseconds: float = ...,
            minutes: float = ...,
            hours: float = ...,
            weeks: float = ...,
    ) -> None:
        ...

    @overload
    @classmethod
    def tick(
            cls,
            delta: timedelta,  # can become positional-only when Python 3.8 minimum
    ) -> None:
        ...

    @classmethod
    def tick(cls, *args, **kw) -> None:
        """

...and similar across mock_datetime, mock_time and mock_date and the helper types they construct.

Is Python typing just "not there yet" or am I missing something? Happy to be shown the way, ecstatic if someone feels inspired to throw up a PR 😅

[erictraut]

Can you describe the problem you're seeing in more detail? You mentioned in the other thread that the ellipses make mypy unhappy. When I paste your code into a new document, add the missing import statements, and run mypy on it, I don't see any errors.

Normally, you'd need to supply real default argument values rather than using an ellipsis, but type checkers generally allow you to use an ellipsis as a stand-in for the actual value when you're writing overloads or stubs. It's still a good idea to provide the actual default argument values because it allows language servers to show users these values when they call the function. For example, here's what I see when I start typing a call to MockDateTime.set when using the pyright language server.

It would be better if you supplied the actual default values. I presume they are zero by default? If so, replace the ... with 0.

[cjw296]

@hauntsaninja - thanks, your branch is happy with the minimal examples above, however, is this another related bug?

class Foo:
    pass

@overload
def foo(x: int) -> None: ...

@overload
def foo(*, x: Foo) -> None: ...

@overload
def foo(a: str, /) -> None: ...

def foo(*args: int | str, **kw: int | Foo) -> None:
    pass

...gives:

$ mypy demo.py 
demo.py:40: error: Overloaded function implementation does not accept all possible arguments of signature 1  [misc]

The key bit appears to be the | Foo on kw, so while the above is "more complete" (and relates to the TZInfo type from my real world problem), I can still reproduce the error with the following, more minimal:

class Foo:
    pass

@overload
def foo(x: int) -> None: ...

@overload
def foo(a: str, /) -> None: ...

def foo(*args: int | str, **kw: int | Foo) -> None:
    pass

$ mypy demo.py 
demo.py:37: error: Overloaded function implementation does not accept all possible arguments of signature 1  [misc]

Interestingly, the | str does appear important to trigger the bug, as:

class Foo:
    pass

@overload
def foo(x: int) -> None: ...

@overload
def foo(*, x: Foo) -> None: ...

def foo(*args: int, **kw: int | Foo) -> None:
    pass

$ mypy demo.py 
Success: no issues found in 1 source file

[hauntsaninja]

Yeah, that corresponds to the comment in that diff (got merged faster than I expected). Let's try python/mypy#19619

[cjw296]

Okay, great, thanks, and now on to the next one(s):

class MockedCurrent:

    @classmethod
    def add(cls, *args: int | datetime, **kw: int | TZInfo | None) -> None:
        return None


class MockDateTime(MockedCurrent):

    @overload
    @classmethod
    def add(
            cls,
            year: int,
            month: int,
            day: int,
            hour: int = 0,
            minute: int = 0,
            second: int = 0,
            microsecond: int = 0,
            *,
            tzinfo: TZInfo | None = None,
            fold: int = 0
    ) -> None:
        ...

    @overload
    @classmethod
    def add(
            cls,
            instance: datetime,
            /
    ) -> None:
        ...

    @classmethod
    def add(cls, *args: int | datetime, **kw: int | TZInfo | None) -> None:
        return cls.add(*args, **kw)

On mypy master, I get:

demo.py:14: error: Signature of "add" incompatible with supertype "MockedCurrent"  [override]
demo.py:14: note:      Superclass:
demo.py:14: note:          @classmethod
demo.py:14: note:          def add(cls, *args: int | datetime, **kw: int | tzinfo | None) -> None
demo.py:14: note:      Subclass:
demo.py:14: note:          @overload
demo.py:14: note:          @classmethod
demo.py:14: note:          def add(cls, year: int, month: int, day: int, hour: int = ..., minute: int = ..., second: int = ..., microsecond: int = ..., *, tzinfo: tzinfo | None = ..., fold: int = ...) -> None
demo.py:14: note:          @overload
demo.py:14: note:          @classmethod
demo.py:14: note:          def add(cls, datetime, /) -> None
demo.py:42: error: Argument 1 to "add" of "MockDateTime" has incompatible type "*tuple[int | datetime, ...]"; expected "int"  [arg-type]
demo.py:42: error: Argument 2 to "add" of "MockDateTime" has incompatible type "**dict[str, int | tzinfo | None]"; expected "int"  [arg-type]
demo.py:42: error: Argument 2 to "add" of "MockDateTime" has incompatible type "**dict[str, int | tzinfo | None]"; expected "tzinfo | None"  [arg-type]

Why is the superclass considered incompatible? It can be called in exactly the same way as the subclass.

However, more confusing are the subsequent errors, what are those trying to communicate? Are they further bugs?

[cjw296]

Here's a minimal reproducer for the the confusing stuff above:

class Base:

    @overload
    def add(cls, x: int) -> None: ...

    @overload
    def add(cls, y: str, /) -> None: ...

    def add(cls, *args: int | str, **kw: int) -> None:
        return None


class Sample(Base):

    @overload
    def add(cls, x: int) -> None: ...

    @overload
    def add(cls, y: str, /) -> None: ...

    def add(cls, *args: int | str, **kw: int) -> None:
        return cls.add(*args, **kw)

demo.py:26: error: Argument 1 to "add" of "Sample" has incompatible type "*tuple[int | str, ...]"; expected "int"  [arg-type]

...although if this is a bug, I suspect there's a parallel one for **kw as shown in the previous comment...

[hauntsaninja]

I think both errors are legit:

Incompatible with supertype

This boils down to:

class A:
    def foo(self, *args: int, **kw: str) -> None: ...

class B(A):
    def foo(self, year: int, *, tzinfo: str) -> None: ...

which is unsound because you could do:

def calls_foo(a: A):
    a.foo(1, 2, 3, 4, 5, oops="A.foo handles these args, Liskov substitution principle implies B.foo should as well")

calls_foo(B())  # boom

pyright doesn't complain about the datetime example in your second last comment though, so maybe myself and mypy are missing something

Errors when calling superclass from your subclass

This is because only the overload types matter to the caller. The caller needs to prove to the type checker that the way it is calling the function matches the overloads.

pyright complains about this as well