Higher-Kinded TypeVars

[tek]

aka type constructors, generic TypeVars

Has there already been discussion about those?
I do a lot of FP that results in impossible situations because of this. Consider an example:

A = TypeVar('A')
B = TypeVar('B')
F = TypeVar('F')

class M(Generic[F[X], A]):
    def x(fa: F[A], f: Callable[[A], B]) -> F[B]:
        return map(f, fa)

M().x([1], str)

I haven't found a way to make this work, does anyone know a trick or is it impossible?
If not, consider the syntax as a proposal.
Reference implementations would be Haskell, Scala.
optimally, the HK's type param would be indexable as well, allowing for F[X[X, X], X[X]]

---

Summary of current status (by @smheidrich, 2024-02-08):

• @JelleZijlstra has indicated interest in sponsoring a PEP, conditional on a prototype implementation in a major type checker and a well-specified draft PEP.
• Drafting the PEP takes place in @nekitdev's fork of the peps repo. The stub PEP draft so far contains a few examples of the proposed syntax.
• That same repo's GitHub Discussions forum forum has been designated as the place to discuss the PEP (and presumably the prototype implementation?). Some limited further discussions have taken place there.
  • If you want to be notified of new discussion threads, I think you have to set the whole repo as "watched" in GitHub?

[ilevkivskyi]

I think this came up few times in other discussions, for example one use case is python/mypy#4395. But TBH this is low priority, since such use cases are quite rare.

[tek]

damn, I searched very thoroughly but did not find this one! 😄
So, consider this my +1!

[gvanrossum]

Adding your +1 is all fine, but who's going to do the (probably pretty complicated) implementation work?

[tek]

are you approving the feature?

[gvanrossum]

I am neither approving nor disapproving. Just observing that it may be a lot of work for marginal benefits in most codebases.

[tek]

awfully pragmatic. Where's your sense of adventure? 😄
anyways, I'll work on it, though it's gonna take a while to get into the project.

[landonpoch]

Similar to microsoft/TypeScript#1213

Not sure if the discussion over there provides any useful insights to the effort over here.

[rcalsaverini]

Hi @tek. I'm also very interested in this, so I'd like to ask if you had any progress with this and volunteer to help if you want.

[tek]

@rcalsaverini sorry, I've been migrating my legacy code to haskell and am abandoning python altogether. but I wish you great success!

[rcalsaverini]

Oh, sad to hear but I see your point. Thanks.

[syastrov]

Just to add another use case (which I think relates to this issue):

Using Literal types along with overloading __new__, along with higher-kinded typevars could allow implementing a generic "nullable" ORM Field class, using a descriptor to provide access to the appropriate nullable-or-not field values. The descriptor wouldn't have to be reimplemented in subclasses.

It is one step closer to being possible due to the most recent mypy release's support for honoring the return type of __new__ (python/mypy#1020).

Note: this is basically a stripped-down version of Django's Field class:

# in stub file

from typing import Generic, Optional, TypeVar, Union, overload, Type
from typing_extensions import Literal

_T = TypeVar("_T", bound="Field")
_GT = TypeVar("_GT")

class Field(Generic[_GT]):
    # on the line after the overload: error: Type variable "_T" used with arguments
    @overload
    def __new__(cls: Type[_T], null: Literal[False] = False, *args, **kwargs) -> _T[_GT]: ...
    @overload
    def __new__(cls: Type[_T], null: Literal[True], *args, **kwargs) -> _T[Optional[_GT]]: ...
    def __get__(self, instance, owner) -> _GT: ...

class CharField(Field[str]): ...
class IntegerField(Field[int]): ...
# etc...

# in code

class User:
  f1 = CharField(null=False)
  f2 = CharField(null=True)

reveal_type(User().f1) # Expected: str
reveal_type(User().f2) # Expected: Union[str, None]