I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
- Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
- Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them
TypeErrors. - Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the
.and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
- Find bugs sooner: This is self-evident, I believe.
- The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from a behavioral aspect) you require.
- Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
- Function annotations (PEP 3107).
- Stub files for built-in/user modules.
- Special
# type: typecomments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for# type: typecomments)
Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).
For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints
the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy, on the other hand, we get the following response:
(Python3)jimmi@jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).
Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__ attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi@jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the
PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
- You need to type check a module for which you do not want to directly alter the function signatures
- You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been
mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
- Mypy: as described here.
- PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
- typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:
class Counter(Dict[_T, int], Generic[_T]):
@overload
def __init__(self) -> None: ...
@overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
@overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.
Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Answer from Dimitris Fasarakis Hilliard on Stack Overflowhttps://blog.jonathanchun.com/2025/02/16/to-type-or-not-to-type/
I wrote this blog post as I've seen a lot of newer developers complain about Type hints and how they seem unnecessary. I tried to copy-paste a short excerpt from the blog post here but it kept detecting it as a question which is not allowed, so decided to leave it out.
I know there's plenty of content on this topic, but IMO there's still way too much untyped Python code!
What are type hints in Python 3.5? - Stack Overflow
implementation - Why does Python ignore type hints? - Programming Language Design and Implementation Stack Exchange
Why Type Hinting Sucks!
Why do we use type hints?
Videos
I would suggest reading PEP 483 and PEP 484 and watching this presentation by Guido on type hinting.
In a nutshell: Type hinting is literally what the words mean. You hint the type of the object(s) you're using.
Due to the dynamic nature of Python, inferring or checking the type of an object being used is especially hard. This fact makes it hard for developers to understand what exactly is going on in code they haven't written and, most importantly, for type checking tools found in many IDEs (PyCharm and PyDev come to mind) that are limited due to the fact that they don't have any indicator of what type the objects are. As a result they resort to trying to infer the type with (as mentioned in the presentation) around 50% success rate.
To take two important slides from the type hinting presentation:
Why type hints?
- Helps type checkers: By hinting at what type you want the object to be the type checker can easily detect if, for instance, you're passing an object with a type that isn't expected.
- Helps with documentation: A third person viewing your code will know what is expected where, ergo, how to use it without getting them
TypeErrors. - Helps IDEs develop more accurate and robust tools: Development Environments will be better suited at suggesting appropriate methods when know what type your object is. You have probably experienced this with some IDE at some point, hitting the
.and having methods/attributes pop up which aren't defined for an object.
Why use static type checkers?
- Find bugs sooner: This is self-evident, I believe.
- The larger your project the more you need it: Again, makes sense. Static languages offer a robustness and control that dynamic languages lack. The bigger and more complex your application becomes the more control and predictability (from a behavioral aspect) you require.
- Large teams are already running static analysis: I'm guessing this verifies the first two points.
As a closing note for this small introduction: This is an optional feature and, from what I understand, it has been introduced in order to reap some of the benefits of static typing.
You generally do not need to worry about it and definitely don't need to use it (especially in cases where you use Python as an auxiliary scripting language). It should be helpful when developing large projects as it offers much needed robustness, control and additional debugging capabilities.
Type hinting with mypy:
In order to make this answer more complete, I think a little demonstration would be suitable. I'll be using mypy, the library which inspired Type Hints as they are presented in the PEP. This is mainly written for anybody bumping into this question and wondering where to begin.
Before I do that let me reiterate the following: PEP 484 doesn't enforce anything; it is simply setting a direction for function annotations and proposing guidelines for how type checking can/should be performed. You can annotate your functions and hint as many things as you want; your scripts will still run regardless of the presence of annotations because Python itself doesn't use them.
Anyways, as noted in the PEP, hinting types should generally take three forms:
- Function annotations (PEP 3107).
- Stub files for built-in/user modules.
- Special
# type: typecomments that complement the first two forms. (See: What are variable annotations? for a Python 3.6 update for# type: typecomments)
Additionally, you'll want to use type hints in conjunction with the new typing module introduced in Py3.5. In it, many (additional) ABCs (abstract base classes) are defined along with helper functions and decorators for use in static checking. Most ABCs in collections.abc are included, but in a generic form in order to allow subscription (by defining a __getitem__() method).
For anyone interested in a more in-depth explanation of these, the mypy documentation is written very nicely and has a lot of code samples demonstrating/describing the functionality of their checker; it is definitely worth a read.
Function annotations and special comments:
First, it's interesting to observe some of the behavior we can get when using special comments. Special # type: type comments
can be added during variable assignments to indicate the type of an object if one cannot be directly inferred. Simple assignments are
generally easily inferred but others, like lists (with regard to their contents), cannot.
Note: If we want to use any derivative of containers and need to specify the contents for that container we must use the generic types from the typing module. These support indexing.
# Generic List, supports indexing.
from typing import List
# In this case, the type is easily inferred as type: int.
i = 0
# Even though the type can be inferred as of type list
# there is no way to know the contents of this list.
# By using type: List[str] we indicate we want to use a list of strings.
a = [] # type: List[str]
# Appending an int to our list
# is statically not correct.
a.append(i)
# Appending a string is fine.
a.append("i")
print(a) # [0, 'i']
If we add these commands to a file and execute them with our interpreter, everything works just fine and print(a) just prints
the contents of list a. The # type comments have been discarded, treated as plain comments which have no additional semantic meaning.
By running this with mypy, on the other hand, we get the following response:
(Python3)jimmi@jim: mypy typeHintsCode.py
typesInline.py:14: error: Argument 1 to "append" of "list" has incompatible type "int"; expected "str"
Indicating that a list of str objects cannot contain an int, which, statically speaking, is sound. This can be fixed by either abiding to the type of a and only appending str objects or by changing the type of the contents of a to indicate that any value is acceptable (Intuitively performed with List[Any] after Any has been imported from typing).
Function annotations are added in the form param_name : type after each parameter in your function signature and a return type is specified using the -> type notation before the ending function colon; all annotations are stored in the __annotations__ attribute for that function in a handy dictionary form. Using a trivial example (which doesn't require extra types from the typing module):
def annotated(x: int, y: str) -> bool:
return x < y
The annotated.__annotations__ attribute now has the following values:
{'y': <class 'str'>, 'return': <class 'bool'>, 'x': <class 'int'>}
If we're a complete newbie, or we are familiar with Python 2.7 concepts and are consequently unaware of the TypeError lurking in the comparison of annotated, we can perform another static check, catch the error and save us some trouble:
(Python3)jimmi@jim: mypy typeHintsCode.py
typeFunction.py: note: In function "annotated":
typeFunction.py:2: error: Unsupported operand types for > ("str" and "int")
Among other things, calling the function with invalid arguments will also get caught:
annotated(20, 20)
# mypy complains:
typeHintsCode.py:4: error: Argument 2 to "annotated" has incompatible type "int"; expected "str"
These can be extended to basically any use case and the errors caught extend further than basic calls and operations. The types you
can check for are really flexible and I have merely given a small sneak peak of its potential. A look in the typing module, the
PEPs or the mypy documentation will give you a more comprehensive idea of the capabilities offered.
Stub files:
Stub files can be used in two different non mutually exclusive cases:
- You need to type check a module for which you do not want to directly alter the function signatures
- You want to write modules and have type-checking but additionally want to separate annotations from content.
What stub files (with an extension of .pyi) are is an annotated interface of the module you are making/want to use. They contain
the signatures of the functions you want to type-check with the body of the functions discarded. To get a feel of this, given a set
of three random functions in a module named randfunc.py:
def message(s):
print(s)
def alterContents(myIterable):
return [i for i in myIterable if i % 2 == 0]
def combine(messageFunc, itFunc):
messageFunc("Printing the Iterable")
a = alterContents(range(1, 20))
return set(a)
We can create a stub file randfunc.pyi, in which we can place some restrictions if we wish to do so. The downside is that
somebody viewing the source without the stub won't really get that annotation assistance when trying to understand what is supposed
to be passed where.
Anyway, the structure of a stub file is pretty simplistic: Add all function definitions with empty bodies (pass filled) and
supply the annotations based on your requirements. Here, let's assume we only want to work with int types for our Containers.
# Stub for randfucn.py
from typing import Iterable, List, Set, Callable
def message(s: str) -> None: pass
def alterContents(myIterable: Iterable[int])-> List[int]: pass
def combine(
messageFunc: Callable[[str], Any],
itFunc: Callable[[Iterable[int]], List[int]]
)-> Set[int]: pass
The combine function gives an indication of why you might want to use annotations in a different file, they some times clutter up
the code and reduce readability (big no-no for Python). You could of course use type aliases but that sometime confuses more than it
helps (so use them wisely).
This should get you familiarized with the basic concepts of type hints in Python. Even though the type checker used has been
mypy you should gradually start to see more of them pop-up, some internally in IDEs (PyCharm,) and others as standard Python modules.
I'll try and add additional checkers/related packages in the following list when and if I find them (or if suggested).
Checkers I know of:
- Mypy: as described here.
- PyType: By Google, uses different notation from what I gather, probably worth a look.
Related Packages/Projects:
- typeshed: Official Python repository housing an assortment of stub files for the standard library.
The typeshed project is actually one of the best places you can look to see how type hinting might be used in a project of your own. Let's take as an example the __init__ dunders of the Counter class in the corresponding .pyi file:
class Counter(Dict[_T, int], Generic[_T]):
@overload
def __init__(self) -> None: ...
@overload
def __init__(self, Mapping: Mapping[_T, int]) -> None: ...
@overload
def __init__(self, iterable: Iterable[_T]) -> None: ...
Where _T = TypeVar('_T') is used to define generic classes. For the Counter class we can see that it can either take no arguments in its initializer, get a single Mapping from any type to an int or take an Iterable of any type.
Notice: One thing I forgot to mention was that the typing module has been introduced on a provisional basis. From PEP 411:
A provisional package may have its API modified prior to "graduating" into a "stable" state. On one hand, this state provides the package with the benefits of being formally part of the Python distribution. On the other hand, the core development team explicitly states that no promises are made with regards to the the stability of the package's API, which may change for the next release. While it is considered an unlikely outcome, such packages may even be removed from the standard library without a deprecation period if the concerns regarding their API or maintenance prove well-founded.
So take things here with a pinch of salt; I'm doubtful it will be removed or altered in significant ways, but one can never know.
** Another topic altogether, but valid in the scope of type-hints: PEP 526: Syntax for Variable Annotations is an effort to replace # type comments by introducing new syntax which allows users to annotate the type of variables in simple varname: type statements.
See What are variable annotations?, as previously mentioned, for a small introduction to these.
Adding to Jim's elaborate answer:
Check the typing module -- this module supports type hints as specified by PEP 484.
For example, the function below takes and returns values of type str and is annotated as follows:
def greeting(name: str) -> str:
return 'Hello ' + name
The typing module also supports:
- Type aliasing.
- Type hinting for callback functions.
- Generics - Abstract base classes have been extended to support subscription to denote expected types for container elements.
- User-defined generic types - A user-defined class can be defined as a generic class.
- Any type - Every type is a subtype of Any.
Type hinting is a relatively recent addition to Python, so older code, and still most code, doesn't use them at all. Annotated code still needs to work with that existing code, so Python is a gradual system where typed and untyped code mixes together in one program.
When they are used, for most types they are purely structural, saying that certain methods are available but not where they came from, but for some builtins at least (like int) they do convey concrete implementation aspects that could be optimised. There are cases where static type information could usefully help in optimising code execution, but perhaps not as many as you'd expect.
Relying on the annotations would also require that they be correct, which Python makes no guarantee of. Ensuring this correctness needs sound gradual typing, which requires run-time checking of types as well and carries a whole lot of run-time and semantic effects. It would be a really significant change to Python semantics to start enforcing these type annotations in that way, so the informational content of an annotation is even a little bit lower than it looks. In any case, the costs of the dynamic checking to make it sound enough to rely on almost certainly outweigh the performance benefits in almost all cases.
For a language like Python where code making use of very dynamic features and metaprogramming is fairly common, the benefit for most programs likely isn't high enough to be worth the costs no matter what. Even the long-standing nominal typing elements in Python (like collections.abc.Sequence) doesn't convey much about the actual implementation, and mutating classes at run time is not uncommon inside common Python frameworks. These objects need to be usable with all the type-hinted code, so that code needs to deal with a big variety of object shapes. This is all very different to what's typical in C++, where these things might happen at (or before) compile time, if anything.
Instead, just-in-time compilation is a more realistic performance enhancement to Python, which is also not included in the CPython implementation, but is in some alternative versions. For high-performance Python code, as used in some scientific computation, there are both libraries like numpy that keep everything inside, and specialised JIT systems like Numba that optimise specified functions at run time. These are more practical performance targets for Python.
Python's typing model is intended for outside checkers to check python code. Pretty much any expression can go in a type hint:
x: print("hi") = 1
print(x)
produces
hi
1
Obviously any typechecker will reject that as bogus, but the Python interpreter has to support it in order to fulfil the Python standard.
Leading on from that, there's no guarantee that the type hints are going to be correct, because by the Python standard, they don't have to be. This is also perfectly valid to an interpreter:
x: str = 1
As for why the Python standard has been designed this way, it's a very long story that I don't know enough of to recount here, but it comes down to "it doesn't fit Python's ideals as a language".
In terms of using types to speed something up, check out mypyc - it uses the fact that the code has to be correctly checked by mypy before it compiles code in order to speed up typed python code significantly.
For example, this would be sped up in exactly the manner you've described by mypyc, but it can't be by a normal Python interpreter:
def fib(n: int) -> int:
if n <= 1:
return n
return fib(n - 1) + fib(n - 2)
Type hints are great! But I was playing Devil's advocate on a thread recently where I claimed actually type hinting can be legitimately annoying, especially to old school Python programmers.
But I think a lot of people were skeptical, so let's go through a made up scenario trying to type hint a simple Python package. Go to the end for a TL;DR.
The scenario
This is completely made up, all the events are fictitious unless explicitly stated otherwise (also editing this I realized attempts 4-6 have even more mistakes in them than intended but I'm not rewriting this again):
You maintain a popular third party library slowadd, your library has many supporting functions, decorators, classes, and metaclasses, but your main function is:
def slow_add(a, b):
time.sleep(0.1)
return a + bYou've always used traditional Python duck typing, if a and b don't add then the function throws an exception. But you just dropped support for Python 2 and your users are demanding type hinting, so it's your next major milestone.
First attempt at type hinting
You update your function:
def slow_add(a: int, b: int) -> int:
time.sleep(0.1)
return a + bAll your tests pass, mypy passes against your personal code base, so you ship with the release note "Type Hinting Support added!"
Second attempt at type hinting
Users immediately flood your GitHub issues with complaints! MyPy is now failing for them because they pass floats to slow_add, build processes are broken, they can't downgrade because of internal Enterprise policies of always having to increase type hint coverage, their weekend is ruined from this issue.
You do some investigating and find that MyPy supports Duck type compatibility for ints -> floats -> complex. That's cool! New release:
def slow_add(a: complex, b: complex) -> complex:
time.sleep(0.1)
return a + bFunny that this is a MyPy note and not a PEP standard...
Third attempt at type hinting
Your users thank you for your quick release, but a couple of days later one user asks why you no longer support Decimal. You replace complex with Decimal but now your other MyPy tests are failing.
You remember Python 3 added Numeric abstract base classes, what a perfect use case, just type hint everything as numbers.Number.
Hmmm, MyPy doesn't consider any of integers, or floats, or Decimals to be numbers :(.
After reading through typing you guess you'll just Union in the Decimals:
def slow_add(
a: Union[complex, Decimal], b: Union[complex, Decimal]
) -> Union[complex, Decimal]:
time.sleep(0.1)
return a + bOh no! MyPy is complaining that you can't add your other number types to Decimals, well that wasn't your intention anyway...
More reading later and you try overload:
@overload
def slow_add(a: Decimal, b: Decimal) -> Decimal:
...
@overload
def slow_add(a: complex, b: complex) -> complex:
...
def slow_add(a, b):
time.sleep(0.1)
return a + b
But MyPy on strict is complaining that slow_add is missing a type annotation, after reading this issue you realize that @overload is only useful for users of your function but the body of your function will not be tested using @overload. Fortunately in the discussion on that issue there is an alternative example of how to implement:
T = TypeVar("T", Decimal, complex)
def slow_add(a: T, b: T) -> T:
time.sleep(0.1)
return a + bFourth attempt at type hinting
You make a new release, and a few days later more users start complaining. A very passionate user explains the super critical use case of adding tuples, e.g. slow_add((1, ), (2, ))
You don't want to start adding each type one by one, there must be a better way! You learn about Protocols, and Type Variables, and positional only parameters, phew, this is a lot but this should be perfect now:
T = TypeVar("T")
class Addable(Protocol):
def __add__(self: T, other: T, /) -> T:
...
def slow_add(a: Addable, b: Addable) -> Addable:
time.sleep(0.1)
return a + bA mild diversion
You make a new release noting "now supports any addable type".
Immediately the tuple user complains again and says type hints don't work for longer Tuples: slow_add((1, 2), (3, 4)). That's weird because you tested multiple lengths of Tuples and MyPy was happy.
After debugging the users environment, via a series of "back and forth"s over GitHub issues, you discover that pyright is throwing this as an error but MyPy is not (even in strict mode). You assume MyPy is correct and move on in bliss ignoring there is actually a fundamental mistake in your approach so far.
(Author Side Note - It's not clear if MyPy is wrong but it defiantly makes sense for Pyright to throw an error here, I've filed issues against both projects and a pyright maintainer has explained the gory details if you're interested. Unfortunately this was not really addressed in this story until the "Seventh attempt")
Fifth attempt at type hinting
A week later a user files an issue, the most recent release said that "now supports any addable type" but they have a bunch of classes that can only be implemented using __radd__ and the new release throws typing errors.
You try a few approaches and find this seems to best solve it:
T = TypeVar("T")
class Addable(Protocol):
def __add__(self: T, other: T, /) -> T:
...
class RAddable(Protocol):
def __radd__(self: T, other: Any, /) -> T:
...
@overload
def slow_add(a: Addable, b: Addable) -> Addable:
...
@overload
def slow_add(a: Any, b: RAddable) -> RAddable:
...
def slow_add(a: Any, b: Any) -> Any:
time.sleep(0.1)
return a + bAnnoyingly there is now no consistent way for MyPy to do anything with the body of the function. Also you weren't able to fully express that when b is "RAddable" that "a" should not be the same type because Python type annotations don't yet support being able to exclude types.
Sixth attempt at type hinting
A couple of days later a new user complains they are getting type hint errors when trying to raise the output to a power, e.g. pow(slow_add(1, 1), slow_add(1, 1)). Actually this one isn't too bad, you quick realize the problem is your annotating Protocols, but really you need to be annotating Type Variables, easy fix:
T = TypeVar("T")
class Addable(Protocol):
def __add__(self: T, other: T, /) -> T:
...
A = TypeVar("A", bound=Addable)
class RAddable(Protocol):
def __radd__(self: T, other: Any, /) -> T:
...
R = TypeVar("R", bound=RAddable)
@overload
def slow_add(a: A, b: A) -> A:
...
@overload
def slow_add(a: Any, b: R) -> R:
...
def slow_add(a: Any, b: Any) -> Any:
time.sleep(0.1)
return a + bSeventh attempt at type hinting
Tuple user returns! He says MyPy in strict mode is now complaining with the expression slow_add((1,), (2,)) == (1, 2) giving the error:
Non-overlapping equality check (left operand type: "Tuple[int]", right operand type: "Tuple[int, int]")
You realize you can't actually guarantee anything about the return type from some arbitrary __add__ or __radd__, so you starting throwing Any Liberally around:
class Addable(Protocol):
def __add__(self: "Addable", other: Any, /) -> Any:
...
class RAddable(Protocol):
def __radd__(self: "RAddable", other: Any, /) -> Any:
...
@overload
def slow_add(a: Addable, b: Any) -> Any:
...
@overload
def slow_add(a: Any, b: RAddable) -> Any:
...
def slow_add(a: Any, b: Any) -> Any:
time.sleep(0.1)
return a + bEighth attempt at type hinting
Users go crazy! The nice autosuggestions their IDE provided them in the previous release have all gone! Well you can't type hint the world, but I guess you could include type hints for the built-in types and maybe some Standard Library types like Decimal:
You think you can rely on some of that MyPy duck typing but you test:
@overload
def slow_add(a: complex, b: complex) -> complex:
...
And realize that MyPy throws an error on something like slow_add(1, 1.0).as_integer_ratio(). So much for that nice duck typing article on MyPy you read earlier.
So you end up implementing:
class Addable(Protocol):
def __add__(self: "Addable", other: Any, /) -> Any:
...
class RAddable(Protocol):
def __radd__(self: "RAddable", other: Any, /) -> Any:
...
@overload
def slow_add(a: int, b: int) -> int:
...
@overload
def slow_add(a: float, b: float) -> float:
...
@overload
def slow_add(a: complex, b: complex) -> complex:
...
@overload
def slow_add(a: str, b: str) -> str:
...
@overload
def slow_add(a: tuple[Any, ...], b: tuple[Any, ...]) -> tuple[Any, ...]:
...
@overload
def slow_add(a: list[Any], b: list[Any]) -> list[Any]:
...
@overload
def slow_add(a: Decimal, b: Decimal) -> Decimal:
...
@overload
def slow_add(a: Fraction, b: Fraction) -> Fraction:
...
@overload
def slow_add(a: Addable, b: Any) -> Any:
...
@overload
def slow_add(a: Any, b: RAddable) -> Any:
...
def slow_add(a: Any, b: Any) -> Any:
time.sleep(0.1)
return a + bAs discussed earlier MyPy doesn't use the signature of any of the overloads and compares them to the body of the function, so all these type hints have to manually validated as accurate by you.
Ninth attempt at type hinting
A few months later a user says they are using an embedded version of Python and it hasn't implemented the Decimal module, they don't understand why your package is even importing it given it doesn't use it. So finally your code looks like:
from __future__ import annotations
import time
from typing import TYPE_CHECKING, Any, Protocol, TypeVar, overload
if TYPE_CHECKING:
from decimal import Decimal
from fractions import Fraction
class Addable(Protocol):
def __add__(self: "Addable", other: Any, /) -> Any:
...
class RAddable(Protocol):
def __radd__(self: "RAddable", other: Any, /) -> Any:
...
@overload
def slow_add(a: int, b: int) -> int:
...
@overload
def slow_add(a: float, b: float) -> float:
...
@overload
def slow_add(a: complex, b: complex) -> complex:
...
@overload
def slow_add(a: str, b: str) -> str:
...
@overload
def slow_add(a: tuple[Any, ...], b: tuple[Any, ...]) -> tuple[Any, ...]:
...
@overload
def slow_add(a: list[Any], b: list[Any]) -> list[Any]:
...
@overload
def slow_add(a: Decimal, b: Decimal) -> Decimal:
...
@overload
def slow_add(a: Fraction, b: Fraction) -> Fraction:
...
@overload
def slow_add(a: Addable, b: Any) -> Any:
...
@overload
def slow_add(a: Any, b: RAddable) -> Any:
...
def slow_add(a: Any, b: Any) -> Any:
time.sleep(0.1)
return a + bTL;DR
Turning even the simplest function that relied on Duck Typing into a Type Hinted function that is useful can be painfully difficult.
Please always put on your empathetic hat first when asking someone to update their code to how you think it should work.
In writing up this post I learnt a lot about type hinting, please try and find edge cases where my type hints are wrong or could be improved, it's a good exercise.
Edit: Had to fix a broken link.
Edit 2: It was late last night and I gave up on fixing everything, some smart people nicely spotted the errors!
I have a "tenth attempt" to address these error. But pyright complains about it because my overloads overlap, however I don't think there's a way to express what I want in Python annotations without overlap. Also Mypy complains about some of the user code I posted earlier giving the error comparison-overlap, interestingly though pyright seems to be able to detect here that the types don't overlap in the user code.
I'm going to file issues on pyright and mypy, but fundamentally they might be design choices rather than strictly bugs and therefore a limit on the current state of Python Type Hinting:
T = TypeVar("T")
class SameAddable(Protocol):
def __add__(self: T, other: T, /) -> T:
...
class Addable(Protocol):
def __add__(self: "Addable", other: Any, /) -> Any:
...
class SameRAddable(Protocol):
def __radd__(self: T, other: Any, /) -> T:
...
class RAddable(Protocol):
def __radd__(self: "RAddable", other: Any, /) -> Any:
...
SA = TypeVar("SA", bound=SameAddable)
RA = TypeVar("RA", bound=SameRAddable)
@overload
def slow_add(a: SA, b: SA) -> SA:
...
@overload
def slow_add(a: Addable, b: Any) -> Any:
...
@overload
def slow_add(a: Any, b: RA) -> RA:
...
@overload
def slow_add(a: Any, b: RAddable) -> Any:
...
def slow_add(a: Any, b: Any) -> Any:
time.sleep(0.1)
return a + bI'm struggling to understand the point of type hints in Python. You can specify everything's type, but it is not enforced. So, it's all the work of statically typed languages without any of the errors?
Why not just use a strongly-typed language then?
Mypy (and PEP 484 in general) is designed so that in the most ideal case, you only need to add type annotations to the "boundaries" or "interfaces" of your code.
For example, you basically must add annotations/type metadata in the following places:
- The parameter and return types of functions and methods.
- Any object fields (assuming the types of your fields are not inferrable just by looking at your constructor)
- When you inherit a class. For example, if you specifically want to subclass a dict of ints to strs, you should do
class MyClass(Dict[int, str]): ..., notclass MyClass(dict): ....
These are all examples of "boundaries" of your code. Type hints on parameter/return types let the caller of the function make sure they're calling it correctly, type hints on fields let the caller know they're using the object correctly, etc...
Mypy (and other PEP 484 compliant tools) will then use that information and try to infer the types of everything else. This behavior is designed to roughly mimic how humans read code: once you know what types are being passed in, for example, it's usually pretty easy to understand what the rest of the code does.
After all, Python is a language that was designed from the start to be readable! We don't need to scatter type hints everywhere to enhance our understanding of what the code does.
Of course, mypy (and other PEP 484-compliant tools) aren't perfect, and sometimes they might not correctly infer what the type of some local variable will be. In that case, you might need to add a type hint to help mypy along. Ethan's answer gives a good overview of some common cases to watch out for. (Interestingly, these cases also tend to be examples of where a human reader might struggle to understand your code!)
So, to put everything together, the general recommendation is to:
- Add type hints to all of the "boundaries" of your code, like function parameters and return types.
- Default to not annotating variables. If mypy is unable to infer what type some variable should be, add an annotation to help it.
- If you find yourself needing to annotate lots of variables to make mypy happy, consider refactoring your code. If mypy is getting confused easily, a human reader is also likely to get confused easily.
So, to go back to your examples, you would not add type hints in either case. Both a human reader and mypy can tell that your _volume field must be a float: it's immediately obvious that must be the case since the parameter is a float and multiplying a float by an int will always produce another float.
Similarly, you would not add an annotation to your my_volume variable. Since return_volume() has type hints, it's trivially easy to see what type it's returning and understand that my_volume is of type float. (And if you make a mistake and accidentally think it's something other then a float, then mypy will catch that for you.)
Mypy does some pretty advanced type inference. Usually, you do not need to annotate variables. The mypy documentation [1] says this about inference:
Mypy considers the initial assignment as the definition of a variable. If you do not explicitly specify the type of the variable, mypy infers the type based on the static type of the value expression
The general rule of thumb then is "annotate variables whose types are not inferrable at their initial assignment".
Here are some examples:
Empty containers. If I define
aasa = [], mypy will not know what types are valid in the lista.Optionaltypes. Oftentimes, if I define anOptionaltype, I will assign the variable toNone. For example, if I doa = None, mypy will infer thatahas typeNoneType, if you want to assignato5later on, you need to annotate it:a: Optional[int] = None.Complex nested containers. For example, if you have a dictionary with both list and string values, mypy might, for example, infer
Dict[str, Any]. You may need to annotate it to be more accurate.
Of course there are many more cases.
In your examples, mypy can infer the types of the expressions.
[1] https://mypy.readthedocs.io/en/latest/type_inference_and_annotations.html