Python 3.5 type hinting in PyCharm 5

Python 3.5 introduces type hinting to help code-writing during development. Let’s take a look at this feature and show it in action.

Motivation

Some folks, but not all, like help from their tools when writing code and documentation. For example, lint-like tools can spot errors before your code executes, and editors and IDEs can harness that to show you mistakes, as you type. Particularly for large code bases and large teams, such help is quite valuable.

These mistakes can include passing the wrong type of value to a function. Wouldn’t it be great if you could get that help once typed, or even as you are typing (autocompletion)? Other languages have some concept of type checking. We don’t want to turn Python into a statically-typed language. But can’t we get a little help?

Python’s PEP 484 brings optional type hinting to Python 3.5. Using the typing module, you can provide type hint information in your code. This can be used by yourself or others to flag certain problems, while you are developing.

Quick Example

Code speaks, so let’s start with an example. mypy was the originator of Python 3.5’s typing module. Let’s use one of its examples:

This example shows the most important part of type hinting: it is optional. Python hasn’t changed to suddenly require static typing. But if we add a little more:

…then we can let an IDE such as PyCharm help us:

02a

What Is (and Isn’t) Type Hinting?

There’s a lot of apprehension out there about this new language feature. Before showing more in-depth code, let’s try to clear this up. First, here’s what type hinting is for:

  • Develop better/faster. Think of it like linting, or as Guido said, “linting on steroids”. With the right tools, it can surface mistakes more quickly.
  • Help users of your code. If someone wants to use your function, type hinting helps explain (autocomplete) and flag when they mess up.
  • Documentation. Rather than pack argument and response type information into carefully-formatted docstrings, use something in the language.

So that’s the benefit, but what is type hinting not?

  • Not required. You can ignore it. You can use it a little. You can use it a lot. It’s all the same. Heck, Python itself (at runtime) ignores type hinting.
  • Not static typing. This is not, not, not changing Python into Java. The language is not enforcing anything.
  • Not Performance boosting. Providing type information makes people think of compilation speedups. But this has no impact on Python execution. Also, the Cython and PyPy teams have said these type hints are unlikely to make a difference to them for speedups.
  • Not incompatible. The extra characters added into your code are ignored by older versions of Python 3. This requirement – to be syntactically-valid pre-3.5 – was a constraint on the options for expressing the hints. If you’re one that thinks the syntax is ugly, this constraint is likely to blame.

Type hinting is most useful when you share your code with a consumer: when you publish a library, or code in a project with others, or helping out your future self who needs to live with the sins of your current self.

With all that preface out of the way, let’s take a look at some usages.

Generics

The teeny-tiny examples above just cover a single argument or return value. What if I want to say hello to a list of names?

Python 3.5’s type hinting provides an answer for this. Namely, you can express the hint as “a list of strings”:

What happens if we ignore the hint and supply a list of integers? PyCharm warns us:

04a

PEP 484 refers to hints such as List as “generics”. That is, a hint that contains other hints. An Iterator for example is a list-y kind of generic. As you would expect, dictionaries are supported:

In this case we see an unusual-looking structure: Dict[str, int]. Think of the values between the brackets as “arguments” to the generic Dict type. The first argument is the type of the dictionary keys. The second is the type of the dictionary values.

Optional Values

We are now passing in a string to use for formatting. But maybe we’d like our class to provide a default string. Let’s allow None as a value that can be passed in, in addition to a string:

Alas, type hinting now warns us that the first argument has to be a string:

07

Fortunately there are several ways to solve this, each useful in various circumstances. For example, we could use regular Python and make this a keyword option with a default:

With this, you can omit passing in a format string and instead use a value of None. (Of course, you could skip the following test and just make the format string the keyword default.) If you pass in an integer, though, you’ll get warned:

07a-broken

Here’s another variation: allow your users to pass in a formatting string or False:

The Union type from typing, as shown here, lets the type be from any of the provided type values. In this case, str or bool.

Here, though, is the approach that best conveys the meaning of optional:

That is, PEP 484 type hinting defines a generic type of Optional.

String-Based Hints

I follow a pattern where I use a static method on my class to query for instances. Hey, I saw someone smart post it once, don’t judge me. For example, in a SQLAlchemy model:

Unfortunately, that fails. As the mypy docs explain: “Python does not allow references to a class object before the class is defined.”:

09-broken

To fix this, type hinting has the concept of a forward reference. In the location where you would normally provide the hint, just provide that same hint, but in a string:

This lets the Python interpreter parse the module at import time, then deal with the type hinting later.

Stub Files

As mentioned in the introduction, some people might find all this type hinting to be noise that distracts from the readability of the code. Wouldn’t it be great if you could get the benefits of type hinting, but keep that information separate? Or in other cases, you can’t actually put in the type hints: it’s in old version of Python code (before function annotations), or in C code, or in code you don’t control.

For these cases, Python type hinting provides stub files. These are extra files, named the same as the module they are stubbing, which contain the type hints. For example, imagine we have a simple function in a module with a filename of greeting.py:

No type hinting. Then, imagine a second file, in the same directory, named greeting.pyi:

The addition of the .pyi suffix makes this shadow the original module. PyCharm helps by denoting the stubbing with an asterisk in the gutter, and a mouseover gives more detail:

greeting-broken1

If we now try to “break the rules” of type hinting, we’ll get the same effect we would if we put the type hints directly in our code:

greeting-broken2

Type Aliases

Sometimes the function arguments and return values can get quite unwieldy. Imagine a greeting function which could accept either a list of strings, or a dictionary where each value is a list of strings. Based on this, you’d also need return different types of values.

Here’s what the type hinting would look like:

Gorgeous, isn’t it? Well, not so much. For cases like this, Python 3.5 type hinting provides a type alias: a way to define the type hint information in one place, then use it in one or more other places. Let’s define a type alias for a GreetingType:

That’s a lot tidier, letting the function usage concentrate on the meaning and the declaration contain the noise….erm, provide the detail.

And yes, this section was added in response to a StackOverflow complaint about the unwieldy syntax.

Downsides

For the audience that wants tools to warn them earlier, Python 3.5 type hinting is very valuable. But not everybody falls into that category. They might use simpler editors. Or they might view those extra characters as seriously breaking Python’s readability.

In theory the answer is simple: “Then don’t use type hinting, it’s optional.” But the worry is that if type hinting becomes “best practice”, library authors will be pressured into supporting it. At that point, adoption will go up and you’ll constantly come across Python code that gouges your eyeballs. It won’t feel so optional then.

On the other side of the argument, some might wish it went further. Perhaps it should cover more cases. Perhaps it should bend the language syntax further to avoid the forward-reference comment-based type hinting, or provide a less line-noise-looking way to express complicated typings.

Conclusion

That’s a walkthrough Python 3.5 type hinting: what it is and isn’t, and ways that it can help you write quality code faster. This article isn’t an exhaustive treatment of all the options. For more information, take a look at the mypy examples and PEP 484.

This entry was posted in Cool Feature, Tutorial. Bookmark the permalink.

46 Responses to Python 3.5 type hinting in PyCharm 5

  1. Damon Lynch says:

    For those of use developing using Python 3.4, but still wanting to use the typing module, will PyCharm perform all of the assistance outlined above if we install a backported version of typing module ourselves?

    Assuming PyCharm will work as outlined above with Python 3.4 & the typing module , there (of course) remains the problem that Python 3.4 does not have the typing module in the standard library. When distributing code designed to function in both Python 3.4 & 3.5 that imports the typing module, my initial thought to work around this problem is to install a copy of the typing module alongside my own code’s modules (giving it a different name, e.g. localtyping), and then importing that only on an ImportError exception when trying to “import typing”. Does that strike you as a wise strategy?

    • Andrey Vlasovskikh says:

      PyCharm supports the typing module from PyPI for Python 2.7, Python 3.2-3.4. For 2.7 you have to put type hints in *.pyi stub files since function annotations were added in Python 3.0.

      I believe it’s OK to specify typing as a conditional dependency (sys.version_info < = (3, 5)) in your setup.py or in a special version of requirements.txt for Python < 3.5.

  2. Kiran says:

    One big part of PyCharm that would help with creating proper type hinting and proper documentation is _still_ broken: https://youtrack.jetbrains.com/issue/PY-17722! Has been since the release of 5.0, please fix it.

  3. Chris Arena says:

    I’m curious if your team had considered implementing a way to rip all of the annotations from a file and put them into a .pyi stub in one easy c0mmand. It doesn’t seem like it would be too difficult to implement, and would be valuable when inevitable complaints about verb0sity come. If the response to that can be “sure, one sec … okay done, rebase with master” I imagine it could quell a lot of the opposition to usage of this feature.

  4. NikosC says:

    Assuming we have type hinted some (or all) of our code, can pycharm provide a “report” for type errors project-wise? How different is from mypy’s type checker?

    • Andrey Vlasovskikh says:

      You can run any subset of the code inspections offered by PyCharm via “Code | Inspect Code” menu.

      Type checker is only one of many inspections that are based on the type information found in the source code. You can browse the list of the inspections in “File | Settings | Editor | Inspections”. Mypy is focused on just checking the types and a few extra things.

  5. Fellipe says:

    Any chance to build a more fast IDE? Since version 4.0 get more and more slow in my Mac!! Even change the options on vmoptions file and with I5 with 8G RAM. I tried these version 5 and same “problem”… :(

  6. Shish says:

    What are the differences between using this syntax and using eg “@param” in a docstring?

    (AFAIK this syntax is easy to interact with at runtime, though I guess you *can* parse a docstring at runtime; and this syntax is standard so it should have wider support among other third-party tools – anything else?)

    • Andrey Vlasovskikh says:

      The syntax you can use in @param tags inside docstrings isn’t specified by any standard so the tools are not compatible with respect to type hints. PEP 484 defines the standard type hinting syntax for Python that you can rely upon.

      We do our code analysis and type checking statically, i. e. based almost only on the source code. A few runtime things we do is evaluate sys.path in order to find the installed libraries and record the types of expressions while debugging if the “Collect run-time type information for code insight” option is set.

  7. Dmytro says:

    > But the worry is that if type hinting becomes “best practice”, library authors will be pressured into supporting it.

    Meh. PEP8 has became a “best practice” years ago. Nevertheless, there is a plenty of projects out there which ignore PEP8 completely. Now they’ll just have one more thing to ignore.

  8. Alexander Malahov says:

    Nitpick:
    We don’t want to turn Python into a strongly-typed language.
    But Python is strongly-typed (As opposed to e.g. C, which has static weak typing)

    :)

  9. sergey says:

    not sure its bug or feature but pycharm 5.0.2 don’t complain about wrong return type i.g. “def greeting() -> str: return 1” produce no type hint warning.

  10. I.e., Python3 code with perfectly implemented type hints would only make it easier on the developer, and not the interpreter compiler, yeah?

  11. Gabor Bernat says:

    The inspections in 5.0.2 complain about valid code:

    it wants typing to be added to requirements.txt
    and says that Lists getitem is not implemented, like what _

    http://i.imgur.com/bxWngl4.png

  12. Evgeny says:

    Any chance you can implement support of MyPy style type hinting for Python 2.7?
    Basically it’s based on PEP-484, so the inline type-comments are the same as PEP-484 (and PyCharm already understands them), and methods annotations are also defined in comments:

    def get_users(user_ids):
    # type: (Iterable[int]) -> List[User]
    ....

    Thanks!

  13. Evgeny says:

    Oh, by the way, after reading PEP-484, I’ve realized that the syntax I mentioned above is actually part of PEP-484: https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code
    However, it doesn’t look like PyCharms supports this. Could you please add this syntax as well?

    Thanks!

  14. Dustin Wyatt says:

    What if we want to create stub files for libraries? I can’t imagine create .pyi files and putting them in your site_packages directory is very maintainable.

  15. Charles Merriam says:

    How do I turn this off. We have our own annotation system, and pyCharm now trips all over the place. PEP484 is provisional and optional; but our type checker causes all sorts of “shadowing” errors when types are mentioned in strings. Can we turn of trying to parse those strings? Simply turning off the TypeChecker does not stop it.

  16. HariKrishna Mukkanti says:

    In my machine Annotate is not working so can you guys please suggest me on this??

    • Paul Everitt says:

      Sorry to hear that. Can you describe the steps you took when trying to use it?

      • Jared Miller says:

        I am curious if I’m overlooking something as well:

        https://puu.sh/qI9hX/44d05db8e8.png

        So, it is working in that I can see the properties/methods of the type hinted class.

        However if I set self.client to a string, I’d expect it to yell at or warn me.

        https://puu.sh/qI9kU/b79f23bf87.png

        • Mikhail Golubev says:

          As for the first case, I don’t think that type checker error here would make much sense, since there is no rule that requires that class attributes initialized in “__init__()” and respective parameters should have identical names. I mean, the annotation “TelnetClient” for the parameter “client” doesn’t imply that “self.client” should have the same type.

          As for the second question, you’re right, it’s a known problem with augmented assignments for strings — PY-6426. It’s pretty old, but, no doubt, we’re going to address it in upcoming releases. Thanks for reminding us of it!

      • Jared Miller says:

        Well, it appears type checking works in every other scenario: calling the method and passing in the wrong type (as demonstrated in this blog post) and instantiating a class.

        So:

        def say(msg: str):
        return(msg)

        say(1) yells at you, which is good

        def say(msg: str):
        msg += 1
        return msg

        The above does not alert you of anything, however.

        Maybe there’s a good reason for this, though.

  17. Florian B. says:

    Hi !

    Is there any plan to support annotation like


    def main(my_param: 'A beautiful param'):
    pass

    as suggested in https://www.python.org/dev/peps/pep-3107/#fundamentals-of-function-annotations

    The goal is not to use its, just do not display an error when doing that in PyCharm, because somes libraries use its (like begins for example)

  18. Mikhaill Golubev says:

    In PEP 484 they describe a way to suppress type checking in such cases using “@no_type_check” decorator or “# type: ignore” comments. Also there is special “@no_type_check_decorator” decorator that the author of “begin” library could use for “begins.start”. Unfortunately, none of these methods are supported in PyCharm at the moment. Please feel free to leave a vote for PY-17044, in particular.

  19. Mike Cline says:

    Type checking on class member variables doesn’t seem to work.

    PyCharm (Community Edition, 2016.2.3) does not point out any type mismatch in the following code:

    class TestClass:

    x = None #type:int

    t = TestClass()
    t.x = “foo”

    I read the chain above started by Evgeny regarding this style of type hinting (introduced in PEP-484), to which Paul Everitt responds with the link

    https://youtrack.jetbrains.com/issue/PY-17623

    Is it true that PyCharm hasn’t made any progress on this since Nov 2015? If so that strikes me as rather sad.

    MyPy is very active at the moment, with Guido seeming to be leading the way on that project, perhaps you should consider integrating MyPy with PyCharm instead of re-implementing type-checking in your product and being behind the curve.

    Also, PEP 526 seems to be a superior spec for annotating class members, and MyPy already seems to be heading in that direction. It would be good to see PyCharm keeping up with this: https://github.com/python/mypy/pull/2131

    • Mikhail Golubev says:

      Actually comments with type hints were supported long ago. You can see that PyCharm is able to infer the type of “TestClass.x” because there are code completion on this attribute and warnings if you try passing it to some function that expects argument of type other than “int” or accessing any method that doesn’t belong to “int”. The actual problem here is that “Type checker” inspection is a bit more lenient than mypy, In particular, it’s allowed to redefine variable by assigning value of another type to it (even when type annotation is present). We’re planning to implement stricter mode closer to mypy in future, though.

      We indeed could create a dedicated inspection to run mypy in batch mode when you run “Inspect Code…”, but it’s going to work well only there. For instance, mypy is not able to handle incomplete, syntactically invalid code as our parser does. Neither can it perform type checks incrementally, when type information for unmodified files is persisted and it’s not necessary to parse and infer types for these files all anew every time. The later limitation is critical from performance point of view in the IDE, since most refactorings, inspections, intentions and “find usages” rely heavily on type inference engine to operate properly. Unfortunately, it also makes it difficult to quickly integrate recent updates in “typing” specification in PyCharm.

      As for PEP 526, it is going to be available in PyCharm 2016.3 with initial support of variable annotations included in the first EAP.

  20. Pingback: I like Python! Concessions of a Java Developer

Leave a Reply

Your email address will not be published. Required fields are marked *