Faster debugger in PyCharm 5.1

The first EAP for PyCharm 5.1 was released last week, with lots of enticing features. Here’s one that jumps out for long-time users: “Debugger performance improvements”. Let’s take a look at this in a little more depth, then provide some of the back story.

Imagine we’re debugging a semi-large code base with PyCharm 5.0.4. If we set a breakpoint and time how long it takes to get there, it measures out as 12 seconds:

debugger_speedup 5.1 pre

In PyCharm 5.1, the debugger performance has improved, especially for large code bases. Let’s give the same scenario a try:

debugger_speedup

Almost 3.5 seconds of improvement on OS X.

Wouldn’t it be great to get a couple more seconds? And now comes the real story: the new debugger has some performance improvements implemented in Cython. On OS X and Linux we need to install them manually, as the debugger console message helpfully tells us:

debugger_speedup 5.1 screenshot

Windows users get these Python speedups pre-bundled. Let’s install them on OS X by copying the text above after “Debugger speedups using Cython not found. Run…”:

debugger_speedup 5.1 install

That is, we’re running:

…using the Python in this project’s virtual environment, just to make sure we have the correct Python version. Once more, let’s measure the time to get to the breakpoint:

debugger_speedup 5.1 post

5.7 seconds! That’s a meaningful difference. As stated by the developers, the debugger is 40% faster in the general case and over 130% faster when you have the Cython modules compiled and installed. Again, if you are on Windows, you don’t need to do anything — you will get these improvements automatically for Python 2.7, Python 3.4, and Python 3.5 interpreters.

On OS X and Linux, you need to do a one-time compilation using any Python on your system matching the Python version you need, as shown in the screenshot above. For example, if you use Python 2.7 and Python 3.4, you need to run the Cython speedups with an interpreter matching those two versions. Each time you run it, a compiled speedup will be saved in your PyCharm application, for that Python version. For example, on my system, this was created:

Note the “-2.7” in the directory name. Also, note that you don’t have to do this with your virtual environment’s Python, but it certainly makes sense to do so, as you’ll be sure to match the version. You don’t have to do this once for every Python 2.7 virtual environment, as these speedups aren’t stored in the virtual environment. They are stored inside PyCharm’s pydevd helper.

Now, on to the backstory. As explained in the blog post announcing the 5.1 EAP, JetBrains joined efforts with PyDev, helping sponsor the work on pydevd which is shared by the two projects. Both projects require a sophisticated debugger backend and previously merged their work. This performance improvement is another step forward in the collaboration.
If you debug a large code base, give this EAP a try with the Cython speedups and post a comment letting us know your results. We expect performance improvements to be higher for larger code bases.

PyCharm Team
The Drive to Develop

This entry was posted in Uncategorized. Bookmark the permalink.

7 Responses to Faster debugger in PyCharm 5.1

  1. Matthew Rubenstein says:

    I tried running (/usr/bin/python2.7 /usr/lib/pycharm-community/helpers/pydev/setup_cython.py build_ext –inplace) as directed by the warning (and by this article), but it failed (as my normal user, not sudo or as root, on Ubuntu 15.10).

    First it failed with:
    ———————————————————
    _pydevd_bundle/pydevd_cython.c:10:20: fatal error: Python.h: No such file or directory
    ———————————————————

    So I ran (apt-get install python-dev) to get Python.h , but it still failed on rerunning setup_cython.py:
    ———————————————————
    copying build/lib.linux-x86_64-2.7/_pydevd_bundle/pydevd_cython.so -> _pydevd_bundle
    error: [Errno 13] Permission denied: ‘_pydevd_bundle/pydevd_cython.so’
    ———————————————————

    So I ran it as sudo for root’s permissions, but it still failed:
    ———————————————————
    copying build/lib.linux-x86_64-2.7/_pydevd_bundle/pydevd_cython.so -> _pydevd_bundle
    error: could not delete ‘_pydevd_bundle/pydevd_cython.so’: Permission denied
    ———————————————————

    So I ran su and ran setup_cython.py again, as root, and seemed to complete OK (no errors, but no explicit indication of success):
    ———————————————————
    running build_ext
    copying build/lib.linux-x86_64-2.7/_pydevd_bundle/pydevd_cython.so -> _pydevd_bundle
    ———————————————————

    When I ran PyCharm again and started the debugger against a script it didn’t report the warning to run setup_cython.py . So it seems the installation was finally complete.

    This procedure shouldn’t require running as root, or even as sudo. In fact the Ubuntu .deb that my normal update process offered me to install from the normal Ubuntu repos should have the dependency on setup_cython.py included, not as a separate procedure we have to find in this blog entry. And if installation does depend on reading this blog entry, the blog entry should specify that running as root will be necessary. And the setup_cython.py script should indicate success when it succeeds.

  2. Joaquin Menchaca says:

    Running into similar issues, cannot get this to work, and documentation is sometimes ambiguous. IDE on Mac, Code on Ubuntu, but cannot find find setup_cython.py on Ubuntu 14. I installed pydevd through pip, and with apt-get python-pip, python-dev, cython, but alas no setup_cython.py.

    Besides the warning, cannot get it to work. Not sure where I am suppose to place the line:
    pydevd.settrace(“192.168.54.4”, port=5678)

    I get the following:

    Could not connect to localhost: 5678
    Traceback (most recent call last):
    File “/usr/local/lib/python2.7/dist-packages/_pydevd_bundle/pydevd_comm.py”, line 488, in start_client
    s.connect((host, port))
    File “/usr/lib/python2.7/socket.py”, line 224, in meth
    return getattr(self._sock,name)(*args)
    error: [Errno 111] Connection refused

  3. Georg Heiler says:

    I get an error trying to install the cython speed upgrade:

    limits.h seems to be missing

    “/usr/local/opt/python3/bin/python3.5” “/Applications/PyCharm.app/Contents/helpers/pydev/setup_cython.py” build_ext –inplace
    running build_ext
    building ‘_pydevd_bundle.pydevd_cython’ extension
    gcc-6 -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m -c _pydevd_bundle/pydevd_cython.c -o build/temp.macosx-10.11-x86_64-3.5/_pydevd_bundle/pydevd_cython.o
    In file included from /usr/local/Cellar/gcc/6.2.0/lib/gcc/6/gcc/x86_64-apple-darwin15.6.0/6.2.0/include-fixed/syslimits.h:7:0,
    from /usr/local/Cellar/gcc/6.2.0/lib/gcc/6/gcc/x86_64-apple-darwin15.6.0/6.2.0/include-fixed/limits.h:34,
    from /usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m/Python.h:11,
    from _pydevd_bundle/pydevd_cython.c:10:
    /usr/local/Cellar/gcc/6.2.0/lib/gcc/6/gcc/x86_64-apple-darwin15.6.0/6.2.0/include-fixed/limits.h:168:61: fatal error: limits.h: No such file or directory
    #include_next /* recurse down to the real one */
    ^
    compilation terminated.
    error: command ‘gcc-6’ failed with exit status 1

Leave a Reply

Your email address will not be published. Required fields are marked *