Debugger Interview with PyDev and PyCharm

PyCharm’s visual debugger is one its most powerful and useful features. The debugger got a big speedup in the recent PyCharm, and has an interesting backstory: JetBrains collaborated with PyDev, the popular Python plugin for Eclipse, and funded the work around performance improvements for the common debugger’s backend.

To tell us more about the improvements as well as cross-project cooperation, we interviewed the principles: Fabio Zadrozny, creator of PyDev, and Dmitry Trofimov, Team Lead for PyCharm.

Let’s jump right into it. Tell us about the speedups in the latest pydevd release.

FZ: The performance has always been a major focus of the debugger. I think that’s actually a requisite for a pure-python debugger.

To give an example here: Python debuggers work through the Python tracing facility (i.e.: sys.settrace), by handling tracing calls and deciding what to do at each call.

Usually a debugger would be called at each step to decide what to do, but pydevd is actually able to completely disable the tracing for most contexts (any context that doesn’t have a breakpoint inside it should run untraced) and re-evaluate its assumptions if a breakpoint is added.

Now, even having the performance as a major focus, the latest release was still able to give really nice speedups (the plain Python version had a 40% speed improvement overall while the Cython version had a 140% increase).

I must say that at that point, there weren’t any low-hanging fruits for speeding up the debugger, so, the improvement actually came from many small improvements and Cython has shown that it can give a pretty nice improvement given just a few hints to it.

DT: The performance of the debugger was one of the top voted requests in PyCharm tracker. The latest release addresses this by implementing some parts of the debugger in Cython, which leads to huge performance improvements on all type of projects.

Was the Cython decision an easy one?

FZ: Actually, yes, it was a pretty straightforward decision…

The main selling point is that the Cython version is very similar to the Python version, so, the same codebase is used for Cython and plain Python code — the Cython version is generated from the plain Python version by preprocessing it with a mechanism analogous to #IFDEF statements in C/C++.

Also, this means that with the same codebase it’s possible to support CPython (which can have the Cython speedups) while also supporting Jython, PyPy, IronPython, etc. I even saw someone post about the debugger being used in a javascript implementation of Python.

DT: The idea was to make the debugger faster by rewriting the bottlenecks in C, but at the same time optional to have any compiled binaries, so that pure Python version would still work. Also, it was desirable to have as little code duplication as possible. Cython let us do all that perfectly, so it was a natural decision.

Let’s take a step back and discuss the 2014 decision to merge efforts. How did this conversation get started?

FZ: I was doing a crowdfunding for PyDev which had a profiler as one of its main points, which was something that PyCharm wanted to add too. Although the initial proposal didn’t come through, we started talking about what we already had in common, which was the debugger backend and how each version had different features at that point. I think PyCharm had just backported some of the changes I had done in the latest PyDev version at that time to its fork, and we agreed it would be really nice if we could actually work in the same codebase.

DT: We have used the fork of Pydev debugger since the beginning of the PyCharm and occasionally I would check what was going in Pydev branch to backport features and fixes from there to PyCharm. Meanwhile, Fabio does the same, taking the latest fixes from PyCharm branch. As time passed and branches diverged, it was getting more and more difficult to compare the branches and backport fixes from one another.

After one of the tough merges, I thought, maybe we’d better create a common project that would be used in both IDEs. So I decided to contact Fabio and was very happy when he supported the idea.

Did the merging/un-forking go as you planned, or were there technical or project challenges?

FZ: The merging did go as planned…

The main challenge was the different feature set each version had back then. For instance, PyDev had some improvements on dealing with exceptions, finding referrers, stackless and debugger reload, whereas PyCharm had things such as the multiprocessing, gevent and Django templates (and the final version had to support everything from both sides).

The major pain point on the whole merging was actually on the gevent support, because the debugger really needs threads to work and gevent has an option for monkey-patching the threading library, which made the debugger go haywire.

DT: The main challenge was to test all the fixes done for the PyCharm fork of the debugger for the possible regressions in the merged version. We had a set of tests for debugger, but the coverage, of course, wasn’t 100%. So we made the list of all debugger issues fixed for the last 3 years (around 150 issues,) and just tested them. That helped us to ensure that we won’t have regressions in a release.

Fabio, how did it go on your end, having JetBrains sponsor some of your work? Any pushback in your community?

FZ: I must say I didn’t really have any pushback from the community. I’ve always been pretty open-minded about the code on PyDev (which was being used early on in PyCharm for the debugger) and I believe IDEs are a really personal choice. So I’m happy that the code I did can reach more people, even if not directly inside PyDev. Also, I think the community saw it as a nice thing as the improvements in the debugger made both, PyDev and PyCharm, better IDEs.

The Python-oriented IDEs likely have some other areas where they face common needs. What do you think are some top issues for Python IDEs in 2016 and beyond?

FZ: I agree that there are many common needs on IDEs — they do have the same target after all, although with wildly different implementations 😉

Python code in particular is pretty hard to analyze in real-time — which contrasts with being simple and straightforward to read — and that’s something all “smart” Python IDEs have to deal with, so, there’s a fine balance on performance vs. features there, and that’s probably always going to be a top issue in any Python IDE.

Unfortunately, this is probably also a place where it’s pretty difficult to collaborate as the type inference engine is the heart of a Python IDE (and it’s also what makes it unique in a sense as each implementation ends up favoring one side or the other).

DT: The dynamic nature of Python was always the main challenge for IDEs to provide an assistance to developers. A huge step forward was done with Python 3.5, by adding a type hinting notation and typeshed repository from which we will all benefit a lot. But still this thing is in its early stage and we need to define and learn effective ways to adopt type hinting.

Python performance is also a challenge. In the Python world, when you care about performance, you switch from using pure Python to libraries written in C, like numpy. Or you try pypy. But in both cases performance and memory profiling becomes hard or even impossible with current standard tools and libraries. I think that tool developers can collaborate on that to provide better instruments for measuring and improving the performance of Python apps.

What’s in the future for pydevd, performance or otherwise?

FZ: I must say that performance wise, I think it has reached a nice balance on ease of development and speed, so, right now, the plan is not having any regression 😉

Regarding new development, I don’t personally have any new features planned — the focus right now is on making it rock-solid!

DT: One of the additions to pydevd from the PyCharm side is the ability to capture the types of the function arguments in the running program. PyCharm tries to use this information for code completion, but this feature now is optional and off by default. With the new type hinting in Python 3.5 this idea gets a new spin and the types collected in run-time could be used to annotate functions with types or verify the existing annotations. We are currently experimenting only with types, but it could be taken further to analyse call hierarchy etc.

Posted in Uncategorized | 2 Comments

PyCharm 2016.1.3 RC is available

It’s been a while since we released PyCharm 2016.1.2 and today we announce the Release Candidate build of PyCharm 2016.1.3.

Some highlights of the PyCharm 2016.1.3 RC build 145.969 are:

  • A number of fixes for Docker-based interpreters support
  • Fixes for Docker Compose support
  • Fixes for false positive warnings in strings inspections
  • Various performance improvements for the code insight engine
  • And much more

The list of numerous bug fixes and improvements for this build can be found in the Release Notes.

We encourage you to give PyCharm 2016.1.3 RC a try before its official release. Please change the updates channel to “Early Access Program” in the updates settings (Help|Check for updates…) to get the patch update to the EAP build from the recently released PyCharm 2016.1.2 or download the full installer for your platform. Please report any bugs and feature request to our issue tracker.

-PyCharm team

Posted in Early Access Preview | 1 Comment

PyCharm Migration Tutorial for Text Editors

If you’re a Python developer who uses a text editor such as Vim, Emacs, or Sublime Text, you might wonder what it takes to switch to PyCharm as an IDE for your development. We’ve written a helpful Migrating from Text Editors tutorial for just this topic.

The tutorial starts with the basic question of “What is an IDE?” The line between text editor and IDE can be blurry. PyCharm views the distinction as: a project-level view of your code and coding activities, with project-wide features such as coding assistance and refactoring.

This document then goes over some of the important points when migrating: the project-oriented UI, working with projects instead of files, Vim/Emacs specifics, keyboard shortcuts, customizing, and a discussion of facilities important to text editor users (multiple cursors, split windows, etc.) It then closes by discussing areas the IDE can really help, for example, the managed running and debugging of your code.

Of course, this document is just an overview. Vim and Emacs in particularly have decades of development and features, and PyCharm itself is now very mature with many features itself, so a complete comparison would break the Internet. If you have a specific question, feel free to comment, and we hope you find the tutorial helpful.

Posted in Tutorial | Leave a comment

In-Depth Screencast on Testing

Earlier this year we rolled out a Getting Started Series of screencast videos on the basics of using PyCharm: setup, the UI, running Python code, debugging, etc. We knew at the time that some topics would need more treatment than a quick screencast, so we planned a follow-on series of “in-depth” screencasts, each on a single topic.

Here’s the first: In-Depth: Testing covers more topics than we went over in the Getting Started: Testing screencast. Which makes sense, as PyCharm itself has such a wealth of support in each of its features:

Here are the topics in this just-over-four-minute video:

We hope you enjoy this first in the series of In-Depth screencasts. We have more planned, such as version control. And please, if you have any topics that you’d like to see get expanded screencast attention, let us know.

Posted in Screencast, Tutorial, Video | 4 Comments

Announcing PyCharm 2016.1.2

PyCharm 2016.1.2 is now available from the download page.

Soon it will also be available as a patch update from within the IDE. A patch update from v2016.1.1, which exposed a severe performance regression, will be available shortly today. The patch update from v2016.1 is still under way and will be available tomorrow.

The Release Notes lists all fixes for this update.

We bring you this build shortly after releasing 2016.1.1 because of the important bug-fix, PyCharm freezes on large string literals [PY-19075].

Other fixes for various Docker Compose support problems and a fix for extra libraries indexing are currently in progress and planned for the next bug-fix update.

Download PyСharm 2016.1.2 for your platform from our website and please report any problem you found in the Issue Tracker.

If you would like to discuss your experiences with PyCharm, we look forward to your feedback in comments to this post and on twitter.

Your PyCharm Team
The Drive to Develop

Posted in Release Announcements | 12 Comments

Announcing PyCharm 2016.1.1

Today we are pleased to announce the PyCharm 2016.1.1 bug-fix update. It has been uploaded and is now available from the download page. It also will be available shortly as a patch update from within the IDE (from PyCharm 2016.1 only).

Some notable highlights of this update include:

  • Fixed wrong indentation after copy-paste
  • Fixed generation of reStructuredText doc strings for methods with parameters
  • Fixed folding of custom regions
  • Fixed numerous false positives in formatted strings
  • Fixed wrong highlighting of parameters in formatted strings
  • A fix for a problem with “Gevent compatible” debugger’s mode

Fixes for various Docker Compose support problems and a fix for extra libraries indexing are currently under way and will get to the next bug update.

For further details on the bug fixes and changes, please consult the Release Notes.
As usual, please report any problem you found in the issue tracker.

If you would like to discuss your experiences with PyCharm, we look forward to your feedback in the comments to this blog post and on twitter.

The Drive to Develop
PyCharm Team

Posted in Release Announcements | 16 Comments

PyCharm 2016.1.1 EAP is available

It’s been a week since we released PyCharm 2016.1 and today we announce the early preview build of PyCharm 2016.1.1.

Some highlights of the PyCharm 2016.1.1 EAP build 145.597 are:

  • Supported Python versions changes
    • Dropped support for older Python versions before Python 3.5
    • Added preliminary support for Python 8
  • Various bug fixes
    • Fixed wrong indentation after copy-paste
    • Fixed generation of reStructuredText doc strings for methods with parameters
    • Fixed folding of custom regions

For the complete list of the changes see the release notes.

There’re other fixes that are currently under way and will hopefully get into the final PyCharm 2016.1.1 release. They’re: fixes for various Docker Compose support problems, a fix for extra libraries indexing. Stay tuned for further announcements.

We encourage you to give PyCharm 2016.1.1 EAP a try before its official release. Please change the updates channel to “Early Access Program” in the updates settings (Help|Check for updates…) to get the patch update to the EAP build from the recently released PyCharm 2016.1 or download the installer for your platform. Please report any bugs and feature request to our issue tracker.

-PyCharm team

Posted in Early Access Preview | 4 Comments

Announcing General Availability of PyCharm 2016.1

Today we bring you PyCharm 2016.1, now available for download. This is the first update in the series of releases planned for 2016. Its outstanding new features for professional Python, Web and scientific development work together smoothly to offer you a unique coding experience.


As usual PyCharm 2016.1 is available as a full-featured Professional Edition for Python and Web development, or as a free and open-source Community Edition for pure Python and scientific development.

Here are some notable highlights of this release.

Python-related improvements:

  • Python 2 and Python 3 type hinting and compatibility inspection
  • Docker Compose and Docker container settings
  • Tox Support
  • Improved Support for Django Forms
  • Significant debugger improvements
  • Enhanced support for IPython Notebook

Platform enhancements:

  • Git Rebase and Rename
  • Git Worktree Support
  • By-word changes highlighting in Diff Viewer
  • Enhanced Database Tools
  • Local Terminal Improvements

Web development improvements:

  • Improvements in ECMAScript 6 and TypeScript support
  • Major enhancements in Angular 2 support
  • Support for debugging asynchronous client-side code
  • Running and debugging Node.js apps on remote hosts
  • And even more

For more details please watch this short What’s New in PyCharm 2016.1 video:

Read more about what’s new in PyCharm 2016.1 on the product website and download the IDE for your platform.

New versioning
As we announced earlier, with this release we’re changing the versioning scheme and moving away from one “major” release per year to several “equal” releases per year. Also, we’re aligning releases and version numbers across all products under JetBrains Toolbox. The new versioning will follow the format “YYYY.R” where yyyy represents the year, and “r” the release within that year.

The JetBrains Team
The Drive to Develop

Posted in Release Announcements | Tagged , , , , , | 42 Comments

PyCharm 2016.1 RC2 is Available

With just a day before the official release of PyCharm 2016.1, today we’ve published the PyCharm 2016.1 RC2 build, which is already available for download and evaluation from the EAP page.

The full list of fixes and improvements for this build can be found in the release notes. The most notable fixes are:

  • a fix for an error on creating Docker Compose interpreter (PY-18854)
  • a fix for docker-compose.yml v2 support (PY-18862)
  • a fix for an exception in handling references inside formatted strings (PY-18836)

Download PyСharm 2016.1 RC2 build for your platform and please report any bugs and feature request to our Issue Tracker. It also will be available shortly as a patch update from within the IDE (from previous PyCharm 2016.1 RC only), make sure you’ve selected the Beta Releases or Public Previews channel in the update settings.

Stay tuned for a PyCharm 2016.1 release announcement and follow us on twitter.

-PyCharm Team

Posted in Early Access Preview | 1 Comment

PyCharm 2016.1 RC is Available along with New Versioning

We are now approaching the final steps towards the PyCharm 2016.1 release. So today we’ve published the PyCharm 2016.1 Release Candidate.

The PyCharm 2016.1 RC build 145.258 is available for download from the Early Access Preview page. Please take it for a spin and give us your feedback.

A week ago we announced the new versioning model for all JetBrains Toolbox products. With the upcoming release, former PyCharm 5.1 will become PyCharm 2016.1. The new versioning is aligned with releases of other JetBrains tools, following the new YYYY.R format. Please read more about new versioning in the JetBrains Toolbox—Release and Versioning Changes blog post.

PyCharm 2016.1 Release Candidate includes mostly a consolidation of bug fixes as compared to the previous PyCharm 5.1(2016.1) Beta build. For the detailed list of notable changes and improvements in this build, please check the Release Notes.

In case you missed what’s new in upcoming PyCharm 2016.1 – please see the announcement of PyCharm 5.1 Beta or for detailed information, check the announcements of previous EAP builds.

You can download the build or upgrade from within the IDE (from previous PyCharm 2016.1 Beta only) without a full re-installation using the patch-based upgrade.

We hope that there will be no major bugs, however, should you encounter any problems, please report them to YouTrack – we’ll still have a bit of time to fix stuff before the final release.

Stay tuned for a PyCharm 2016.1 release announcement and follow us on twitter.

-PyCharm team

Posted in Early Access Preview | 14 Comments