PyCharm puts a visual face on debugging and we’d like to help more developers take the plunge and debug. In this webinar, Paul Everitt, PyCharm Developer Advocate, introduces the debugger and goes through many of the essentials in the context of writing a 2d game:
– Debugging without PyCharm’s debugger: print and pdb
– First use of the debugger (and the Cython speedups)
– Using the console/IPython at a breakpoint
– Exception and Conditional breakpoints
– All flavors of stepping through code, including filters
– Moving through stack frames
– Setting watches
– Django/Jinja2 template debugging
– Attaching to processes
– Debugging during testing
– Viewing numpy/pandas data frames
– Extracting type information
This webinar is aimed at developers who have never used the debugger, but even those familiar with PyCharm’s visual debugging will learn some new tricks and see some deeper usages.
Paul Everitt is the PyCharm Developer Advocate at JetBrains. Before that, Paul was a co-founder of Agendaless Consulting and a co-founder of Zope Corporation, taking the first open source application server through $14M of funding. Paul has bootstrapped both the Python Software Foundation and the Plone Foundation. Prior to that, Paul was an officer in the US Navy, starting in Python and launching www.navy.mil in 1993.
Back in 2016, we partnered with Stepik to announce a brand-new Python course. It adapted to each student’s level of knowledge and interest and helped them stay more motivated and productive while learning Python. This course was the first step towards adopting Adaptive Learning, which we believe is one of the key trends in the future online education.
After an adoption period, we’ve fixed all the issues in the latest PyCharm Edu 3.5.1. The Adaptive Python course is now polished and ready to use! Let’s explore it in more detail.
The Adaptive Python Course
To start the course, go to File -> New Project -> Educational and choose Adaptive Python from the list:
A couple of issues were resolved in PyCharm 2017.1, and Docker for Mac should now work out of the box. In this blog post we’ll show you how to set up a project with Docker Compose on a Mac. If you’re on Linux, the process is very similar. Docker Compose is supported on Windows from PyCharm 2017.2.
In this tutorial we’ll show how to create a very simple ‘Hello World’ Flask application, and then how to run it within a Docker container. For the full code, and to follow along, see GitHub: https://github.com/ErnstHaagsman/flask-compose.
First things first
Before we get started, let’s check a couple of things: make sure you are using PyCharm 2017.1 Professional Edition or later. Docker support is not available in PyCharm Community Edition. Then, please ensure your Docker and Docker Compose are up to date. To check, open a terminal, and run docker -v, and docker-compose -v:
Then, although it is the default setting in PyCharm 2017.1, it never hurts to check that your Docker API URL (Preferences | Build, Execution, Deployment | Docker) is set to unix:///var/run/docker.sock.
Now we can create a new Flask project. Let’s create a virtualenv so that PyCharm can stub out our Flask project before we have Docker configured. You can create a Virtualenv by using the ‘…’ button next to the interpreter dropdown.
After you click create, you should see the standard “Hello World” Flask template. So let’s see if we can get Flask to show us “Hello World” from a Docker container. To do this, we’ll add four files:
requirements.txt, just put “Flask==0.12” here to install Flask
Dockerfile, where we will set up the Python environment for the Flask app
docker-compose.yml to set up how to run the Dockerfile, and add a database
docker-compose.dev.yml to make some changes for local development
In the Dockerfile, we’ll use the ‘python:3’ image, expose port 5000, install packages from requirements.txt, and afterwards copy the rest of our project into /app. See the full file on GitHub.
Then, we’ll create a Compose file where we define the Flask app as our only service (for now). Just use build: . to make Docker Compose build the container from the Dockerfile.
Docker will bake our code into the image, and that way the image is self-contained. If we wanted to, we could stop here and rebuild our image whenever we change the code. It makes development quite a bit faster to mount our code with a volume. So we’ll create an additional Compose file to do that while we’re developing.
In docker-compose.dev.yml, all we’re doing is adding a volume mapping for .:/app for the web service. This overlays the code in the container with a volume, making sure that code changes are applied immediately. Keep in mind that you will need to rebuild the image before pushing it anywhere.
Now that everything is configured, let’s quickly run docker-compose up in the Terminal to make sure that works. Open the terminal with Alt+F12 (in PyCharm) and run docker-compose up.
As we can see, Docker built our container, and Flask is running! Let’s go and check out our ‘Hello World’ message!
A trap for young players!
What happened here? Didn’t docker say that Flask was running?
For security reasons, many modern web frameworks actually limit incoming connections to only come from localhost. This means that our Flask app is running, but only accessible from within the Docker container, not from our Mac host. The easiest way to fix this is to add host='0.0.0.0' as a keyword argument to app.run() in flask-compose.py:
This change tells Flask to listen not only to requests from localhost (in this case the docker container), but on all network connections. In the case of our container this means we can access it from our host, and depending on configuration from other computers in our network.
If we now stop our container (Ctrl+C in the PyCharm Terminal), and rebuild it by running docker-compose up --build. We can see in the Docker Compose output that Flask is now listening on 0.0.0.0 instead of 127.0.0.1:
And reloading the page in Safari actually shows ‘Hello World’ now. So let’s stop the container (Ctrl+C in the Terminal), and then configure PyCharm.
To make our project run using a PyCharm run configuration, we need to set the project interpreter to the Docker Compose service. We can do this from the Project interpreter page in preferences: Preferences | Project: <Project Name> | Project Interpreter.
Click the little button next to the interpreter dropdown (the white one, not the blue one), and choose “Add Remote”. If you don’t see “Add Remote” here, you may be using PyCharm Community Edition, which doesn’t support remote interpreters.
Now choose “docker-compose”, and almost everything will be pre-configured, I only had to add the docker-compose.dev.yml configuration to the list. We need to add it manually as this isn’t part of a standard docker project. You can do this with the ‘+’ button underneath the list of configuration files.
Now, we should set up the path mapping. We are inserting all our project code into the container’s “/app” directory. So let’s add this on the Project Interpreter screen, use the ‘…’ button next to the Path mappings field to change them.
After you close the preferences window, you can just use the regular Run and Debug buttons to start and debug your project:
To celebrate, let’s change “Hello World!” to “Hello from Docker within PyCharm!”
The Python debugger got forty times faster for Python 3.6 projects, and up to two times faster for older versions of Python
We’ve added support for the six compatibility library
Unit test runners for Python have been rebuilt from the ground up: you can now run any test configuration with PyCharm
Zero-latency typing is now on by default: typing latencies for PyCharm 2017.1 are lower than those for Sublime Text and Emacs
Support for native Docker for Mac – no more need to use SOCAT! (only available in PyCharm Professional edition)
This Monday Jim Fulton, one of the first Python contributors, hosted a webinar about storing JSONB documents in PostgreSQL. Watch it now:
Known mostly for its mature SQL and data-at-scale infrastructure, the PostgreSQL project added a “JSONB” column type in its 9.4 release, then refined it over the next two releases. While using it is straightforward, combining it in hybrid structured/unstructured applications along with other facilities in the database can require skill.
In this webinar, Python and database consultant Jim Fulton shows us how to use JSONB and related machinery for pure and hybrid Python document-oriented applications. We also briefly discuss his long history back to the start of Python, and finish with his unique NewtDB library for native Python objects coupled to JSONB queries.
Jim uses PyCharm Professional during the webinar. PyCharm Professional bundles the database tools from JetBrains DataGrip, our database IDE. However, the webinar itself is focused on the concepts of JSONB.
PyCharm 2017.1 has several notable improvements, but there’s one that’s particularly fun to talk about: debugger speedups. PyCharm’s visual debugger regularly gets top billing as a feature our customer value the highest. Over the last year, the debugger saw a number of feature improvements and several very impressive speedups. In particular, for Python 3.6 projects, PyCharm can use a new Python API to close the gap with a non-debug run configuration.
If you’ve been to PyCon or EuroPython and come by our booth, chances are you’ve seen Elizaveta Shashkova talking about the debugger to a PyCharm user, or giving a conference talk. Let’s talk to Liza about her work on PyCharm, the debugger, and her upcoming talk at PyCon.
Can you share with us a bit of your background and what you do on the PyCharm development team?
I started my career at JetBrains as a Summer Intern two and half years ago – I implemented a debugger for Jinja2 templates and Dmitry Trofimov (the creator of the PyCharm’s debugger) was my mentor. After that, I joined the PyCharm Team as a Junior developer and implemented a Thread Concurrency Visualizer under the supervision of Andrey Vlasovskikh, and my graduation thesis was based on it.
At the moment I’m supporting the Debugger and the Console in PyCharm.
People really like PyCharm’s debugger. Can you describe how it works, behind the scenes?
The debugger consists of two main parts: the UI (written in Java and Kotlin) and the Python debugger (written in Python). The most interesting part is on the Python side – the pydevd module, which we share with PyDev (the Python plugin for Eclipse).
We don’t use the pdb standard debugger for Python, but we implement our own debugger. At first glance, it’s quite simple, because it’s based on the standard sys.settrace() function, and in fact it just handles events which the Python interpreter produces for every line in the running script.
Of course, there are a lot of interesting frameworks or Python modules, where debugging doesn’t work by default, that’s why we add special support inside the debugger: for PyQt threads, for interactive mode in pyplot, for creation new processes, for debugging in Docker and others.
The Cython extensions gave a big speedup. Can you explain how it works and the performance benefit?
Yes, Cython speedups were implemented by Fabio Zadrozny and they gave a 40% speed improvement for the debugger. Fabio found the most significant debugger bottlenecks and optimized them. Some of them were rewritten in Cython and it gave even more – a 140% speed improvement. Fabio has done a really great job!
On to new stuff. The upcoming PyCharm, paired with Python 3.6, gives some more debugger speedups, right?
Yes, as I’ve already mentioned, for Python interpreters before version 3.6 we used to use the standard Python tracing function. But in Python 3.6 the new frame evaluation API was introduced and it gave us a great opportunity to avoid using tracing functions and instead implement a new mechanism for debugging.
And it gave us a really significant performance improvement, for example, in some cases the debugger has become 80 times faster than it used to be, and it has become at least 10 times faster in the worst case. In some partial cases, it has become almost as fast as running without debugging.
We have so many users’ reports about debugger’s slowness, and now I hope they will be happy to try the new fast version of the debugger. Unfortunately, it’s available for Python 3.6 only.
What changed in Python 3.6 to allow this?
The new frame evaluation API was introduced to CPython in PEP 523 and it allows to specify a per-interpreter function pointer to handle the evaluation of frames.
In other words, it means that we can get access to the code when entering a new frame, but before the execution of this frame has started. And this means that we can modify the frame’s code and introduce our breakpoints right into bytecode: execution of the frame hasn’t started yet so we won’t break anything.
When we used the tracing function, the idea was similar: when entering a new frame we checked if there are any breakpoints in the current frame and, if they exist, continue tracing for every line in the frame. And sometimes it led to the serious performance problems.
But in the new frame evaluation debugger, this problem was solved: we just introduce the breakpoint into the code and the other lines in the scope don’t matter. Instead of adding an additional call to the tracing function for every line in the frame, with Python 3.6 we add just one call to “breakpoint”, and that means that the script under debugging runs almost as fast as without the debugger.
Congratulations on your Python 3.6 Debugging talk being accepted for PyCon. Who will be interested in your talk?
This talk will be interesting for people who want to learn something new about the features of Python 3.6. Also, it will be useful for people who want to learn yet another reason to move to the Python 3.6: a fast debugger, which should appear in many Python IDEs.
Moreover, after the talk people will understand, how the PyCharm’s debugger works, and why such fast debugging wasn’t possible in the previous versions of Python.
This talk is for experienced Python developers, who aren’t afraid of calling Python’s C API functions and doing bytecode modifications.
What is the next big thing in debugging to work on in the next year?
We have a rather old and important problem in the debugger related to the evaluation and showing big objects in the debugger. This problem has existed from the beginning, but it has become really important during the last few years. I believe it gained visibility due to the increased number of scientists who use PyCharm and work with big data frames. At the moment we have some technical restrictions for implementing this feature, but we’re going to implement it in the near future.
We hope that there will be no major bugs in the RC build, however, should you encounter any problems, please report them to YouTrack – we’ll still have a bit of time to fix stuff before the final release.
Stay tuned for a PyCharm 2017.1 release announcement and follow us on Twitter.