September feels just right to start learning, or continue with even more effort and joy. And today feels just right for the next post in our series covering learning methods and tips & tricks designed to help you to learn more effectively and make you more excited about learning Python with PyCharm Edu!
Last time we posted about how to make your IDE suit your needs and preferences, and help you stay focused and avoid distraction. Today we’re going to focus more on the process of writing code.
Learn faster with a smart code editor
As you learn to code, it is encouraging to have an intelligent assistant right inside your IDE. Language-specific syntax & error highlighting are there to help you avoid code mistakes. You can use code formatting capabilities to learn how to style your code. And you won’t get stuck with your Python coding with code completion and quick documentation at hand.
After a strong start, we continue our early access program (EAP) with its second release. Download EAP 2 now!
Testing RESTful Applications
Many of us work on web applications which expose a RESTful API, or at least an API that pretends to be RESTful. To test these some of us use cURL, some browser extension, or some other piece of software. There is a REST client in PyCharm, but we’ve decided it can use some improvement, so we’re making an all new one.
The new REST client is entirely editor based, you write your request in a file, and then run the request to get a response. Sounds easy enough, right?
We’ll start out by creating a new todo. This is done by POST-ing to the /todos/ endpoint. To use the new PyCharm REST client, we should start by creating a .http file. If we don’t intend to save this, we can create a scratch file. Press Ctrl+Alt+Shift+Insert (Shift+Command+N on macOS) to start creating a scratch file and choose ‘HTTP request’ as the type. Let’s type our request into the file:
### Post a todo
"task":"Create my task!"
Now click the green play button next to the first line, and you should see that the task was created:
You can see the response in the Run tool window, and you might also notice that PyCharm wrote a new line in our file with the name of a .json file. This file contains the response, so if we Ctrl+Click (Cmd+Click) the filename, or use Ctrl+B (Cmd+B) to go to definition we see the full response in a separate file.
Those files become really useful when we do the same request a couple times but get different results. If we use a GET request to get our todo, and then use a PUT to change it, and redo our GET, we’ll now have two files there. We can then use the blue icon with the arrows to see the difference between the responses:
At JetBrains, we believe it’s important to involve our customers at an early stage. Before releasing a new version, we test our software to make sure everything works well. However, in software development there are as many unique configurations as there are developers. Therefore we provide Early Access Preview (EAP) versions free of charge, so you can make sure all of our improvements work in your environment prior to release.
More and more Python developers are doing scientific development, and we’ve created a new feature especially for them. When we detect that you use a scientific package in your project (like numpy or pandas), we will suggest to enable scientific mode:
If you chose not to use scientific mode when we asked, you can always enable it later in View | Scientific Mode.
In this mode, we open several tool windows by default: the Python console (bottom left), a variable viewer (bottom right), and a documentation viewer (top right).
The new scientific mode makes exploring data a lot easier: you always have an overview of the variables in your current session, and we look up the documentation of the method you’re using as you’re typing. No longer will you need to have your browser open with 20 tabs just to find the details about Pandas methods. In other words: you can focus on your data, rather than the details of the library you’re using.
You can now configure exception breakpoints to be conditional, or to evaluate a statement and log
For those who want to install PyPI packages into Conda envs, we’ve added PyPI and Conda Modes for the PyCharm Package Manager. Simply click the Conda button on the right to switch between Conda and PyPI mode:
Further Fixes and Improvements
We now correctly identify test classes, even if they have methods that don’t start with test_
Many inspections and autocomplete fixes, for example we’ve improved a couple of cases where we misinterpreted __all__, and we resolved some NamedTuple issues
Bugs involving Twisted Trial
Some improvements to editing BDD feature and step files
As always, EAP versions of PyCharm Professional Edition are free of charge, and will work for 30 days from the day of the release. If you upgrade regularly during the EAP program, you will be able to use PyCharm Professional Edition for free for the entire duration.
Testing in Python often requires you to spend a lot of time looking at dots in your terminal. PyCharm, though, can make testing easier and friendlier! When you configure PyCharm to run your Python tests for you, you can get useful visual feedback, tools for quickly re-running your test suite or just failing tests, and more.
Besides teaching Python, Kenneth Love (@kennethlove) is a husband and father. He is the teacher for the popular Python and Django courses at Treehouse and has created a few Python libraries (django-braces being the most popular), worked at most levels of the web stack, and was the Creative Director of a newspaper. He likes tabletop games, activism, and dry humor.
Connecting to virtual machines for debugging is a feature that is only available in PyCharm Professional Edition. If you haven’t got it yet, you can get a free trial from our website.
Check out Reddit
To start, we need to get the code, so let’s go to Reddit’s GitHub account, and clone their repo. Please make sure to check it out in a folder called ‘reddit’, you’ll run into issues later if you don’t.
If you’re using Windows, there’s an important side note here: Reddit uses shell files, config files, and python files to configure the Vagrant box, and you need to check these out with Linux-style line endings. The easiest way to get this is to clone my fork instead, I’ve added a gitattributes file to ensure the correct line endings.
At the time of writing, there’s an issue in the reddit codebase that prevents you from running the code. GitHub user ironyman has fixed the problem, however, his pull request hasn’t been accepted. If you checked out the code from reddit, you’ll need to apply this change manually. I’ve already made this change in my fork, so if you’re not sure how to apply the change, simply clone my fork instead.
After checking out the code (and applying the fix if necessary) you can start the vagrant VM. Either run vagrant up on the command line or choose Tools | Vagrant | Up in PyCharm (if you’re asked to choose between default and Travis, choose default). After a couple of minutes of installation, you should see a line similar to:
==> default: reddit:15655 started 03baa2a at 15:40:26 (took 2.22s)
Now you’re probably excited, and want to see your very own Reddit. To do so, we’ll need to first take a quick detour to our hosts file. If you’re on macOS or Linux, this is in /etc/hosts, on Windows it’s in C:\Windows\System32\drivers\etc\hosts. Reddit’s Vagrantfile statically specifies the IP address, and we’ll need to add it here to access Reddit over the reddit.local domain name they use for development:
And now you can open your browser, and check out your local reddit instance by going to http://reddit.local (the http is important this time!):
Another provisioning script in the Vagrantfile will populate your local reddit setup with test data. So if it’s still empty, just wait a little, and there’ll be some content.
Let’s get hacking!
To make PyCharm able to do anything with the Reddit inside of our VM, we need to configure the Python interpreter. Go to Settings | Project Interpreter, and choose ‘Add Remote’:
Then select ‘Vagrant’, and make sure to select the ‘default’ machine. Reddit also specified another machine (Travis) to make unit testing easier, but we’re not using that machine here:
Furthermore, Reddit is a Python 2 application, and they’ve configured it using the system Python, so we don’t need to select a virtualenv.
If you get a scary looking warning about the remote host identification, don’t worry, this is normal for Vagrant boxes that are regularly recreated.
Reddit is a complex application, and they use overlayfs to merge configuration files on the VM with the code base mounted from the host machine. Therefore even though PyCharm correctly detects that the code is mounted in /media/reddit_code, we will need to manually add a path mapping.
Click the ‘…’ button next to the path mappings in the interpreter settings window and add the project directory (ending in ‘reddit’) on your local machine, and /home/vagrant/src/reddit as the remote path:
After we do this, we’re able to run and debug Python scripts in the Vagrant box. However, we’re not done yet, Reddit is a Pylons application, and uses a tool called ‘paster’ to start the application.
Running Reddit from PyCharm
If we open an SSH terminal window (Ctrl+Shift+A, ‘Start SSH session’, and then pick the Vagrant box) and run which paster we can see that they’ve installed the script in /usr/bin/paster. By running cat /usr/bin/paster we can see that it’s a very simple two-line script. The easiest way to enable PyCharm to run reddit, is to copy this script into the project. So let’s create a script called paster.py in the root of the project, with the following code:
If you’re using my fork rather than the code checked out from Reddit, this file should already be there.
Another thing to do before running Reddit from PyCharm is: stop the Reddit that’s already running on the VM (you can’t have two applications listening to the same port). So let’s go back to the SSH terminal window, if you’ve closed it, you can go to Tools | Start SSH Session to start a new one. Reddit uses Upstart to run all of its services, and we can use standard upstart commands to manage these services. To stop the main application, run sudo initctl stop reddit-paster.
At this point, we can create our run configuration, and start Reddit from PyCharm. Be sure to specify full paths, if you specify relative paths, there’s a high chance you’ll get a FileNotFoundError somewhere.
Click, OK. And now let’s click the bug icon (or use Shift+F9) to start debugging. And if everything is setup correctly, you should see:
And if you open http://reddit.local (again, the http:// is important) in your browser, you should see your local instance of Reddit running. If you’re wondering how we’re seeing our application on port 80 even though we specified 8001, Reddit is using HAProxy in the VM.
Let’s Mess with Reddit!
After all this effort, we should do something more interesting than just clicking play, right? So let’s change the main menu to say “Hello from PyCharm”:
To find out how to do this, we should have a look at the routing first, to see which controller serves the front page. Use Ctrl+Shift+N to find routing.py (in reddit/r2/r2/config/). After scrolling through this file, we find (on line 283):
In Pylons, this means that we need to have a HotController, which exposes a ‘listing’ action. So let’s find the controller, press Ctrl+N and look for HotController, we find it in listingcontroller.py. In HotController, there’s a GET_listing method that looks promising. So let’s put a breakpoint there.
When we refresh the page in the browser, PyCharm breaks here, so we’ve found the right place. FYI: if you take some time debugging, you might get a 504 gateway timeout in the browser, that’s just because HAProxy gets impatient, the backend Python application is still running.
Let’s use step over (F8), and step into (F7) to see where the code goes. We see a lot of places in the Reddit codebase and the Pylons code, but we don’t find anything useful.
The code here calls a ListingController.GET_listing method on its superclass. Let’s go have a look at that method, we can go there by putting the cursor on the name where we’re calling it (on line 573) and then us Ctrl+B to navigate to the function’s declaration (it’s on line 115). At the end of that function there’s a call to self.build_listing, which looks interesting. So let’s follow the path further (Ctrl + click on build_listing).
In this method we see nav_menus=self.menus, which looks promising. So let’s go to self.menus, and we see an empty list. Let’s see what happens if we change this:
If we go to the console in the debugger, we can see that the server automatically reloaded with our code changes. If your server didn’t automatically reload, check to see if you have --reload in the script parameters of your run configuration. You can manually restart the server by using the icon with the green round arrow in the top-left of the debugger tool window. Let’s mute the breakpoints, and refresh:
Close, but no cigar:
Let’s undo our changes first. And then let’s see if there’s another way to find the menu we’re interested in. If we look in the Chrome inspector, we find that the menu we’re interested in has the CSS class tabmenu:
Let’s use Find in Path (Ctrl+Shift+F) in the r2 directory to see if we find anything if we look for tabmenu. And we find a file menus.py, let’s have a closer look!
And what do we see in this file? class NavMenu, which in the docstring says it generates a navigation menu. So let’s see if this is the right place. Let’s put a breakpoint here (line 223 in menus.py), and then go to the breakpoints overview (Ctrl+Shift+F8) to remove our old breakpoint:
Don’t forget to unmute the breakpoints, and then let’s refresh! Now when we see the variables in this point, there’s an interesting looking options list. So let’s inspect it further, and we notice that every NamedButton has a title variable.
Clicking through this for every menu (and there are a couple) seems like a lot of work, so let’s make our life easier. We’re interested in seeing the titles of the menus to make sure that the one we’re interested in is here. In other words, we only want to see: [button.title for button in options]. We can check if our list comprehension is correct by going to the ‘Console’ tab of the debugger and running it. Looks good!
Let’s go back to the breakpoints overview: for now, we don’t want to stop, we only want to log, so we’ll uncheck ‘Suspend Thread’, and check ‘Evaluate and Log’ with that expression:
Now when we refresh the page, and look in the console tab, we see which menus are being handled by this class:
We can clearly see that the ‘hot’, ‘new’, ‘rising’, etc menu is here. So let’s break on that one, and that one alone:
When we refresh, we hit the breakpoint again (we may need to switch from the ‘Console’ tab to the ‘Debugger’ tab to see the variables), we can see in the Frames view that we’re being called by build_toolbars on line 971 in pages.py:
Let’s remove our breakpoint here, and have a look at the build_toolbars function. Use Ctrl+Shift+N to open pages.py, and then use Ctrl+G to go to line 971 (or click in the Frames view). Looking around in this function, we see that the main_buttons list is being populated with ‘hot’, ‘new’, etc. so it looks like we’ve found our menu!
Let’s see what happens when we add our message here:
If you have debug mode enabled, which you should have with the Vagrant box, you’ll just see the stack trace, but this picture is funnier. If we have a look at the stack trace though (if you don’t have debug mode enabled, you can see it in the debug console in PyCharm). We can see that we’re getting a KeyError in strings.py on line 249.
Looking at the code there, it looks like Reddit doesn’t know how to make plurals out of the words we’ve added, and that’s what’s breaking it. So let’s add these words to the list:
And let’s refresh again to see if we’ve fixed Reddit:
We didn’t just fix it, we successfully altered the menu!
We just used the debugger to quickly get to learn how a new, large, unknown codebase works. By finding clues, and then following them until we found what we were looking for, or found that we were on the wrong path.
Breakpoints are useful, but they become very powerful when you explore the additional options they have (like conditions, and logging expressions).
Let us know in the comments if you enjoyed these shenanigans, and what kind of content you’d like to see on this blog in the future!
We often get questions about the difference between PyCharm Community Edition and PyCharm Professional Edition. We receive further questions about the difference between an individual and a commercial subscription for PyCharm Professional Edition. So let’s try to address some of these questions here:
PyCharm Community Edition
The community edition of PyCharm is Apache 2 licensed: meaning it is free and open source and you can go to GitHub, and look at the source code. You’re free to use it whenever, and wherever you like, including at work. Additionally, you can fork and modify it. See the python subfolder README.md for details about PyCharm rather than IntelliJ IDEA.
What can I use PyCharm Community Edition for?
Let’s go to the LICENSE.txt in the root of the GitHub repo. JetBrains’ open source projects are generally licensed under the Apache 2.0 License. This means that you can use it anywhere you’d like to, and modify it freely. There are some restrictions, which we’ll look into below.
Can I use PyCharm Community Edition at work?
Yes, you can. You are allowed to use PyCharm Community Edition for commercial use.
Can I use PyCharm Community Edition at my university?
Yes, you can. The Apache 2.0 license doesn’t just permit using this code, but also allows modification. However, before you release YourNamePyCharm, you need to be aware that that the JetBrains and PyCharm trademarks are restricted. So if you do want to fork PyCharm, you will need to take out our branding. Furthermore, in your derived version, you will need to credit us. So you could make ‘YourNameIDE’, with a notice that it’s based on software made by JetBrains.
What can’t I do with PyCharm Community Edition?
There are some restrictions that apply when you fork PyCharm. The Apache 2.0 license requires:
You need to attribute us. So if you fork PyCharm, you’re not allowed to remove all notices that JetBrains made it
You need to include a full copy of the Apache 2 license
You need to include the NOTICES file
We’re not lawyers, so please be aware that this is not legal advice.
So why would I use PyCharm Professional Edition?
The professional edition of PyCharm gives you access to additional features that you don’t get in the community edition:
Support for Remote Development. PyCharm Pro can deploy and debug python code running on remote machines, virtual machines, and Docker containers.
Database support. PyCharm Professional takes its database support from DataGrip, the SQL IDE by JetBrains. This means you can explore your database within the IDE, and get schema-aware code completion when writing an SQL statement in Python code.
Alright, so what about Individual v Commercial subscriptions?
Many people get confused when we tell them that they are allowed to use a personal license at work. We believe though, that it’s important that developers can use the tools that are right for the job, and therefore we offer low price options to individual developers.
The difference between personal and commercial licenses isn’t about who uses the software; it is about who owns the software.
The personal license is yours: you pay, and you own it. You can use it at work, and if you change jobs you can use it at your next job.
The commercial license is your employer’s: they pay, and they get to keep it if you leave. However, if you buy it and get reimbursed by your employer, you still need a commercial license: if the employer pays, it needs to be a commercial license.
Can I use my license on multiple machines?
Individual licenses: yes. Commercial licenses: yes, as long as your user name (login) is the same on all the machines that you’re running it on.
Can I still use PyCharm Professional Edition after my subscription expires?
If you’ve had a subscription for at least one year: yes. You have a perpetual fallback license for the version that was released one year before your subscription expired, and all its minor updates. See here for details.
Are you a core contributor or committer for an open source project, no matter the size? As long as you don’t make money from your project, you can likely get a free license to work on your project. Apply for an open source license
If you don’t have a PyCharm Professional subscription yet, and you’d like to support the Django Software Foundation, we have good news for you: if you get PyCharm using the ISUPPORTDJANGO promo code, you get a 30% discount, and the full price you pay will go to the Django Software Foundation to support future Django development.
PyCharm Professional has long had great support for developing Django applications, including a run configuration tailored to the Django server. This winds up being a wonderful workflow, with a tool window showing the server output.
Sometimes, though, you hit a problem and want to debug your code. You stop the server, run it under the debugger, and do your debugging. Which also works really well: PyCharm’s visual debugger is a key selling point and integrates nicely into Django (e.g. template debugging.) And recently, PyCharm’s debugger has undergone dramatic speedups, especially when using Python 3.6. In fact, running under the debugger is approaching the speed of normal run configurations.
But still, it’s irritating to stop you server, restart under the debugger, stop the server, and restart under the regular runner. It messes up your flow, and you’ll be tempted to blasphemy of debugging with print().
So let’s break with the past with a crazy idea: always run under the debugger when developing Django.
Let’s start with the “normal” path. PyCharm usually makes a Django run configuration automatically, but if not, here’s how to do so: Run -> Edit Configurations -> + -> Django Server:
With this run configuration selected, click the green play button to start the Django server:
When done, click the red button to stop the server:
You can also restart the server, use keystrokes instead of the mouse, let Django restart automatically on code changes, get linked tracebacks to jump directly to errors, etc.
You’re likely familiar with all that. Let’s now do the same thing, but running under the debugger. Click the green button with the bug on it to start that Run/Debug configuration, but running under the debugger instead of the normal runner:
Note: If you get a link saying you can compile the debugger speedups under Cython (on Windows, we ship it by default), click the link, the speedups are worth it.
Let’s compare the restart-on-edit times side-by-side:
As you can see, the debugger’s restart hit isn’t that big, certainly compared to the savings versus other parts of your development workflow (e.g. moving print statements and reload versus moving breakpoints without restarting.)
I can proceed as normal, writing new views and trying them in the browser or the built-in REST client. But when I hit a problem, I don’t reach for print() — no, I set a breakpoint!
In fact, I can set a breakpoint in a template and then poke around at that line of template execution:
Much better than random print statements, which require a restart each time I want to inspect something.
Testing…Under the Debugger
Always develop under the debugger? Sounds weird…the debugger is for debugging. Let’s make it weirder: always do testing under the debugger.
For example, you might run Django tests under PyCharm’s handy Django Test run configuration:
But you can also launch your test runner under the debugger:
Test running under the debugger is an interesting case. Often with testing you are exploring, and sometimes, you are actually intending to produce bugs when writing a test case. Having the debugger close at hand can fit well with TDD:
No free lunch, of course. Even if we have increased performance, dramatically for Python 3.6, there is still a speed hit. Not just on startup, but on execution.
You may run into quirks in obscure usages. For example, when running pytest tests under the debugger, you can’t get to the console (known issue slated to be fixed in 2017.3.) Also, tracebacks might be longer since the code is running under pydevd.
At two recent conferences I mentioned this — do your Django development under the debugger — to people who visited the booth. The reactions were fascinating, as a look of horror turned to confusion then curiosity ending with interest.
For some, it might never fit your brain. For others, it might make all the sense in the world. Want to tell me I’m crazy? It’s possible. Leave me a comment below with any feedback.