PyCharm 2018.1.3

PyCharm 2018.1.3 is now available for download. Please update PyCharm by choosing Help | Check for Updates, or by downloading the new version from our website.

What’s New

Python Console Messages

Since several versions, PyCharm has folded the startup messages of the Python interpreter in the Python console. This means that you have more room to write your own code, yet you can still see the startup messages by hovering over the folded output.

Unfortunately, if you had configured custom startup code (which you can do in Settings | Build, Execution, Deployment | Console | Python Console), your own code’s output would be folded as well. In PyCharm 2018.1.3, we’re making sure to only fold large startup messages (like IPython’s), and show the output of your custom code:

IPython Console

Further Improvements

  • In the RC, there was an issue with choosing a service in a Docker Compose project, this was resolved in this version.
  • An issue with remotely debugging Python code on Windows computers was resolved
  • Several issues regarding Angular 6 were resolved. Did you know that PyCharm Professional Edition comes with all JavaScript support from WebStorm?
  • Read more in our release notes
Posted in Release Announcements | Tagged | 2 Comments

PyCharm 2018.1.3 RC

We’re happy to announce that the release candidate for a new version of PyCharm is now available: get it now from our confluence page.

What’s New

Python Console Messages

Since several versions, PyCharm has folded the startup messages of the Python interpreter in the Python console. This means that you have more room to write your own code, yet you can still see the startup messages by hovering over the folded output.

Unfortunately, if you had configured custom startup code (which you can do in Settings | Build, Execution, Deployment | Console | Python Console), your own code’s output would be folded as well. In PyCharm 2018.1.3, we’re making sure to only fold large startup messages (like IPython’s), and show the output of your custom code:

IPython Console

Further Improvements

  • An issue with remotely debugging Python code on Windows computers was resolved
  • Several issues regarding Angular 6 were resolved. Did you know that PyCharm Professional Edition comes with all JavaScript support from WebStorm?
  • Read more in our release notes
Posted in Early Access Preview | Tagged | Leave a comment

Webinar Recording: “Set Theory and Practice: Grok Pythonic Collection Types” with Luciano Ramalho

Last Wednesday we were fortunate to have Luciano Ramalho with us for a webinar “Set Theory and Practice: Grok Pythonic Collection”, using material from his book Fluent Python. The recording is now available.

Luciano covered many topics related to the Python data model with an emphasis on collection-like objects:

  • Python collection types
  • Theory and algebraic logic behind set-less and set types
  • Python protocols and operations for collections
  • Code examples for implementations of kinds of sets

Luciano’s slides and repo are also available. If you have any questions for us or for him, please feel free to leave comments below on this blog post.

-PyCharm Team-
The Drive to Develop

Posted in Video, Webinar | Tagged | Leave a comment

Interview with Luciano Ramalho, webinar presenter this Thursday

Python’s combination of power and simplicity has long been one of its key selling points. This includes its data model. Luciano Ramalho, who is our guest presenter for the next webinar, recently wrote about this in Fluent Python, an O’Reilly book widely considered one of the best recent books on Python.

r2

There’s another well-known strength of Python: its community, and Luciano is one of those long-time key people who is warm, inviting, smart, and part of Python’s appeal. We’re lucky to have him with us for the last webinar before PyCon and decided to do a quick interview with him.

Give us a sneak peak on what you’re going to discuss in the webinar and what audience it is aimed at.

What makes an API “Pythonic”? That’s somewhat subjective, but we can learn a lot by looking at concrete examples like the built-in set types: set and frozenset. They “feel right” because they leverage some of Python’s strongest features. The set types are iterable, so they play well with core Python constructs and with powerful libraries like itertools. They also provide handy methods that consume iterables of other types, so you can compute the union of a set and a list, for example. And they leverage operator overloading, enabling concise coding of set expressions.

After covering the strengths of Python sets, I will show how to implement a new set class optimized for dense sets of integers. This talk will be accessible to anyone who knows how to write object-oriented code in Python.

This is part of what you evangelize in the book about the hidden beauty of Python’s data model. Can you explain more about that?

Once you know the basics, I believe anyone interested in mastering Python needs to learn about the Data Model: the set of core interfaces that make the language and standard library so consistent and so powerful.

For example, we expect any collection type in Python to be iterable, so that we can use it in for loops and with handy functions such as sorted, any, all, dropwhile etc. The Data Model specifies the interfaces you need to implement to build an iterable collection.

For debugging and testing, we want objects to be printable and comparable. Supporting operators such as + to join custom data structures? The Data Model explains how to do all of those things and more.

How are book sales and what was it like writing an O’Reilly book?

I was lucky because I decided to write an intermediate Python book when the language was growing faster than ever, and there were lots of basic books but not so many intermediate ones. So Fluent Python was O’Reilly’s best selling item the month after its release and has been going strong since then. It’s also been translated into 7 languages so far.

I’ve always been an avid consumer of O’Reilly books, so writing one for them was a dream come true. My editor, Meghan Blanchette, was excellent, and I could not have asked for a better team of tech reviewers: Alex Martelli, Anna Ravenscroft, Lennart Regebro, Leonardo Rochael, and Victor Stinner.

I also enjoyed writing the book in Asciidoc: I was a fan of RestructuredText, but Asciidoc has friendlier syntax and is better suited for book writing because it was designed to target DocBook, a publishing industry XML standard. You can render Asciidoc to HTML, ePub, PDF and other formats using Python tools, but the best toolset is Asciidoctor, written in Ruby.

Let’s go back in time. You and I became really good friends in the late 90’s, when Python was really taking off and you were the key person in Brazil. Can you give us your Python origin story?

For me, the best thing about getting involved with Python was making friends like you, Paul. I started doing Web development in 1994, using the most popular language for that purpose at the time: Perl. Before that, I had taught myself object-oriented programming with Smalltalk, but Perl did not have a strong OO culture and class libraries, even after Perl 5 came out.

In Perl forums, it was common for people to suggest looking at Python code for inspiration about how to design classes in a language that had more similarities with Perl than Smalltalk or C++ did. So I decided to check out that obscure language and fell in love as soon as a read the first chapter of Guido van Rossum’s tutorial. It seemed as elegant as Smalltalk, but with a more readable Algol-like syntax, and it was as practical as Perl.

In 1998 I started a company focusing on publishing systems for large-scale Web sites. I chose Python as our language, and I chose an even more obscure but incredibly advanced Web framework called Zope as the basis of our systems. Before that year was over, we had deployed a new publishing system for the most important Brazilian IT news portal at the time, and it became one of the first marketing cases for Zope worldwide. You were one of the makers of Zope, and that’s how our friendship started.

You historically have done a ton of teaching. What’s unique about Python as a teaching language?

Python is an awesome teaching language for several reasons. First, because it is a “real” language — not a toy — so people use it in many domains, and there are Python libraries for a lot of different domains. This is the reason why it replaced Scheme as the main language at MIT’s introductory programming classes.

Second, it is easy to learn because it has a simple syntax, very consistent semantics, and a fail fast philosophy: Python “refuses to guess” so it raises exceptions where other scripting languages fail silently or behave unexpectedly (looking at you, JavaScript, Perl, and PHP).

Third, Python’s interactive console is a great learning and exploration tool. IPython and Jupyter Notebooks make that even better. Now we have a virtuous cycle: as Python becomes more widely used for teaching, more teaching resources are available for it, like Philip Guo’s Python Tutor, programmable devices like the BBC micro:bit and Adafruit’s CircuitPython products, MOOCs, beginner’s books, academic textbooks, etc.

You’ve always been a language junkie and that’s accelerated recently. Based on that, what’s the next big thing for Python, and the next big thing outside of Python?

I learned about a dozen languages before Python, but I got a bit lazy to learn others because Python is so practical and fits my brain so well. I was very interested in Ruby around 2006, and a few times I’ve tried hard to like JavaScript, but failed. I became more interested in concurrency because of the success of Node.js. But JavaScript, Python, and Ruby are not ideal to tackle large-scale concurrency challenges. Yes, we can do efficient “IO bound” programming in these languages, but I’ve learned that IO bound systems become CPU bound as they grow: lots of small functions slow down the event loop, which slows down everything else. To deal with this, the language runtime must be able to spread the load over multiple CPU cores, so that no single slow function blocks all others. So I am very interested in languages that were designed from the start for concurrent programming. Right now, I am focusing on Go and Elixir.

The next big thing for Python seems to be type hinting and all the tooling it will enable. But I would be most excited if we got rid of the infamous GIL (Global Interpreter Lock), or found a way to work around it that’s mostly transparent to the user. However, the GIL makes writing Python extensions in other languages easier than it would be otherwise. And Python owes much of its popularity to the huge number of extensions people have written. So the GIL is like a deal we made with the devil. We had a lot of success… Can we walk away from that deal?

After Go and Elixir, I want to dive into Rust, but for a different reason: it seems to be a great language for writing Python extensions. I also want to try Cython. As long as the GIL is around, we might as well enjoy its benefits by writing more awesome extensions.

Posted in Interview, Webinar | Tagged | Leave a comment

Interview: Python Development at edX with Robert Raposa and Ned Batchelder

In this interview with Robert Raposa and Ned Batchelder, software architects at edX, we’re going to look under the hood of the edX project, where more than 95% of the entire codebase is in Python. Robert is a core contributor to the Open edX LMS and Studio products, as well as their supporting infrastructure. Ned is on the Open edX team, advocating for the community using the software. We’re going to learn about the project, how they develop, what their technology preferences are, and some of the reasons they chose Python as the main language. Many edX core developers are using PyCharm, so we’ll also learn what developers value the most in it.

edx_logo_final

– Hi Robert and Ned, could you tell us a little bit about yourselves?

Robert: After getting my CS degree, I have spent my entire career working on software related to education. I started off my career coding software used for management training. I then switched to a product for administrators and teachers of pre-school. At this point, I decided to try teaching Math for a year. It turns out that writing software to facilitate teaching is much easier than teaching, so I next worked on a K-12 LMS for many years before finally landing at edX as a software architect. Like most edX employees, I was so excited about joining a nonprofit with an incredible mission and commitment to open source.

Ned: I’ve been deeply embedded in the Python community for a long time. I loved edX’s mission, and also its open source approach.

– For those of us who are only slightly familiar with edX, could you tell us what it’s all about?

Robert: edX was initially started by MIT and Harvard in 2012. Today, it is one of the leading MOOC providers, and the only one that is both a nonprofit and open source. Its mission is to increase access to everyone, everywhere. We offer courses from our partners from many of the top universities across the globe, as well as other nonprofits and institutions.

The core of the Open edX platform, the open source platform upon which edX.org is built, includes two products: the Learning Management System (LMS) and Studio. Studio is an authoring tool used to create the courses that are run and taught through the LMS. There are also many supporting subsystems, for example, for accessing analytics or for discovering courses. These subsystems are also made available through the larger Open edX infrastructure.

– Tell us about how edX is organized as an open source project.

Ned: There are over 800 sites running Open edX, with over 15,000 courses available across the globe. We are making a push to find more adopters, and they are easy to find, so the numbers keep going up! As a non-profit, we provide free online education. We also provide free tools for others to run their own online education. This is all part of our mission to increase access to education for everyone everywhere.

All of the Open edX source code can be found on github at https://github.com/edx. Here you will find 206 repositories, including many Django packages and other libraries that edX has created to support the entire development lifecycle. Well over 90% of what edX develops is open source. Most of the proprietary code concerns how edX markets and lists its courses, and Open edX includes an open source version used by our community members.

We’re a little different from many other open source projects, in that we run a site ourselves with the code we are writing. This creates some unique pressures. One of our challenges as an open source project is, how can we keep our own site running well while also delivering this complex suite of software to other sites, and accepting contributions from them? We don’t know the best answer yet, but we’re continuing to work on it.

– How big is the edX core team?

Ned: We have about 80 in engineers in Cambridge. We’re up to approximately 150 if you add contractors and regular contributors in the community. Given the pervasiveness of Python in our work, I’d say nearly every engineer is a Python developer.

– What are the main languages and technologies generally used at edX?

Robert: Python is used for the vast majority of code for Open edX. We use Django for our web applications, including the Django REST framework.

On the front-end, we have a lot of legacy Backbone.js and Underscore.js, but are slowly moving more and more to React. We also use Sass and Bootstrap.

EdX.org is hosted in AWS. Some example technologies we host there include Memcached, ElasticSearch, MySql, and Mongo. We use a mix of CloudFlare and CloudFront for CDN.

For development, Continuous Integration, and deployments we use a mix of Docker, GitHub, Jenkins, GoCD, Asgard, and Terraform, among others.

Finally, like any undertaking this large, we’ve got our special snowflakes like our Ruby-based discussion service that no one wants to work on except to rewrite it in Python, which still hasn’t happened.

– What’s so special about Python and why is it so widely used at edX?

Ned: Python has strong web development tools, and makes it easy to build quickly. We wanted to provide extensibility in the platform, so using an approachable language that would let people add packages to their installations was essential. The edX Studio authoring tool lets course teams add code to their courses, to randomize or grade assessments, so a dynamic language that let us execute that code was also really powerful.

Python being an open-source language, with a strong culture of open-source tools, has also been important for us, as an open-source project. It lets us affect the tooling we rely on, and it means that we can attract contributors who are already familiar with the Python world.

One last factor in choosing Python: edX started as an MIT project, and MIT’s teaching language is Python. Never underestimate the power of becoming familiar among people at a strong institution like MIT!

– Do you also do some data analysis or ML with Python?

Robert: In addition to using Python with Django, we also use Python for various scripts, linters, testing frameworks, and data analysis. My colleague Cale tells me he used a combination of ipython notebooks, pandas, and ggplot for his analysis work.

– Which Python version is currently in use at edX?

Ned: We are still using Python 2. We’ve been making some advances toward Python 3 where and when we can. As part of our Django upgrade process, we recently introduced tox in most of our repos to test against these various combinations. We’ll likely be switching to Python 3 with the rest of the Django community as they drop support for Python 2.

– What, in your opinion, are the main development challenges for edX developers today?

Robert: Keeping track of such a large codebase. We are trying to introduce more and more best practices as we can, and move more and more of the codebase in the right direction, but we have a lot of legacy to work with at this point. Like many development efforts, we have a big monolith as one of the many components of our architecture, and we are trying to work towards an architecture that is split enough, but not too much. It is a balancing act that is difficult to get right.

– There are many core developers at edX using PyCharm for their development. How does PyCharm help them be more productive?

Robert: We probably have about 40 developers using PyCharm. Many of the other developers end up using some combination of technologies like sublime, vim, and pdb. When I need to pair with someone who doesn’t use PyCharm, I often find myself asking how they can stand not being able to jump into the definition of a method.

Many people choose PyCharm for its debugging capabilities, as well as having an editor that understands Python. When you watch someone debug in a modern IDE, it is hard not to want to be able to do the same. For PyCharm users, we often use debugging, refactoring, autocompletion, version control, find definition or class, and PyCharm has great support for these technologies.

Over the last year, we’ve migrated our development environment from Vagrant to Docker. It was in tandem with PyCharm adding more and more Docker support. There have been some hiccups on this front, but it is nice to still be able to debug.

– Does PyCharm help boost team productivity on your project?

Ned: We really have a mix of people that like and dislike full IDEs. For those who like it, like myself, it definitely improves our workflow. For those who don’t, they think it doesn’t.

Since we need to accommodate both types of engineers, we generally rely on tools that work outside the IDE and run with our continuous builds to enforce rules like code style. Some of these tools have been difficult to fully integrate with PyCharm, but that would give us the best of both worlds.

– What about your individual productivity?

Robert: I have always used visual IDEs and can’t imagine why anyone would not want to. I find it much easier to do so many different things in PyCharm. I know I could use tools like grep, but why wouldn’t I want to search for code and be able to edit it and jump to definitions all in one seamless flow? I even like using the visual tools for resolving git conflicts.

There are also many things I do from the command line. It definitely has its place. I just find so many things that go more smoothly in PyCharm.

– How did you personally first learn about PyCharm?

Robert: I came to edX with a Java background and no experience with Python. At my last job we mostly standardized on Eclipse, but even that changed over time because you get what you pay for, I guess.

Other engineers at edX were using PyCharm, so that’s what I started to use, and I found it does a pretty decent job. I am often surprised by how much it can do given that Python is not compiled.

– Thank you for the interview, Robert and Ned!

If you want to learn more about Robert’s and Ned’s experiences, follow them on GitHub: Robert’s GitHub and Ned’s GitHub.

Posted in Interview | Tagged | Leave a comment

PyCharm 2018.1.2

PyCharm 2018.1.2 is out: download PyCharm now from our website.

What’s New

Docker Compose Improvements

Our Docker Compose interpreter in PyCharm 2018.1.1 starts your application service together with its dependencies, but leaves your dependencies running after shutting down the application. This has now been changed to match the command-line behavior, and will shut down your dependencies as well. Have you not tried using Docker Compose interpreters yet? Learn how to do so on our blog with Django on Windows, or with Flask on Linux.

Docker Compose users on Windows will be happy to learn that we’re now using named pipes to connect to the Docker daemon, which resolves an issue where some users were unable to run their scripts.

Further Improvements

  • The Python Console now receives focus when its opened
  • Various improvements to database support: columns that show the result of a custom function in MSSQL are now correctly highlighted, and more. Did you know that PyCharm Professional Edition includes all database features from DataGrip, JetBrains’ SQL IDE?
  • Improvements in optimizing Python imports
  • Various issues regarding React lifecycles have been resolved
  • Read more in our release notes

Posted in Release Announcements | Tagged | Leave a comment

Python 3.7: Introducing Data Classes

Python 3.7 is set to be released this summer, let’s have a sneak peek at some of the new features! If you’d like to play along at home with PyCharm, make sure you get PyCharm 2018.1 (or later if you’re reading this from the future).

There are many new things in Python 3.7: various character set improvements, postponed evaluation of annotations, and more. One of the most exciting new features is support for the dataclass decorator.

What is a Data Class?

Most Python developers will have written many classes which looks like:

Data classes help you by automatically generating dunder methods for simple cases. For example, a __init__ which accepted those arguments and assigned each to self. The small example before could be rewritten like:

A key difference is that type hints are actually required for data classes. If you’ve never used a type hint before: they allow you to mark what type a certain variable _should_ be. At runtime, these types are not checked, but you can use PyCharm or a command-line tool like mypy to check your code statically.

So let’s have a look at how we can use this!

The Star Wars API

You know a movie’s fanbase is passionate when a fan creates a REST API with the movie’s data in it. One Star Wars fan has done exactly that, and created the Star Wars API. He’s actually gone even further, and created a Python wrapper library for it.

Let’s forget for a second that there’s already a wrapper out there, and see how we could write our own.

We can use the requests library to get a resource from the Star Wars API:

This endpoint (like all swapi endpoints) responds with a JSON message. Requests makes our life easier by offering JSON parsing:

And at this point we have our data in a dictionary. Let’s have a look at it (shortened):

Wrapping the API

To properly wrap an API, we should create objects that our wrapper’s user can use in their application. So let’s define an object in Python 3.6 to contain the responses of requests to the /films/ endpoint:

Careful readers may have noticed a little bit of duplicated code here. Not so careful readers may want to have a look at the complete Python 3.6 implementation: it’s not short.

This is a classic case of where the data class decorator can help you out. We’re creating a class that mostly holds data, and only does a little validation. So let’s have a look at what we need to change.

Firstly, data classes automatically generate several dunder methods. If we don’t specify any options to the dataclass decorator, the generated methods are: __init__, __eq__, and __repr__. Python by default (not just for data classes) will implement __str__ to return the output of __repr__ if you’ve defined __repr__ but not __str__. Therefore, you get four dunder methods implemented just by changing the code to:

We removed the __init__ method here to make sure the data class decorator can add the one it generates. Unfortunately, we lost a bit of functionality in the process. Our Python 3.6 constructor didn’t just define all values, but it also attempted to parse dates. How can we do that with a data class?

If we were to override __init__, we’d lose the benefit of the data class. Therefore a new dunder method was defined for any additional processing: __post_init__. Let’s see what a __post_init__ method would look like for our wrapper class:

And that’s it! We could implement our class using the data class decorator in under a third of the number of lines as we could without the data class decorator.

More goodies

By using options with the decorator, you can tailor data classes further for your use case. The default options are:

  • init determines whether to generate the __init__ dunder method.
  • repr determines whether to generate the __repr__ dunder method.
  • eq does the same for the __eq__ dunder method, which determines the behavior for equality checks (your_class_instance == another_instance).
  • order actually creates four dunder methods, which determine the behavior for all lesser than and/or more than checks. If you set this to true, you can sort a list of your objects.

The last two options determine whether or not your object can be hashed. This is necessary (for example) if you want to use your class’ objects as dictionary keys. A hash function should remain constant for the life of the objects, otherwise the dictionary will not be able to find your objects anymore. The default implementation of a data class’ __hash__ function will return a hash over all objects in the data class. Therefore it’s only generated by default if you also make your objects read-only (by specifying frozen=True).

By setting frozen=True any write to your object will raise an error. If you think this is too draconian, but you still know it will never change, you could specify unsafe_hash=True instead. The authors of the data class decorator recommend you don’t though.

If you want to learn more about data classes, you can read the PEP or just get started and play with them yourself! Let us know in the comments what you’re using data classes for!

Posted in Tutorial | Tagged , | 18 Comments

PyCharm 2018.1.2 RC

We’re happy to announce that the release candidate of PyCharm 2018.1.2 is available for download on our Confluence page.

What’s New

Docker Compose Improvements

Our Docker Compose interpreter in PyCharm 2018.1.1 starts your application service together with its dependencies, but leaves your dependencies running after shutting down the application. This has now been changed to match the command-line behavior, and will shut down your dependencies as well. Have you not tried using Docker Compose interpreters yet? Learn how to do so on our blog with Django on Windows, or with Flask on Linux.

Docker Compose users on Windows will be happy to learn that we’re now using named pipes to connect to the Docker daemon, which resolves an issue where some users were unable to run their scripts.

Further Improvements

  • The Python Console now receives focus when its opened
  • Various improvements to database support: columns that show the result of a custom function in MSSQL are now correctly highlighted, and more. Did you know that PyCharm Professional Edition includes all database features from DataGrip, JetBrains’ SQL IDE?
  • Improvements in optimizing Python imports
  • Various issues regarding React lifecycles have been resolved
  • Read more in our release notes

Interested?

Download PyCharm now. The release candidate is not an EAP version. Therefore, if you’d like to try out the Professional Edition, you will either need to have an active license, or you’ll receive a 30-day trial period. The Community Edition is free and open source software and can be used without restrictions (apart from the Apache License’s terms).

If you have any comments on our RC version (or any other version of PyCharm), please reach out to us! We’re @pycharm on Twitter, and you can of course always create a ticket on YouTrack, our issue tracker.

Posted in Early Access Preview | Tagged | Leave a comment

Webinar: “Set Theory and Practice: Grok Pythonic Collection Types” with Luciano Ramalho

With PyCon US coming up, we wanted to squeeze in one more webinar, one themed towards PyCon. Luciano Ramalho, long-time Python speaker and teacher, PyCon luminary, and author of one of the best recent Python books, will join us to talk about Python’s data model.

  • Thursday, May 3
  • 5:00 PM – 6:00 PM CEST (11:00 AM – 12:00 PM EDT)
  • Register here
  • Aimed at intermediate Python developers

r2

Luciano’s Fluent Python book from O’Reilly gives deep treatment to this topic, and Luciano is focusing on one aspect: Python’s collection types. It’s a real pleasure for me personally to have Luciano on our webinar: we’ve been friends for many years and he’s one of the truly kind people that makes our community remarkable.

Speaking to You

Luciano Ramalho is a Principal Consultant at ThoughtWorks and the author of Fluent Python. He is the co-founder of the Brazilian Python Association and of Garoa Hacker Clube and a longtime web pioneer.

-PyCharm Team-
The Drive to Develop

Posted in Webinar | Tagged | Leave a comment

Webinar Recording: “Getting the Most Out of Django’s User Model” with Julia Looney

Yesterday we hosted Julia Looney for a webinar on Django user models. Julia has spoken on this topic at recent conferences and we were fortunate to have her with us. Julia’s slides, repositories, and the recording are now available.

During the webinar, Julia gave an overview of 3 options for custom user models:

  • Proxy Model
  • One-to-One Relationship
  • Custom User Model

-PyCharm Team-
The Drive to Develop

Posted in Video, Webinar | Tagged | Leave a comment