We’ve just released a new build of the Big Data Tools plugin that is compatible with the 2021.3 versions of IntelliJ IDEA Ultimate, PyCharm Professional Edition, and DataGrip. The plugin also supports our new data science IDE, DataSpell. In this release, we’ve introduced a number of changes to the user interface, added several features, and fixed a long list of bugs. Let’s take a closer look.

The holiday season is almost upon us! And what better time than the holidays to catch up on the latest news and read about other interesting topics? Hi, I’m Pasha Finkelshteyn, and I’ll be your guide today through this month’s installment of the Data Engineering Annotated Monthly. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider

We’ve just released a new build of the Big Data Tools plugin that is compatible with the 2021.1, 2021.2, and 2021.3 versions of IntelliJ IDEA Ultimate, PyCharm Professional, and DataGrip. The plugin also supports our new data science IDE, DataSpell. In this release, we’ve introduced a number of changes to the user interface, added a bunch of features, and fixed a long list of bugs. Let’s take a

The lockdowns are back again in Moscow, which means that conferences are again out of the question for me for some time. The good news is that I had time to put together this new installment of our Data Engineering Annotated! Hi, I’m Pasha Finkelshteyn, and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new

In most countries, students start learning in September. As data engineers, let’s follow their lead and learn something new, too! I’m Pasha Finkelshteyn, and I’ll be your guide through this month’s news. I’ll offer my impressions of developments and highlight ideas from the wider community. If you think I missed something worthwhile, ping me on Twitter and suggest a topic, link, or anything else.

Recently we released a new build of the Big Data Tools plugin that is compatible with the 2021.3 EAP versions of IntelliJ IDEA Ultimate and PyCharm Professional. DataGrip 2021.3 EAP support will be available immediately after the release in October. The plugin also supports our new data science IDE – JetBrains DataSpell. If you still use previous versions, now is the perfect time to upgrade both y

August is usually a quiet month, with vacations taking their toll. But data engineering never stops. I’m Pasha Finkelshteyn and I will be your guide through this month’s news, my impressions of the developments, and ideas from the wider community. If you think I missed something worthwhile, ping me on Twitter and suggest a topic, link, or anything else. News A lot of engineering is about lea

August is a good time to start new things – some people are on vacation and have more spare time to read than usual, while others are back and looking for a quick refresher on what’s new in data engineering. We’re launching this Annotated series to find interesting and useful content on different topics around data engineering, such as news, technical articles, tools, future conferences, and more. And yes, at the time of writing “we” is just me, Pasha. If you think I missed something worthwhile, feel free to ping me on Twitter and suggest a topic, link, idea, or anything else.

The Kotlin API for Apache Spark is now widely available. This is the first stable release of the API that we consider to be feature-complete with respect to the user experience and compatibility with core Spark APIs. Get on Maven Central Let’s take a look at the new features this release brings to the API. Typed select and sortMore column functionsMore KeyValueGroupedDataset wrapper funct

In recent Big Data Tools releases, we've added a lot of new features, like Kafka support, a New Project wizard, additional Zeppelin settings, improved Python support, and more. We’ve received a lot of requests for us to polish these new additions, and we’ve taken this feedback to heart. Version 1.2 is a massive bug-fix release that greatly improves a wide array of features, both new and old. He

We released another update of the Big Data Tools plugin recently, and this article is an overview of its new features. The previous release was a momentous event for us as the plugin left the Early Access preview stage and became generally available. Since then, we've received a lot of feedback from you and have started developing features that the community wants. For example, Big Data Tools n

A new update of the Big Data Tools plugin has been released. This is our first version for general use, after a year and a half of the Early Access Preview program. Install the plugin from the JetBrains Plugin Repository or from inside your IDE to edit Zeppelin notebooks, upload files to cloud filesystems, and monitor Hadoop and Spark clusters. The following JetBrains IDEs support the plugin: Inte