News

The highlights of this release include integration with Hive Metastore and the ability to monitor Flink jobs right inside your IDE, as well as SSO authentication on Amazon S3. The new version provides many other noteworthy changes that are covered below. Get the latest version by installing it to the 2022.2 of your IDE.  Hive Metastore Integration In v.1.6, we’ve added the ability to create a Hive Metastore connection from the IDE and browse Hive catalogs, tables, and columns. Big Data Tools now also provides code completion for Spark SQL based on Hive Metastore data.

Everybody in IT works with data, including frontend and backend developers, analysts, QA engineers, product managers, and people in many other roles. The data used and the data processing methods vary with the role, but data itself is more often than not the key. — "It's a very special key, meant only for The One"— "What does it unlock?"— "The future"The Matrix Reloaded In the data engineering world, data is more than “just data” – it’s the lifeblood of our work. It’s all we work with, most of the time. Our code is data-centric, and we use the only real 5th-generation language there

It's the start of June. That means it’s time to start taking summer vacations and enjoying some fresh juice alongside your fresh news! Hi, I'm Pasha Finkelshteyn, and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. If you think I missed something worthwhile, catch me on Twitter and suggest a topic, link, or anything else you want to see. By the way, if you would prefer to receive this information as an email, you can subscribe to the newsletter here. News A lot o

Big Data Tools 1.6 EAP is now available. You can try the newly added features right away by installing the latest plugin version to the 2022.2 EAP of your IDE. Please note this is an Early Access Program build, meaning it’s not fully tested. Hive Metastore support Ability to create a Hive metastore connection from the EMR cluster window and browse Hive catalogs, tables, and columns. Apache Flink Monitoring You can now monitor Flink applications right in your IDE. Just like in the Flink Dashboard, you can launch and stop jobs, all without leaving your IDE. The d

Explore the new features for data engineers in the JetBrains Big Data Tools plugin, including Amazon EMR support and streamlined work with remote file systems.

We’ve just released a new build of the Big Data Tools plugin that is compatible with the 2021.3 versions of IntelliJ IDEA Ultimate, PyCharm Professional Edition, and DataGrip. The plugin also supports our new data science IDE, DataSpell. In this release, we’ve introduced a number of changes to the user interface, added several features, and fixed a long list of bugs. Let’s take a closer look. Kafka support Producer and consumer management in the Kafka connection tool windowThis allows you to quickly test apps by providing test data or checking the contents of existing topics.You’ll no

We’ve just released a new build of the Big Data Tools plugin that is compatible with the 2021.1, 2021.2, and 2021.3 versions of IntelliJ IDEA Ultimate, PyCharm Professional, and DataGrip. The plugin also supports our new data science IDE, DataSpell. In this release, we’ve introduced a number of changes to the user interface, added a bunch of features, and fixed a long list of bugs. Let’s take a closer look. (more…)

Recently we released a new build of the Big Data Tools plugin that is compatible with the 2021.3 EAP versions of IntelliJ IDEA Ultimate and PyCharm Professional. DataGrip 2021.3 EAP support will be available immediately after the release in October. The plugin also supports our new data science IDE – JetBrains DataSpell. If you still use previous versions, now is the perfect time to upgrade both your IDE and the plugin.  This year, we introduced a number of new features as well as some features that have been there for a while, for example, running Spark Submit with a run configuration.

The Kotlin API for Apache Spark is now widely available. This is the first stable release of the API that we consider to be feature-complete with respect to the user experience and compatibility with core Spark APIs. Get on Maven Central Let’s take a look at the new features this release brings to the API. Typed select and sortMore column functionsMore KeyValueGroupedDataset wrapper functionsSupport for Scala TupleN classesSupport for date and time typesSupport for maps encoded as tuplesConclusion Typed select and sort The Scala API has a typed select method that returns Dataset

In recent Big Data Tools releases, we've added a lot of new features, like Kafka support, a New Project wizard, additional Zeppelin settings, improved Python support, and more. We’ve received a lot of requests for us to polish these new additions, and we’ve taken this feedback to heart. Version 1.2 is a massive bug-fix release that greatly improves a wide array of features, both new and old. Here’s the full list of all the bugs we’ve fixed and improvements we’ve made: Improved support for symlinked files and folders (BDIDE-1879, BDIDE-1923)Fixed the problem causing strings in Scala code

We released another update of the Big Data Tools plugin recently, and this article is an overview of its new features. The previous release was a momentous event for us as the plugin left the Early Access preview stage and became generally available. Since then, we've received a lot of feedback from you and have started developing features that the community wants. For example, Big Data Tools now has support for Kafka, which can be easily connected in just a few clicks. Let's take a closer look at what has changed in the new release. Kafka monitoring Throughout the year, we conduct

A new update of the Big Data Tools plugin has been released. This is our first version for general use, after a year and a half of the Early Access Preview program. Install the plugin from the JetBrains Plugin Repository or from inside your IDE to edit Zeppelin notebooks, upload files to cloud filesystems, and monitor Hadoop and Spark clusters. The following JetBrains IDEs support the plugin: IntelliJ IDEA Ultimate, PyCharm Professional Edition, and DataGrip. In this release, we’ve added many useful features and addressed a variety of bugs. Let's dive into the details. Zeppelin 0.9 Support