Big Data Tools 2023.1 Is Out!

Do you remember a long time ago when we introduced a feature in the Big Data Tools plugin called ZTools that allows you to view the current state of variables in a Zeppelin notebook? Do you remember that we made significant changes to ZTools implementation almost a year ago? It’s our tradition to make big announcements about this part of Big Data Tools yearly, and today we’re announcing multiple exciting changes. If you'd rather just check them out  yourself, here is the link to the plugin: (more…)

Roman Poborchiy, the Marketing Manager for the Machine Learning team, interviewed Pasha Finkelshteyn, a Big Data IDE Developer Advocate.

In this update we've added integrations with AWS Glue, Tencent Cloud Object Storage, enhanced Zeppelin notebooks support, and delivered important fixes. Read on to learn about the most important changes in the Big Data Tools plugin or try it right away by installing it to the 2022.3 of IntelliJ IDEA Ultimate, PyCharm Professional, DataSpell or DataGrip.  Tencent Cloud Object Storage (COS) Support We've supported a new distributed storage service. Now you can work with Tencent COS right from the IDE. Supported features include file versioning with the ability to compare two ve

Greetings from sunny Berlin! Yes, it’s still 20+ °C here – perfect conditions for sitting down on your balcony with the latest issue of your favorite Annotated! I'm Pasha Finkelshteyn, and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. If you think I missed something worthwhile, hit me up on Twitter and suggest a topic, link, or anything else you want to see. By the way, if you would prefer to receive this information as an email, you can subscribe to the newsletter h

It’s been a very bustling two months in Berlin. Indeed, it’s been so busy that I had to skip the digests. I am now delighted to have the privilege of returning to the task of collecting for you the most exciting news from the world of data engineering. Greetings from sunny Berlin! I'm Pasha Finkelshteyn, and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. If you think I missed something worthwhile, hit me up on Twitter and suggest a topic, link, or anything else you wa

The highlights of this release include integration with Hive Metastore and the ability to monitor Flink jobs right inside your IDE, as well as SSO authentication on Amazon S3. The new version provides many other noteworthy changes that are covered below. Get the latest version by installing it to the 2022.2 of your IDE.  Hive Metastore Integration We’ve added the ability to create a Hive Metastore connection from the IDE and browse Hive catalogs, tables, and columns. Big Data Tools now also provides code completion for Spark SQL based on Hive Metastore data. Apa

Hi, I'm Pasha Finkelshteyn, and I’ll be your guide today through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. If you think I missed something worthwhile, catch me on Twitter and suggest a topic, link, or anything else you want to see. By the way, if you would prefer to get this monthly source of data engineering information delivered straight to your inbox each month, you can subscribe to the newsletter here. News A lot of engineering is about learning new things and keeping a finger on

Everybody in IT works with data, including frontend and backend developers, analysts, QA engineers, product managers, and people in many other roles. The data used and the data processing methods vary with the role, but data itself is more often than not the key. — "It's a very special key, meant only for The One"— "What does it unlock?"— "The future"The Matrix Reloaded In the data engineering world, data is more than “just data” – it’s the lifeblood of our work. It’s all we work with, most of the time. Our code is data-centric, and we use the only real 5th-generation language there

It's the start of June. That means it’s time to start taking summer vacations and enjoying some fresh juice alongside your fresh news! Hi, I'm Pasha Finkelshteyn, and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. If you think I missed something worthwhile, catch me on Twitter and suggest a topic, link, or anything else you want to see. By the way, if you would prefer to receive this information as an email, you can subscribe to the newsletter here. News A lot o

Hello, fellow data engineers! It’s Pasha here, and today I'm going to introduce you to the new release of Kotlin API for Apache Spark. It's been a long time since the last major release announcements, mainly because we wanted to avoid bothering you with minor improvements. But today's announcement is huge! First, let me remind you what the Kotlin API for Apache Spark is and why it was created. Apache Spark is a framework for distributed computations. It is usually used by data engineers for solving different tasks, for example for the ETL process. It supports multiple languages straight out

Big Data Tools 2022.2 EAP is now available. You can try the newly added features right away by installing the latest plugin version to the 2022.2 EAP of your IDE. Please note this is an Early Access Program build, meaning it’s not fully tested. Hive Metastore support Ability to create a Hive metastore connection from the EMR cluster window and browse Hive catalogs, tables, and columns. Apache Flink Monitoring You can now monitor Flink applications right in your IDE. Just like in the Flink Dashboard, you can launch and stop jobs, all without leaving your IDE. Th

Long time no see! Sorry about the silence, but luckily we’re back. Hi, I'm Pasha Finkelshteyn, and I’ll be your guide through this month’s news. I’ll offer my impressions of recent developments in the data engineering space and highlight new ideas from the wider community. If you think I missed something worthwhile, catch me on Twitter and suggest a topic, link, or anything else you want to see. And please feel free to subscribe to this newsletter to get it in your email inbox every month. News A lot of engineering is about learning new things and keeping a finger on the pulse of new