big data tools

This is the third part of our ongoing series on Big Data, how we see it, and how we build products for it. In this installment, we’ll cover the first responsibility of the data engineer: building pipelines.

At the end of last year, we announced a preview of the IntelliJ IDEA Ultimate plugin that integrated Apache Zeppelin notebooks into the IDE. At the same time we shared our roadmap, in which we promised to support more tools for working with Big Data. Since then, the plugin team has been working hard and has extended the plugin with support for Apache Spark, Apache Hadoop's HDFS, AWS S3, Google Cloud Storage, and Parquet files. Because the plugin originally started with the Scala support in Zeppelin notebooks, it was reasonable for it to only be available for IntelliJ IDEA Ultimate. Now that th