Big Data Tools: Experimental Spark Integration, Slack Community, and Feedback

Andrey Cheptsov

Yesterday we’ve released a fresh update of the Big Data Tools plugin in which we’ve added the integration with AWS S3. With this update, you’ll be able to browse and manage files in your S3 buckets right from the IDE. In yesterday’s announcement we mentioned that there is also another update to the Super-Early-Bird channel. What is this channel? A way the IntelliJ Platform offers to deliver unstable builds with experimental features.

All you have to do to try the experimental features of Big Data Tools, is to register a custom plugin repository URL in the IDE settings. This can be done by clicking the Gear icon in the Plugins settings, and choosing Manage Plugin Repositories…

In the dialog that will popup, add “https://plugins.jetbrains.com/plugins/Super-Early-Bird/12494” and click OK.

If you did everything right, in the list of the plugins, you’ll see an available update from the Super-Early-Bird channel:

If you see it, go ahead and click Update (or Install in case you didn’t install the stable update). After that, you’ll have access to the experimental features. Now, the main question! What experimental features are already available?

We’re excited to let you know that the first available experimental feature is Spark integration. With the current experimental build, you can now configure a Spark UI URL in the Big Data Tools Connections settings:

Once it’s configured, you can access the corresponding information about Spark applications, jobs, stages, and tasks from the Spark tool window (by default, it will appear in the bottom-right corner of the IDE):

With the Spark tool window, you can browse your applications, jobs, stages, tasks, and see the detailed information on any selection:

Note that this integration is experimental, which means two things: first, we’re still working on it, by making it more functional and more convenient, and second, we’re heavily relying on your feedback to steer its development in the direction where it matches your needs and workflows. Only when we are sure the functionality is ready enough, we’ll push it to the stable channel and make available for everyone else.

Slack community

How can you share your feedback? To help you share your feedback (and to be able to engage with the users of the plugin more actively), we’ve created a Slack workspace. In this workspace, we’re going to announce further experimental features, share our roadmap, collect your feedback, and of course help you with questions or troubleshooting of the plugin! In case you haven’t signed up for the workspace, please do it now. Even if you don’t plan to open it often, it still makes sense to sign up as this lets you get important notifications by email. Goes without saying that we aren’t going to abuse it.

Feedback and reward

Last but not least, we always appreciate your feedback in this online form. By filling this form you can quickly tell how satisfied the plugin quality and functionality are, and provide more details on what exactly we can improve!

Important! All active early adopters (those who actively try experimental features) and provide their feedback will get a little token of appreciation from JetBrains. Psst! Eager to try the plugin and share feedback but don’t have a license? Write me, and I’ll get something for you!

That’s all I got for today. Meanwhile, keep the drive, and spread the word!

Big Data Tools Team