spark

Hi everyone, Jolan here, with my first actual blog post! It's been a couple of months since the last release of the Kotlin API for Apache Spark and we feel like all of the exciting changes since then are worth another announcement. Let me remind you what the Kotlin API for Apache Spark is and why it was created. Apache Spark is a framework for distributed computations, which data engineers usually use to solve different tasks, like in the ETL process. It supports multiple languages straight out of the box, including Java, Scala, Python, and R. We at JetBrains are committed to supporting one

Hello, fellow data engineers! It’s Pasha here, and today I'm going to introduce you to the new release of Kotlin API for Apache Spark. It's been a long time since the last major release announcements, mainly because we wanted to avoid bothering you with minor improvements. But today's announcement is huge! First, let me remind you what the Kotlin API for Apache Spark is and why it was created. Apache Spark is a framework for distributed computations. It is usually used by data engineers for solving different tasks, for example for the ETL process. It supports multiple languages straight out

The Kotlin API for Apache Spark is now widely available. This is the first stable release of the API that we consider to be feature-complete with respect to the user experience and compatibility with core Spark APIs. Get on Maven Central Let’s take a look at the new features this release brings to the API. Typed select and sortMore column functionsMore KeyValueGroupedDataset wrapper functionsSupport for Scala TupleN classesSupport for date and time typesSupport for maps encoded as tuplesConclusion Typed select and sort The Scala API has a typed select method that returns Dataset

We have released Preview 2 of Kotlin for Apache Spark. First of all, we’d like to thank the community for providing us with all their feedback and even some pull requests! Now let’s take a look at what we have implemented since Preview 1? Scala 2.11 and Spark 2.4.1+ support The main change in this preview is the introduction of Spark 2.4 and Scala 2.11 support. This means that you can now run jobs written in Kotlin in your production environment. The syntax remains the same as the Apache Spark 3.0 compatible version, but the installation procedure differs a bit. You can now use sbt to