Hello, fellow data engineers! It\u2019s Pasha here, and today I’m going to introduce you to the new release of Kotlin API for Apache Spark. It’s been a long time since the last major release announcements, mainly because we wanted to avoid bothering you with minor improvements. But today’s announcement is huge!<\/p>\n\n\n\n
First, let me remind you what the Kotlin API for Apache Spark is and why it was created. Apache Spark is a framework for distributed computations. It is usually used by data engineers for solving different tasks, for example for the ETL process. It supports multiple languages straight out of the box: Java, Scala, Python, and R. We at JetBrains are committed to supporting one more language for Apache Spark \u2013 Kotlin, as we believe it can combine multiple pros from other language APIs while avoiding their cons.<\/p>\n\n\n\n
If you don’t want to read and just want to try it out \u2013 here is the link to the repo:<\/p>\n\n\n\n
Repository<\/a><\/p>\n\n\n\n Otherwise, let’s begin our overview.<\/p>\n\n\n\n For a long time, we’d supported only one API from Apache Spark: Spark Streaming<\/em> is a solution to build streaming processing systems using Spark. Contrary to other stream processing solutions, Spark Streaming<\/em> works with micro-batches. When reading data from a source, instead of working on one element at a time, it reads all data in defined time frames or “batches” (for example, it might read everything available each 100ms).<\/p>\n\n\n\n There are multiple core entities in Spark Streaming<\/em>:<\/p>\n\n\n\n As you might already know, we have a special The very basic sample usage will look like this:<\/p>\n\n\n\n What can we see here? We create a Spark Streaming<\/em> context with the call As you can see, no other non-obvious abstractions are involved. We can work with the The You can find more examples on our GitHub<\/a> and more detailed documentation is available on our wiki<\/a>.<\/p>\n\n\n\n A Kotlin kernel for Jupyter<\/a> has existed for some time already. You can perform experiments with different Kotlin for Data Science tools, such as multik<\/a> (library for multi-dimensional arrays in Kotlin) or KotlinDL<\/a> (a Deep Learning API written in Kotlin and inspired by Keras, working on top of TensorFlow), and others described in the Kotlin documentation<\/a>. But we are aware that Jupyter is quite popular among data engineers, too, so we’ve added support for the Kotlin API for Apache Spark as well. In fact, all you need to do to start using it in your notebook is put The main features you will find in this support are autocompletion and table rendering. When you use The aforementioned Spark Streaming<\/em> is supported too. To use it, all you need to do is add Please be aware that Jupyter support for Kotlin API for Apache Spark is experimental and may have some limitations. One such limitation we are aware of: You cannot mix batching and streaming in the same notebook \u2013 You can find an example of a notebook with streaming here<\/a>.<\/p>\n\n\n\n Of course, it works in Datalore<\/a> too. Here is an example notebook<\/a>, as well as an example notebook for streaming<\/a>. In case you\u2019re not aware of what Datalore is: It\u2019s an online environment for Jupyter notebooks, developed by JetBrains.<\/p>\n\n\n\n A bit more information on Jupyter integration can be found on our wiki<\/a>.<\/p>\n\n\n\n While preparing this release, we noticed that our own reinvented tuples (called In Kotlin-esque fashion, here are some useful extension methods for tuples:<\/p>\n\n\n\n There is much more information about the new Tuples API available in the wiki<\/a>.<\/p>\n\n\n\n We have also changed the structure of the documentation.<\/p>\n\n\n\n The readme grew too large to easily digest, so we’ve split it up and put its contents into the wiki on GitHub.<\/p>\n\n\n\n These are the most important changes to the Kotlin API for Apache Spark version 1.1. As usual, you can find the release on Maven Central<\/a>. If you have any ideas or if you need help or support, please contact us on Slack<\/a> or GitHub issues.<\/p>\n\n\n\n Also, please give a warm welcome to Jolan Rensen<\/strong>, who contributed to the project for some time and now works at JetBrains. He’s the main person in charge of maintaining the project, while yours truly only tries to help him out wherever he can. Jolan, we\u2019re thrilled to have you with us!<\/p>\n","protected":false},"author":1234,"featured_media":249850,"comment_status":"closed","ping_status":"closed","template":"","categories":[594],"tags":[21,91,590,6607],"cross-post-tag":[6355],"acf":[],"_links":{"self":[{"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/big-data-tools\/249904"}],"collection":[{"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/big-data-tools"}],"about":[{"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/types\/big-data-tools"}],"author":[{"embeddable":true,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/users\/1234"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/comments?post=249904"}],"version-history":[{"count":8,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/big-data-tools\/249904\/revisions"}],"predecessor-version":[{"id":250326,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/big-data-tools\/249904\/revisions\/250326"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/media\/249850"}],"wp:attachment":[{"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/media?parent=249904"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/categories?post=249904"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/tags?post=249904"},{"taxonomy":"cross-post-tag","embeddable":true,"href":"https:\/\/blog.jetbrains.com\/zh-hans\/wp-json\/wp\/v2\/cross-post-tag?post=249904"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Spark Streaming<\/h1>\n\n\n\n
Dataset<\/code> API. While it’s widely popular, we can’t just ignore the fact that there is at least one more trendy extension to Apache Spark: Spark Streaming<\/em>. Its name is self-explanatory, but just to be sure that we’re on the same page, allow me to elaborate a little.<\/p>\n\n\n\nDStream<\/code>) represents a continuous stream of data. It can be created from an input source (socket, Kafka, or even a text file) or from another DStream<\/code>. DStream<\/code> is represented as a sequence of RDD<\/code>s (Resilient Distributed Dataset)<\/a>.<\/li>StreamingContext<\/code><\/a>) is the main entry point for working with DStreams. Its main goal is to provide us with different methods for creating streams from different sources.<\/li><\/ol>\n\n\n\nwithSpark<\/code> function in our core API (you can view it here<\/a>, for example). Of course, for Streaming, we have something similar: withSparkStreaming<\/code>. It has some defaults that we think are reasonable. You can take a look at them here<\/a> if you want.<\/p>\n\n\n\n\n\n\n\n\nwithSparkStreaming { \/\/ this: KSparkStreamingSession\n val lines: JavaReceiverInputDStream<string> = TODO() \/\/ create some string stream, for example, from socket\n val words: JavaDStream<string> = TODO() \/\/ some transformation\n words.foreachRDD { rdd: JavaRDD<string>, _: Time ->\n withSpark(rdd) { \/\/ this: KSparkSession\n val dataframe: Dataset<testrow> = rdd.map { TestRow(it) }.toDS()\n dataframe\n .groupByKey { it.word }\n .count()\n .show()\n }\n }\n}\n<\/pre><\/testrow><\/string><\/string><\/string>\n\n\n\nwithSparkStreaming<\/code>. Many useful things are available inside it, for example, a withSpark<\/code> function that will obtain or create a Spark Session<\/em>. It also provides access to the ssc<\/code> variable (which stands for \u2013 you guessed it \u2013 “spark streaming context”).<\/p>\n\n\n\nRDD<\/code>s, JavaDStream<\/code>s, etc. that we are familiar with.<\/p>\n\n\n\nwithSpark<\/code> function inside withSparkStreaming<\/code> is slightly different from the one you’re familiar with. It can find the right Spark Session from the SparkConf<\/code> of the ssc<\/code> variable or (as seen in the example) from an RDD<\/code>. However, you still get a KSparkSession<\/code> context which can give you the ability to create Dataset<\/code>s or broadcast variables, etc. But in contrast to its batching counterpart, the Spark Session won’t be closed at the end of the withSpark<\/code> block. Lastly, it behaves similarly to the Kotlin run function because it returns the last line of its contents.<\/p>\n\n\n\nJupyter support<\/h1>\n\n\n\n
%use spark<\/code> in your notebook’s cell. You can see an example on our GitHub<\/a>.<\/p>\n\n\n\n%use spark<\/code>, all of the notebook cells are automagically wrapped into one implicit withSpark block, which gives you access to all the sugar we provide.<\/p>\n\n\n\n%use spark-streaming<\/code>. Of course, all the features of dynamic execution are supported \u2013 for example, tables will update automatically.<\/p>\n\n\n\nwithSparkStreaming<\/code> doesn’t work inside of withSpark<\/code> block. Don’t hesitate to provide us with examples if something doesn’t seem to behave as it should for you. We’re always happy to help!<\/p>\n\n\n\nDeprecating c in favor of t<\/h1>\n\n\n\n
ArityN<\/code>) aren\u2019t actually that effective when used extensively, and it’s more effective to reuse Scala Tuple<\/code>s. So, we’ve deprecated our factory c<\/code> method in favor of the factory t<\/code> method. The semantics are totally the same, but the method creates a native Scala Tuple instead of Kotlin one.<\/p>\n\n\n\n\nval a: Tuple2<Int, Long> = tupleOf(1, 2L) \/\/ explicit tupleOf, same convention as with `listOf`\nval b: Tuple3<String, Double, Int> = t("test", 1.0, 2) \/\/ `t` as an alternative to `c`\nval c: Tuple3<Float, String, Int> = 5f X "aaa" X 1 \/\/ infix function which creates tuple of 3 elements\ntupleOf(1) + 2 == tupleOf(1, 2) \/\/ '+'-syntax can be used to extend tuples\n<\/pre>\n\n\n\nConclusion<\/h1>\n\n\n\n