Kotlin Kernel for Jupyter Notebook, v0.9.0

Roman Belov

This update of the Kotlin kernel for Jupyter Notebook primarily targets library authors and enables them to easily integrate Kotlin libraries with Jupyter notebooks. It also includes an upgrade of the Kotlin compiler to version 1.5.0, as well as bug fixes and performance improvements.

pip installConda install

The old way to add library integrations

As you may know, it was already possible to integrate a library by creating a JSON file, which we call a library descriptor. In the kernel repository, we have a number of predefined descriptors. You can find the full list of them here.

Creating library descriptors is rather easy. Just create a JSON file and provide a description section with a library description and a link section with a link to the library’s web page. Then add the repositories and dependencies sections, describing which repositories to use for dependency resolution and which artifacts the library includes. You can also add an imports section, where you list imports that will be automatically added to the notebook when the descriptor is loaded, such as init and initCell code snippets, renderers, and so on. When you are finished, save the created file and refer to it from the kernel in whatever way is most convenient for you. In this release, we’ve added some more ways to load descriptors. You can read more about how to create library descriptors here.

This method for integrating libraries is still supported and works particularly well when you are not the author of the library you want to integrate. But it does have some limitations:

1. It requires additional version synchronization. The integration breaks if a new version of the library is released and a class that was used in the integration is renamed.
2. It’s not that easy to write Kotlin code in JSON without any IDE support. So if your library provides renderers or initialization code, then you have to go through a potentially long process of trial and error.
3. Transitive dependencies are not supported. If libraries A and B provide descriptors and library A depends on library B, then adding just the descriptor for library A is not enough, and you also need to run %use B.
4. Advanced integration techniques are not allowed. See below for more info.

The new way to add library integrations

One of the best things about Kotlin notebooks (compared to Python notebooks) is that you do not have to think about dependencies. You just load the library you need with the @DependsOn annotation and use it. All transitive dependencies are loaded automatically, and you do not have to worry about environments or dependency version clashes. An added bonus is that it will work the same way on all computers. So far, however, there hasn’t been a way to define the descriptor mentioned above and to attach it to a library so you don’t have to create and load it separately.

Now there is such a way. You can now define the descriptor inside your library code and use a Gradle plugin to automatically find and load the ID. This means you do not have to write a separate JSON and %use directive.

If you are the maintainer of a library and can change its code, you may like the new method of integration. It currently utilizes Gradle as a build system, but if you use something else, feel free to open an issue and we will work on adding support for it.

Suppose you have the following Gradle build script written in the Kotlin DSL:

plugins {
    kotlin("jvm")
}

group = "org.example"
version = "1.0"

// ...

The published artifact of your library should then have these coordinates: org.example:library:1.0

You normally add your library to the notebook using the DependsOn file annotation:

@file:DependsOn("org.example:library:1.0")

Now suppose you need to add a default import and a renderer for this library in notebooks. First, you apply the Gradle plugin to your build:

plugins {
    kotlin("jvm")
    kotlin("jupyter.api") version "<jupyterApiVersion>"
}

Then, you write an integration class and mark it with the JupyterLibrary annotation:

package org.example

import org.jetbrains.kotlinx.jupyter.api.annotations.JupyterLibrary
import org.jetbrains.kotlinx.jupyter.api.*
import org.jetbrains.kotlinx.jupyter.api.libraries.*

@JupyterLibrary
internal class Integration : JupyterIntegration() {
    override fun Builder.onLoaded() {
        import("org.example.*")
        render<MyClass> { HTML(it.toHTML()) }
    }
}

It is supposed that MyClass is a class from your library and has the toHTML() method, which returns an HTML snippet represented as a string.

After re-publishing your library, restart the kernel and import the library via DependsOn again. Now you can use all the packages from org.example without specifying additional qualifiers, and you can see rendered HTML in cells that return MyClass instances!

Advanced integration features

Let’s take a look at some advanced techniques you can use to improve the integration. We’ll use the following set of classes for reference:

data class Person(
    val name: String,
    val lastName: String,
    val age: Int,
    val cars: MutableList<Car> = mutableListOf(),
)

data class Car(
    val model: String,
    val inceptionYear: Int,
    val owner: Person,
)

annotation class MarkerAnnotation

Subtype-aware renderers

In the descriptors using the old style, you can define renderers that transform cell results of a specific type. The main problem of this approach is that type matching is done by fully qualified type names. So if you define a renderer for a type A that has a subtype B, the renderer will not be triggered for instances of type B.

The new API offers you two solutions to this problem. First, you can implement the org.jetbrains.kotlinx.jupyter.api.Renderable interface:

import org.jetbrains.kotlinx.jupyter.api.*

class MyClass: Renderable {
    fun toHTML(): String {
        return "<p>Instance of MyClass</p>"
    }

    override fun render(notebook: Notebook) = HTML(toHTML())
}

It yields the following result:

Another way to do the same thing has actually already been presented above:

render<MyClass> { HTML(it.toHTML()) }

This option is preferable if you want to keep integration logic away from the main code.

Variable converters

Variable converters allow you to add callbacks for variables of a specific type:

addTypeConverter(
    FieldHandlerByClass(Person::class) { host, person, kProperty ->
        person as Person
        if (person.name != kProperty.name) {
            host.execute("val <code>${person.name}</code> = ${kProperty.name}")
        }
    }
)

This converter creates a new variable with the name person.name for each Person variable defined in the cell. Here’s how it works:

Annotations callbacks

You can also add callbacks for file annotations (such as the aforementioned DependsOn) and for classes marked with specific annotations.

onClassAnnotation<MarkerAnnotation> { classes ->
    classes.forEach {
        println("Class ${it.simpleName} was marked!")
    }
}

Here we are simply logging the definition of each class marked with MarkerAnnotation:

Cell callbacks

Descriptors allow you to add callbacks that are executed when the library loads (init) and before each cell is executed (initCell). The new integration method also allows you to add these callbacks with ease and provides support for callbacks triggered after cell execution. Let’s see how it works.

beforeCellExecution {
    println("Before cell callback")
}

afterCellExecution { _, result ->
    println("Cell [${notebook.currentCell?.id}] was evaluated, result is $result")
}

Here you see a usage of the notebook variable, which provides some information about the current notebook.

Dependencies, renderers, and more

There are some other methods you can use to improve Jupyter integration, such as render, import, dependencies, repositories, and more. See the JupyterIntegration code for a full list.

Note that you can mark with @JupyterLibrary any class that implements LibraryDefinition or LibraryDefinitionProducer. There’s no need to extend JupyterIntegration.

All of the code from this section is provided here. You can also find a more complex example of integration in the Dataframe library.

Maven artifacts for your use case

We now publish a set of artifacts to Maven Central, and you are welcome to use them in your own libraries.

The kotlin-jupyter-api and kotlin-jupyter-api-annotations artifacts are used in the code-based integration scenario that was described above. You will not usually need to add them manually – the Gradle plugin does it for you. These artifacts may help in some situations, for example, if you don’t use Gradle or just want to use some classes from the API without integrating it.

If you just want to use the Kotlin REPL configuration and the compiling-related features that are used in the kernel, you may be interested in the kotlin-jupyter-shared-compiler artifact. This artifact was designed to be consistent with the IntelliJ Platform, so you can use it to make IntelliJ plugins.

kotlin-jupyter-lib-ext is a general-purpose library that includes functions for images and HTML rendering. You can load it from the notebook with %use lib-ext. It is not included in the kernel distribution because it may require additional dependencies in the future, and it is not a good idea to bundle them by default.

And finally, you can depend on the kotlin-jupyter-kernel artifact if you need the whole kernel bundled into your application. You can use the embedKernel method to start the kernel server.

Other artifacts have no clear use case and are just transitive dependencies of other ones.

If your use case is not covered, please open an issue or contact us in the #datascience channel of Kotlin Slack.

Kotlin 1.5.0 and bug fixes

The underlying Kotlin compiler version was updated to 1.5.0 pre-release. It doesn’t use the new JVM IR backend at the moment, but we’ll make that happen soon. The main thing is that we’ve fixed a bug in the REPL compiler that was affecting the updating of scripts’ implicit receivers, so performance should now be better for notebooks with a large number of executed cells.

A number of additional bugs have also been fixed, including these particularly weird ones:

• Irrelevant error pop-ups in the Notebook client (#109)
• Incorrect parsing of %use magic (#110)
• Resolution of transitive dependencies with runtime scope didn’t work
• Leaking of kernel stdlib into script classpath (#27)

Check out the release changelog for further details.

Let’s Kotlin!

Subscribe to Kotlin Blog updates

Subscribe form

By submitting this form, I agree to the JetBrains Privacy Policy Notification icon

By submitting this form, I agree that JetBrains s.r.o. ("JetBrains") may use my name, email address, and location data to send me newsletters, including commercial communications, and to process my personal data for this purpose. I agree that JetBrains may process said data using third-party services for this purpose in accordance with the JetBrains Privacy Policy. I understand that I can revoke this consent at any time in my profile. In addition, an unsubscribe link is included in each email.

Submit

Thanks, we've got you!