Kotlin
A concise multiplatform language developed by JetBrains
Realm Kotlin Multiplatform SDK
Nabil Hachicha, Android Engineer, from MongoDB Realm shares their story of creating a pure Kotlin-first SDK for use by developers.
Realm is an open-source, easy-to-use local database that helps mobile developers to build better apps faster. It launched 8 years ago with the release of the Realm Java SDK, and since then we’ve worked to make sure our API is always improving to reflect what mobile developers need.
As Kotlin has grown to become the recommended language for building Android apps, we’ve made iterative improvements to our Realm Java SDK to make sure the API is compatible. But even with various releases over the past few years — via providing helper extension functions to improve using the Realm API with Kotlin — we knew Realm would ultimately benefit from the design and implementation of a pure Kotlin-first SDK for use by developers.
In the post, we take you through what motivated the Realm Engineering team’s decision to build a new SDK from the ground-up, and we share some of the key design principles that guided the development of Realm’s Kotlin Multiplatform SDK.
Background
As usage of Kotlin grew among Android developers, Realm’s engineering team started to use Kotlin in our testing environment. This led to introducing helper functions like Query with generic type; executeTransactionAwait – which runs transactions inside coroutines — and JSR305 annotation to improve the strict null-safety type system with Kotlin.
But even with this work, there were still architectural shifts that were difficult to solve, like users adopting unidirectional dataflows based on immutability. The wider Realm community also asked us to provide a more idiomatic Kotlin API that would enable a type safe query builder, among other API improvements. We knew some of these enhancements would require a fundamental refactoring and we wanted to meet our users’ needs. So Realm’s Engineering team started to design and implement a pure Kotlin first SDK for the mobile ecosystem.
Why a Multiplatform Library?
At the same time we began to lay the foundation of our Kotlin SDK, Kotlin Multiplatform started to mature, and we quickly realized we should build Realm’s new Kotlin SDK as a Multiplatform library. There were surprisingly few multiplatform libraries for the persistence layer, and as a multiplatform mobile database, we knew Realm could help fill this gap for developers. We had an opportunity to allow Kotlin developers to build cross platform applications in the same way that Realm’s developers use our React Native and Xamarin SDKs today.
Challenges
Porting the existing Realm-Java code to work on Multiplatform was an interesting (and challenging) task for Realm’s engineers because we currently use some technologies that are platform dependent.
To define a Realm schema, we use the annotation processor to generate a proxy class and then use the Gradle Transform API to do bytecode weaving with Javassist. Both tools are Kotlin JVM specific and are not compatible with Kotlin/Native.
Similarly, Reflection is only partially supported on Kotlin/Native, so any API design based on dynamic runtime instantiation would simply not work — like the Class.forName or class.newInstance primitives.
And, concurrency and the memory model are fundamentally different between platforms. The Kotlin/JVM has a Garbage Collector, whereas Kotlin/Native relies on reference counting with a cyclic collector.
To successfully build a multiplatform Kotlin Native SDK for Realm, we’d need to figure out how to re-architect the SDK in a way that would address these challenges.
The Solution
Note from the Kotlin team: Compiler plugins, compiler internal API, and Cleaner API are not stable yet and will likely change in the future.
Compiler Plugin to the Rescue
Realm’s engineering team quickly decided we would need to build our own fully-fledged compiler plug-in because existing options posed too many constraints.
The Kotlin Symbol Processing API is limited to generating new Kotlin code instead of modifying existing classes, meaning it wouldn’t suit our needs. Realm had used the Kotlin Annotation processor (KAPT) in our Kotlin support for the Realm Java SDK, but it didn’t have a native annotation processing system. Instead, the Realm Java SDK generates Java stubs containing all the declarations and annotations without the methods body. It then runs the Java annotation processor on those stubs, which feeds back to KAPT and Kotlin compiler for another round. This makes compilation and refactoring slow, and isn’t available for Kotlin/Native.
At a high level, the Realm compiler plugin‘s main goal is to make defining Realm models as simple as constructing a regular Kotlin class. This is achieved by modifying the Intermediate Representation (IR) of the compiled Kotlin code in order to change the behaviour of the property accessor (getters and setters). Once this is done a developer can now call Realm’s runtime APIs so that read/writing changes the value of the property instead of the backing field. This is done using a Lowering pass on the IR.
The other task of the compiler plugin is to collect all Realm model classes in order to automatically create a schema definition of a Realm. Since the Compiler plugin API was not yet stable and documented (1.4.30 makes JVM IR Beta stable), we reverse engineered the IR code we needed to write in order to obtain the desired generated code.
Here’s what that looks like:
First, we write the Kotlin code we want the compiler to generate in a classic Kotlin/JVM project. Then we add the compiler flag to use IR and to dump the content of the IR.
This will dump a text file containing the IR code generated by the compiler before transforming it into JVM ByteCode. We then use this IR dump to work out what IR API calls we need to use to produce the desired output. We also use kotlin-compile-testing library to unit test the IR output from the Realm compiler plugin that matches the expected IR.
Releasing Native Resources
Under the hood the Realm SDKs use a C++ storage engine. The compiler plugin adds infrastructure to link the Kotlin objects with the underlying C objects by adding a synthetic native pointer property. This holds the value of the C/ C++ pointer, allowing access to the underlying persisted value inside the storage engine.
In order to avoid leaking memory, the native pointer needs to be closed when the original Kotlin object goes out of scope.\
Since Kotlin/JVM and Kotlin/Native have different memory models, we decided to use two approaches. For Kotlin/JVM we rely on the same technique we use in the current Realm-Java SDK, which is based on the usage of the reference API to track pointers that need to be cleaned once their original Java counterpart is eligible for GC.
For more information about reference API and phantom reference check out this excellent talk by Bob Lee
Note from the Kotlin team: Cleaner API is not stable yet and will change in the future.
Kotlin/Native doesn’t have a Garbage collector. It uses reference counting with a cycle collector instead. Recently the Kotlin/Native team introduced the Cleaner API. This API can be used similarly to a Finalizer/Destructor block that runs after the encompassing object has been disposed of. We use this API to perform cleanups and free native pointers.
Encapsulating the native pointers in an object with the above deallocation strategies, which hook into the normal memory management, frees us from handling explicit deallocations where we use native pointers.
Native code abstraction
Accessing the custom C++ storage engine used by the Realm SDKs varies between Kotlin/JVM and Kotlin/Native. To simplify this, we added a C layer on top, which makes it easy to use tools like cinterop to interface and generate Kotlin stubs. The C layer also allowed us to avoid writing manual JNI code to access the C++ layer, thanks to SWIG tool that generates the corresponding Java and JNI stubs.
We also leveraged the expect/actual mechanism from Kotlin Multiplatform to build a module that hides the specifics of accessing Cinterop and SWIG layers in a platform agnostic manner. This allowed us to focus the design of the library API in a platform-independent way, and to do it in another module that will become the user facing public API.
This process encourages to think about features — like querying and writing to the database — in a way that’s not platform-specific.
Any platform dependent implementation details — like cleaning up native resources — do use different fundamentals per-platform (Garbage Collector for Kotlin/JVM and the Cleaner API for Kotlin/Native). This is the type of feature that should be kept inside the Native Module Abstraction layer and not be implemented in the public library module.
By isolating these low level constructs in a separate Multiplatform module, we managed to scale the development of the new Kotlin SDK within the Realm team. All the low level building blocks are isolated behind an internal platform agnostic API. This also opens up API contributions from the community without worrying about the inherent complexity of C/C++, Kotlin/Native, or JNI.
Conclusion
Now, we’re proud to say that Realm’s engineering team managed to bootstrap the new Realm Kotlin SDK as a Multiplatform library. We believe that the infrastructure is in place to encourage other authors to build Multiplatform libraries, too.
Of course, as an alpha release, we’re still polishing elements of the solution. Realm engineering has been working in lockstep with the team at JetBrains, and we have full confidence that improvements will land soon.\
As we keep working, we’d love your feedback. Try the new Realm Kotlin SDK out at https://github.com/realm/realm-kotlin, and join our discussion around the design of upcoming features here. We want your opinions and suggestions as we keep building.
Author: Nabil Hachicha, Android Engineer, MongoDB Realm
Subscribe to Kotlin Blog updates
