Application development in Kotlin/Native

In this blog post we are discussing development of Kotlin/Native applications. Today we take a look on basic video player, using FFMPEG audio/video decoder and SDL2 for rendering. Hopefully, it will be useful guide for Kotlin/Native development enthusiasts and will explain intended mechanisms of using the platform.

As main focus in our tutorial is on Kotlin/Native, we will give only cursory view on how videoplayer shall be developed. Please see this excellent tutorial, called “How to Write a Video Player in Less Than 1000 Lines” for reference on how it could be done in C language. If you’re interested in comparing how coding in C differs to coding in Kotlin/Native, I would recommend starting with this tutorial.

Every video player in theory does rather simple job: reads input stream with interleaved video and audio frames, decodes frames and shows video frames, synchronising them with the audio stream. Typically, this job is being done by multiple threads, performing stream decoding, video and audio playback. Doing it right requires thread synchronisation and certain real-time guarantees, as if audio stream is not being decoded in time, playback sounds choppy, and if video frame is not available when it’s needed, movie doesn’t look smooth.

Kotlin/Native doesn’t encourage you to use threads, and doesn’t provide a way to share Kotlin objects between threads. However, we believe that concurrent, soft-realtime programming in Kotlin/Native shall be easy, so we decided to design our player in concurrent manner from the very beginning. Let’s see how we achieved that.

Kotlin/Native computational concurrency is built around  workers. Worker is higher level concurrency concept than thread, and instead of object sharing and synchronisation it allows object transfer, so that every moment only single workers have access to particular object. It means, no synchronisation shall be required to access object data, as access could never be concurrent. Workers can receive execution requests, which may accept objects and perform job as needed, and then transfer result back to whoever need computation’s result. Such model ensures that many typical concurrent programming mistakes (such as unsynchronised access to shared data, or deadlocks because of unordered taking of locks)  just cannot be made.

Let’s see, how it translates to the video player architecture. We need to perform decoding of some container format, like .avi, .mkv or .mpg, which perform demultiplexing of interleaved audio and video streams, decoding and then feed decompressed audio to SDL audio thread. Decompressed video frames shall be rendered in sync with sound playback. To achieve that goal, worker’s concept seems to be pretty natural. We spawn a worker for the decoder, and ask it for video and audio data whenever we need it. On multicore machines it means that decoding can happen in parallel with the playback. So decoder worker is a data producer from both UI thread and audio thread.

Whenever we need to fetch next audio or video data chunk, we rely upon almighty schedule() function. It schedules chunk of work to be executed by a particular worker, provides input argument and return Future instance, that could be waited on, until job is executed by the target worker. Future object could be consumed, so that produced object is taken from worker thread back to requester thread.

Kotlin/Native runtime is conceptually thread-bound, so when running multiple threads calling function konan.initRuntimeIfNeeded() is required before other operations, so we do that in audio thread callback. To simplify audio playback we always resample audio frames to 2 channel signed 16-bit integer stream with 44100 samples per second.

Video frames are being decoded to whatever size is requested by the user, with defaults of encoded video size, and bit depth depends on user’s desktop defaults.  Please note Kotlin/Native specific operations for manipulating C pointers, i.e.

We declare resampledAudioFrame being a disposable resource in the C world created with FFMPEG API call av_frame_alloc() and disposed with av_frame_unref(). Then we set fields of whatever it points to to the desired values. Note that we can use defines declared by FFMPEG (such as AV_PIX_FMT_RGB24) as Kotlin constants. But as they do not have type information and are Int by default, if certain field has different type (i.e. channel_layout) adapter function signExtend() must be called. It is compiler’s intrinsic, which inserts appropriate conversions.

After decoder is set up, we start the playback loop. It does nothing fancier but retrieves the next frame, renders it to the texture and shows this texture on the screen. As a result – video frame is rendered. Audio is being actively handled by audio thread callback which fetches next samples buffer from the decoder and feeds it back to audio engine.

Audio/video synchronisation is pretty basic, it just ensures that we don’t have not too many unplayed audio frames.  Real multimedia player probably shall rely on frame timestamps instead, which we compute, but never use.  Here interesting place is using

It manifests how to use APIs accepting C language struct’s. It is declared in libavutil/rational.h as

Thus, to pass it by value, we first need to use readValue() on the field.

So, as a wrap up, we have implemented a simple audio/video player supporting multiple input formats, thanks to FFMPEG library, with relatively minor efforts. We also discussed some basics of  C language interoperability in Kotlin/Native, along with concurrency approaches we consider easier to use and maintain.

About Nikolay Igotti

Kotlin/Native Tech Lead
This entry was posted in Coding, Native and tagged . Bookmark the permalink.

23 Responses to Application development in Kotlin/Native

  1. Zach Klippenstein says:

    JetBrains is already investing heavily in coroutines, a powerful and cross-platform concurrency API. How does that roadmap intersect with that of this new workers API? Are they going to play nice at some point? The extra checks the workers API does could be useful on other platforms too. Or if not, why build a completely separate tool that only works on kotlin native?

    • Nikolay Igotti says:

      Zach, thanks for the great question!

      If you will take a look onto what coroutines are, they are essentially a mechanism for saving from the callback hell with the voluntary preemption/resume mechanisms.
      As such, coroutines are fully available in Kotlin/Native as in any other Kotlin dialects.

      Workers serve for the quite orthogonal purpose: provide mechanism of exploiting actual hardware concurrency by performing work in parallel on multiple cores/CPUs.
      Unfortunately, threads taught people rather bad lessons of mixing those two orthogonal purposes by using single constructions, what leads to rather dramatic code complexity and software maintenance cost increases. So what we try to achieve here, is give people ability to use good coding practises with the workers mechanisms.

      • Max says:

        That’s…still not entirely clear. Coroutines can and are used for parallelism — they’re built on top of a threadpool, and can be used for exploiting actual hardware concurrency across multiple cores. Perhaps workers are to get around the fact that the Kotlin/Native doesn’t have a concept of threadpool?

        • Nikolay Igotti says:

          Not exactly. Coroutines can be used in APIs providing parallelism, but themselves they are not concurrent programming concepts, just a way to represent flow control in more contigous way. Kotlin/Native can be used in threaded environment (i.e. pthread_create() and friends could be called via platform libraries), however, we believe that classical threading with shared object heap is an inferior programming technique, and so we’re trying to provide an alternative.

          • Ildar Sharafutdinov says:

            I guess the question is rather simple.

            While having the high-level abstraction that can exploit hardware concurrency, what is the benefit of using lower level workers instead?

            • Nikolay Igotti says:

              What is “high-level abstraction that can exploit hardware concurrency” you’re talking about?

              • Ildar Sharafutdinov says:

                coroutines

              • Nikolay Igotti says:

                As I mentioned in another thread, coroutines are orthogonal to concurrent programming, they are glorified callbacks, and as such, cannot be considered suitable primitive of concurrent programming.

      • Zach Klippenstein says:

        Nikolay, you mention callback hell. This callback-based Workers API is exactly what you’re talking about – as soon as you’ve got a couple worker jobs that depend on each other, code would quickly get hard to read. The same effect (offloading work onto a different hardware thread) can be achieved using coroutine dispatchers. It would likely be a lot easier to read as the logic got more complex, and composes much more nicely with other higher-level constructs (cancellation, select, etc).

        I like where Max is going – since Kotlin Native lacks the JVM’s Executor, it feels like Workers solves that much lower-level problem. As such, they could be used to implement coroutine schedulers. Is that an accurate assessment?

        • Nikolay Igotti says:

          Sure, nothing prevents building coroutines based APIs on top of workers (such as dispatchers), they just solve somewhat different purposes from the POV of system software developer.

  2. Interesting post!
    As far as I understand, workers are not available in Kotlin/JVM. Isn’t that a major cause of incompatibility between Native and JVM implementations?

    • Nikolay Igotti says:

      Practically speaking, Kotlin/JVM, Kotlin/JS and Kotlin/Native are dialects of same language, and as such may have some differences in implementation details and available functionality. However, workers could be implemented (maybe modulo reliable reachibility checks) on top of JVM, so it doesn’t look like something truly fragmenting the platform.

  3. Clemens says:

    Is there a mechanism for message passing between threads? What about shared memory? (sometimes object passing is not enough)

    In how far would you compare this approach to Rust’s ownership model / to browser worker threads?

    Is there a link to the code?

  4. Nikolay Igotti says:

    Yes, objects could be passed between workers and could be detached and attached to different threads, see for example https://github.com/JetBrains/kotlin-native/blob/c455064ca2c34aa8d6e49994e7e4521e909b68c7/samples/globalState/src/main/kotlin/Global.kt#L64
    where we create an object, detach it from the main thread and attach to another thread.

    Same example shows how one could achieve shared memory amongs two threads (and visible from C as well).

    Browser worker threads are less powerful, as here we could transfer abitrary objects back and forth. Rust ownership model is more powerful, however, it is also (IMHO) requires more efforts from the programmer.

    • Jakub Błaszczyk says:

      I think the topic “workers vs coroutines” deserves a separate blog post and entry in manual, since these concepts seem to be often mistakenly conflated. I’d also love to read more on workers and how they differ from Rust and JS, what are best practices and abuses.

      BTW, when designing workers, do you rely on academic research? are there any papers on that?

  5. vincentsun says:

    I am the editor of InfoQ China which focuses on software development. We like your articles and plan to translate one of them . hope to get your permission.Thanks a lot, hope to get your help. Any more question, please let me know.

  6. The Bam says:

    Its looks very nice, what I dont understand though is why suddenly are you trying to make kotlin so opinionated?
    In the JVM, kotlin is so great because it serves as a “better java” it gives you the same power as java with an easier, safer and more concise interface. In native, why kotlin is not designed to be just a “better c++”? why does it try to invent its own memory management system and threading paradigms?

    • Nikolay Igotti says:

      It’s not about opinions. Unlike JVM/JS there’s no standard native object model, every language has to invent its own object and concurrency model. Kotlin could be better Java, but could unlikely be better C++, as they have very different set of characteristics and design goals. Kotlin is an application language, influenced by Java and FP languages, easy and pleasant for the programmer, so we’re trying to keep that attitude in Kotlin/Native.

  7. Tarun Elankath says:

    Will Kotlin native provide a I/O abstraction layer ? If so, will it look like java io/nio or something different ?

Comments are closed.