Interview with Jason Zaugg on Scala, Dotty and IntelliJ IDEA

Andrey Cheptsov

As some of you may know, this June the whole Scala plugin team took a break and went to Berlin to enjoy Scala Days 2014, and of course to meet IntelliJ IDEA users! Alexander Podkhalyuzin, a lead developer of the Scala plugin, used this chance to catch Jason Zaugg, a Scala rockstar and developer at Typesafe, and talk about Scala Compiler, Dotty and IntelliJ IDEA. Enjoy the interview!

Alexander: How did you first hear about Scala?

Jason: A colleague of mine saw Martin Odersky present Scala at a conference in 2008. He shared his impressions of the language at our team’s weekly tech talk, and my interest was piqued. I had already been drawn to functional programming from the perspective of a Java programmer who disliked code duplication. I’d used Ruby and Groovy on small projects, but was never happy with the compromises in type safety and performance. Scala seemed to offer the best of both worlds.

Alexander: You’ve helped the IntelliJ IDEA team a lot to improve the Scala plugin. What are your impressions about developing for IntelliJ Platform?

Jason: My contributions to the plugin were largely “scratching an itch”. Back in 2009, it was hard to sell Scala to a Java programmer who appreciated a powerful IDE. I knew a tiny bit about the IDE plugin architecture as I had built a plugin a few years earlier, and wondered how hard it would be to dive in and fix a few bugs. I learnt a lot about Scala in the process, and started to get a taste for the unique and rewarding challenge of working on programming language tooling.

I really liked the refactoring-friendly AST that underpins the IDEA architecture, together with the infrastructure for handling big code bases (indices, PSI stubs). I also liked the approach to unit testing taken in the Scala Plugin.

The downside of course is that is makes it hard to reuse the parser and typechecker from the language.

Alexander: How long have you been at Typesafe and how did you get there?

Jason: I’ve been at Typesafe for almost two years. Before, I’d been an open source contributor to the project in my spare time. Joining the Scala team at Typesafe allowed my to focus my full attention on advancing the Scala platform.

I felt that the perspectives I’d gained applying Scala commercially would be valuable to the project, in addition to the work I would perform on the compiler and standard library.

Alexander: The Scala plugin team noticed that the Scala 2.11 compiler is now 10-20% faster. How was this achieved?

Jason: We had two main strategies for improving the compilation times in 2.11. First and foremost, we wanted to “compile less code”, which is to say we wanted to make incremental compilation smarter. Grzegorz Kossakowski added the “Name Hashing” feature to SBT (which, by extension, is also available in Maven, Gradle, Scala IDE and IntellIJ IDEA).

But the speedup you are referring to comes from our second strategy, which is to “compile faster”. I gathered profiles with YourKit and also with Oracle Solaris Studio and set about finding a large number of small performance wins. The themes were pretty typical: reducing memory pressure, caching results, avoiding needless work, avoiding some abstraction penalties in hot code paths.

Solaris Studio offered some useful insights into costs of megamorphic calls. For example, we use `List` frequently in compiler internal data structures It turned out that replacing someList.nonEmpty with !someList.isEmpty avoided a megamorphic call, which in a handful of hot code paths was worthwhile.

One challenge we faced is validation of results. We struggled to find a benchmarking methodology that let us resolve small performance changes against the noise of non-determinism of HotSpot’s JIT and GC. So while we could show that the end result was a clear win, we still feel like we would benefit from more a precise benchmarking rig.

My understanding is that tools like the excellent JMH aren’t what we’re after as we are performing a macro benchmark, but they are geared towards micro benchmarking.

We’re going to keep at this job during the next year, and deliver additional speedups in the 2.11.x releases.

Alexander: What are the plans for Scala 2.12?

Jason: Scala 2.11 delivered on our themes of being faster, more stable, and smaller. We worked on incremental and batch compiler performance, and improved performance of the collections library. We’ve deprecated some problematic parts of the library, and moved others into dedicated modules. Our community build now builds 1M LOC, taken from the Scala open source ecosystem, on a nightly basis to help us detect regressions in the compiler/library.

In Scala 2.12 roadmap, our primary goal is to make a smooth transition to the Java 8 platform. We want to make it easier to consume Scala APIs from Java 8, and vice versa, and we want to exploit the extensions to the JVM bytecode to more efficiently encode functions and traits.

Because we know that some portion of the community might be unable to migrate away from Java 6/7 immediately, we want to ensure that Scala 2.11 will remain a viable platform for them in the meantime. To this end, we aim to keep the language and standard library relatively stable across 2.11 and 2.12, so as to make it dead simple for the open source community to cross-publish libraries against both releases.

Alexander: Do you have any news about Dotty?

Jason: Dotty is a research project at EPFL (the birthplace of Scala), that is trying out new language concepts and compiler architectures for a Scala-like language.

A few examples of the things it is exploring:

• type system simplifications to reduce the number of core concepts
• cleaning up irregularities in syntax, e.g. moving XML literals out of the core language and into a library
• switching from using the “Cake Pattern” to using implicit parameters as the dependency injection facility in the compiler implementation
• cleaning up the way that metadata is versioned across the phases of the compiler

As this research comes up with good results, we’ll seek to opportunistically cherry pick them into the Scala 2.x series. We’re also open to the idea of packaging up a compelling set of changes into a new major version of Scala a few years down the road.

Of course we’re very mindful of the investment that our customers have made in the current Scala platform (as have we!), so any such move will have to have a strong migration / compatibility story, and consider the ecosystem of tools that surround the language.

Alexander: As you know, the IntelliJ IDEA Scala plugin doesn’t use the presentation compiler. Do you think it will complicate the adoption of Dotty in IntelliJ IDEA?

Jason: As I said before, it is a bit too preliminary to know how this will play out. So this remains somewhat hypothetical.

Syntactically, the Scala-like language that Dotty has implemented is pretty much a subset of Scala proper. So I could imagine that IntellIJ could build out its existing plugin to support the Dotty type system as a new “Language Level” (as it did for Java 8 support, for example).

The key will be having a well-specified type system and type checking algorithm. If a Dotty-based Scala version seeks adoption, it will have to meet people’s expectation with tool support, and I think we’ll need to spec this out to a more detailed level than before. The good news is that dotty’s type inference is much simpler to explain.

Alexander: What other Scala projects are you involved in?

Jason: Most recently, I wrote scala-async with Philipp Haller. We were really happy to be able to express the async/await pattern as a macro, rather than having to bake it into the language itself.

Within Typesafe, I find a little time to contribute code and review to SBT, and occasionally to Akka and the Play Framework.

Looking back a few years, I was involved in the big reorganization of Scalaz 7. It was a big challenge to try to modularize the library and encode type classes in a manner that was harmonious with the way that implicit search and type inference works in Scala.

Alexander: What do you do in your spare time, when not fixing bugs in the Scala compiler?

Jason: This year I’ve taken up the rather time consuming hobby of fatherhood! Otherwise, I really enjoy cooking (and eating!), jogging in the woods around Zurich and snowboarding.

Alexander: What other languages apart from Scala do you enjoy programming in?

Jason: It’s something of a guilty pleasure, but I still get a kick out of throwing together a smart SQL query or a beautiful spreadsheet. Both SQL and Excel are excellent gateways into Functional Programming, IMO! They are also great examples of making programming accessible to non-programmers. Of course both have pretty low ceilings on abstraction, which is what leads to projects like Slick or Scala Notebook.

I’d like to learn a dependently typed language next, Idris looks quite fun.

Thank you, Jason!

Jason is a software engineer in the Scala team at Typesafe. He has been writing Scala for the last five years, both in enterprise and open source projects, including the Scala plugin for IntellIJ IDEA.