Try Faster Scala Compiler in IntelliJ IDEA 13.0.2

The latest update to Scala plugin (v0.30.375) available for IntelliJ IDEA 13.0.2 lets you choose what compiler you want to use: SBT incremental or IntelliJ IDEA.

The latter is actually the new built-in incremental Scala compiler based on IntelliJ IDEA internal byte-code dependency analyzer, that is capable of achieving up to 25% faster compiling speeds by ignoring certain Scala-specific dependencies between source files. We call this the lower precision approach, because we’re sacrificing a little bit of accuracy for more performance.

Here are the benchmarks of how the plugin compiles itself:

The new compiler is now used by default, so you can try it right away and see how faster your code is compiled, and if you need precision more than speed, you can always switch back to SBT.

Note: Sometimes when compilation starts producing unexpected or weird errors it helps to rebuild the project entirely. If it doesn’t though, contact us and we’ll look into the problem.

Feel free to share your feedback on the discussion forum and submit bug reports to our issue tracker.

Enjoy!

This entry was posted in New Features and tagged , . Bookmark the permalink.

18 Responses to Try Faster Scala Compiler in IntelliJ IDEA 13.0.2

  1. Mike Slinn says:

    You might want to tell the reader how to display the dialog, including how to set up if IDEA is configured to external build mode – which is the default, I think.

  2. Pingback: IntelliJ IDEA 13.0.2の新しいScalaコンパイラ – 最大で25%高速化 | JetBrains ブログ

  3. Anthony says:

    Can you tell us more about what the following phrases mean?

    “… ignoring certain Scala-specific dependencies between source files.”
    “We call this the lower precision approach, because we’re sacrificing a little bit of accuracy for more performance.”

    What are some practical implications of this trade-off?

    • Alexander Podkhalyuzin says:

      Mostly it’s problems with implicits dependencies (which SBT compiler also has in some cases), which are not visible in bytecode. For example if you add new implicit conversion it can conflict with existing implicits in some other file. So there is no bytecode dependency, however we have sources incompatibility.

      Current state is similar to IntelliJ IDEA 11 or IntelliJ IDEA 12 internal compiler, which was almost ok, but much faster, because less files go for recompilation.

      Best regards,
      Alexander Podkhalyuzin.

  4. Ismael Juma says:

    Hi,

    Have you compared the performance of the new mode with SBT with naming hashing enabled?

    https://groups.google.com/d/msg/sbt-dev/KmGEhFTMve0/NWYk-WW1WAAJ

    That seems like a more promising direction as it doesn’t trade accuracy for performance. It’s pretty bad to end up with runtime errors because a dependency that should have been recompiled but was not.

    Best,
    Ismael

    • Alexander Podkhalyuzin says:

      Thank you for the link, we will take a look for it, I think it’s good idea to use it, if it’s really faster.

      New compiler is similar to IntelliJ IDEA 11 compiler or IntelliJ IDEA 12 internal compiler. As I know accuracy was really good. But anyway, we don’t want to drop SBT compiler with 100% accuracy, that’s why SBT compiler is still working, especially if it’s possible to make it faster (from your link).

      Best regards,
      Alexander Podkhalyuzin.

      • Grzegorz Kossakowski says:

        Hi Alexander!

        I’m the author of name hashing algorithm. You can find more details about it here: https://github.com/gkossakowski/sbt/wiki/Incremental-compiler-notes#wiki-name-hashing-algorithm

        I discuss implicits there and explain how name hashing is able to handle them properly.

        See also: https://gist.github.com/jroper/6374383

        • Alexander Podkhalyuzin says:

          Yes, algorithm is simple and good, I like it. We will check this changes in our SBT based compiler too. Thank you!

          Your algorithm still can’t handle more difficult examples (it’s not possible at all I think):

          object Foo {
          trait A
          trait B
          class C extends A with B
          implicit def a2b(a: A): B = new B {}
          // implicit def b2a(b: B): A = new A {}
          def foo(x: A) = 1
          def foo(x: B) = 2

          def goo =
          foo(new C)
          }

          Imagine implicits in other file. So if you uncomment one of them compilation will be broken, but I’m not sure it will be handled by any incremental compilation algorithm. It was synthetic case, but I’m sure it’s possible to generate something more usable in real life.

          Best regards,
          Alexander Podkhalyuzin.

          • Grzegorz Kossakowski says:

            Hi Alexander!

            Thanks for review of the algorithm! Let me know how your testing went. There’s one issue you should be aware of, though: https://github.com/gkossakowski/sbt/issues/8

            When it comes to your example, I don’t think I follow it. I believe the code you posted with overloaded foo would fail to compile due to ambiguous overload error. Also, since C extends both A and B I don’t see how implicits would kick in. Could you describe a little bit more precisely the scenario you have in mind?

      • Ismael Juma says:

        Looking forward to seeing how it works out. :)

        Ismael

  5. John O'Malley says:

    Alexander – thanks so much for the great work you’ve done on the Scala plugin. Introducing Scala in our organization (we’ve been Java + Intellij IDEA for many, many years) has been a lot easier with all of the improvements you’ve introduced in the past year.

  6. peg says:

    With the new Scala plugin, I get this strange compile error. Reimporting the project does not
    solve the problem. Invalidating/restarting doesn’t help either.

    The project compiles fine (outside the IDE) using SBT at the command line.

    I would appreciate any suggestions or guidance.

    ================================
    scala: Error: String index out of range: -1
    java.lang.StringIndexOutOfBoundsException: String index out of range: -1
    at java.lang.String.charAt(String.java:686)
    at scala.reflect.io.ZipArchive$.splitPath(ZipArchive.scala:49)
    at scala.reflect.io.ZipArchive$.scala$reflect$io$ZipArchive$$dirName(ZipArchive.scala:46)
    at scala.reflect.io.ZipArchive.getDir(ZipArchive.scala:123)
    at scala.reflect.io.FileZipArchive.iterator(ZipArchive.scala:136)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.reflect.io.AbstractFile.foreach(AbstractFile.scala:91)
    at scala.tools.nsc.util.DirectoryClassPath.traverse(ClassPath.scala:307)
    at scala.tools.nsc.util.DirectoryClassPath.x$16$lzycompute(ClassPath.scala:316)
    at scala.tools.nsc.util.DirectoryClassPath.x$16(ClassPath.scala:316)
    at scala.tools.nsc.util.DirectoryClassPath.packages$lzycompute(ClassPath.scala:316)
    at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:316)
    at scala.tools.nsc.util.DirectoryClassPath.packages(ClassPath.scala:296)
    at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:374)
    at scala.tools.nsc.util.MergedClassPath$$anonfun$packages$1.apply(ClassPath.scala:374)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.tools.nsc.util.MergedClassPath.packages$lzycompute(ClassPath.scala:374)
    at scala.tools.nsc.util.MergedClassPath.packages(ClassPath.scala:369)
    at scala.tools.nsc.symtab.SymbolLoaders$PackageLoader.doComplete(SymbolLoaders.scala:243)
    at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:194)
    at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1229)
    at scala.reflect.internal.Mirrors$RootsBase.init(Mirrors.scala:240)
    at scala.tools.nsc.Global.rootMirror$lzycompute(Global.scala:59)
    at scala.tools.nsc.Global.rootMirror(Global.scala:57)
    at scala.tools.nsc.Global.rootMirror(Global.scala:37)
    at scala.reflect.internal.Definitions$DefinitionsClass.(Definitions.scala:164)
    at scala.reflect.internal.Definitions$definitions$.(Definitions.scala:20)
    at scala.reflect.internal.SymbolTable.definitions$lzycompute(SymbolTable.scala:13)
    at scala.reflect.internal.SymbolTable.definitions(SymbolTable.scala:13)
    at scala.tools.nsc.Global$Run.(Global.scala:1290)
    at xsbt.CachedCompiler0$$anon$2.(CompilerInterface.scala:116)
    at xsbt.CachedCompiler0.run(CompilerInterface.scala:116)
    at xsbt.CachedCompiler0.run(CompilerInterface.scala:102)
    at xsbt.CompilerInterface.run(CompilerInterface.scala:27)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:102)
    at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:48)
    at sbt.compiler.AnalyzingCompiler.compile(AnalyzingCompiler.scala:41)
    at sbt.compiler.AggressiveCompile$$anonfun$6$$anonfun$compileScala$1$1$$anonfun$apply$3$$anonfun$apply$1.apply$mcV$sp(AggressiveCompile.scala:106)
    at sbt.compiler.AggressiveCompile$$anonfun$6$$anonfun$compileScala$1$1$$anonfun$apply$3$$anonfun$apply$1.apply(AggressiveCompile.scala:106)
    at sbt.compiler.AggressiveCompile$$anonfun$6$$anonfun$compileScala$1$1$$anonfun$apply$3$$anonfun$apply$1.apply(AggressiveCompile.scala:106)
    at sbt.compiler.AggressiveCompile.sbt$compiler$AggressiveCompile$$timed(AggressiveCompile.scala:173)
    at sbt.compiler.AggressiveCompile$$anonfun$6$$anonfun$compileScala$1$1$$anonfun$apply$3.apply(AggressiveCompile.scala:105)
    at sbt.compiler.AggressiveCompile$$anonfun$6$$anonfun$compileScala$1$1$$anonfun$apply$3.apply(AggressiveCompile.scala:102)
    at scala.Option.foreach(Option.scala:236)
    at sbt.compiler.AggressiveCompile$$anonfun$6$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:102)
    at sbt.compiler.AggressiveCompile$$anonfun$6$$anonfun$compileScala$1$1.apply(AggressiveCompile.scala:102)
    at scala.Option.foreach(Option.scala:236)
    at sbt.compiler.AggressiveCompile$$anonfun$6.compileScala$1(AggressiveCompile.scala:102)
    at sbt.compiler.AggressiveCompile$$anonfun$6.apply(AggressiveCompile.scala:151)
    at sbt.compiler.AggressiveCompile$$anonfun$6.apply(AggressiveCompile.scala:89)
    at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:39)
    at sbt.inc.IncrementalCompile$$anonfun$doCompile$1.apply(Compile.scala:37)
    at sbt.inc.Incremental$.cycle(Incremental.scala:75)
    at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:34)
    at sbt.inc.Incremental$$anonfun$1.apply(Incremental.scala:33)
    at sbt.inc.Incremental$.manageClassfiles(Incremental.scala:42)
    at sbt.inc.Incremental$.compile(Incremental.scala:33)
    at sbt.inc.IncrementalCompile$.apply(Compile.scala:27)
    at sbt.compiler.AggressiveCompile.compile2(AggressiveCompile.scala:164)
    at sbt.compiler.AggressiveCompile.compile1(AggressiveCompile.scala:73)
    at org.jetbrains.jps.incremental.scala.local.SbtCompiler.compile(SbtCompiler.scala:59)
    at org.jetbrains.jps.incremental.scala.local.LocalServer.compile(LocalServer.scala:25)
    at org.jetbrains.jps.incremental.scala.remote.Main$.nailMain(Main.scala:31)
    at org.jetbrains.jps.incremental.scala.remote.Main.nailMain(Main.scala)
    at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.martiansoftware.nailgun.NGSession.run(Unknown Source)

  7. Dejan says:

    Does this apply to Scala projects powered by Gradle also ?

    • Alexander Podkhalyuzin says:

      Gradle for Scala is not yet supported in IntelliJ IDEA. It will be implemented in IDEA 13.1.

      Best regards,
      Alexander Podkhalyuzin.

  8. Yung-Lin Ho says:

    The external build mode is incompatible with TestNG. A bug has been filed but no one ever care about it.

    If you have suffering from this issue

    scala: error while loading ITestContext, class file ‘.m2/repository/org/testng/testng/6.8.7/testng-6.8.7.jar(org/testng/ITestContext.class)’ is broken

    In order to get thing working, you will need to add a google-guice to project dependencies.

    http://youtrack.jetbrains.com/issue/SCL-5781

  9. Sergey Shishkin says:

    Is there any progress on the issue of long ScalaTest compilation cycles? https://stackoverflow.com/questions/21826541/configuring-intellij-idea-with-background-compilation-for-specs2-tests. I observe the issue with IDEA 13.1.5.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">