How to Write Custom Language Support Plugins

Today we would like to share with you a simple tutorial how to write a plugin with custom language support for IntelliJ IDEA and IntelliJ Platform.

As you know IntelliJ IDEA provide powerful facilities for developers to implement advanced code assistance for custom languages and frameworks. In this step-by-step tutorials you will learn how to use Grammar-Kit to generate a parser and PSI elements, how to use JFlex to generate a lexer, how to implement custom formatting, refactoring support, etc.

The sample code for this tutorial can be found on GitHub.

More steps with other aspects of plugin development are coming soon. In the meanwhile don’t miss the opportunity to register for the second live coding webinar about plugin development, which will take place on Tuesday, January 22.

Develop with Pleasure!

About Andrey Cheptsov

Andrey Cheptsov is IntelliJ IDEA product marketing manager at JetBrains. He's passionate about productivity, programming languages and tools.
This entry was posted in Plugin Development, Tutorials. Bookmark the permalink.

28 Responses to How to Write Custom Language Support Plugins

  1. Ladislav Thon says:

    This is great! If only this was available a month ago, when I started playing with this. By now, I’ve figured out almost everything of this by myself, using the Erlang plugin and the Developing Custom Language Plugins for IntelliJ IDEA page (http://confluence.jetbrains.net/display/IDEADEV/Developing+Custom+Language+Plugins+for+IntelliJ+IDEA). I haven’t needed the reference contributor, though. What is it needed for?

    Also, what I’m playing with now involves navigating across files. Stub indexes etc. Cool stuff, although I’m not sure I fully understand it. The Indexing and PSI Stubs in IntelliJ IDEA page (http://confluence.jetbrains.net/display/IDEADEV/Indexing+and+PSI+Stubs+in+IntelliJ+IDEA) only provides a very basic introduction.

  2. Ladislav Thon says:

    Oh, and reference resolution. That’s pretty important topic, I’d suggest covering the scope processors stuff as well. It isn’t needed for such extremely simple language, but it’s absolutely essential once you start playing with something real. Thank you!

  3. Konrad says:

    Please share tutorial how to write a complete plugin for PyCharm;) or API Reference

  4. Alberto says:

    Will such a custom language plugin only work in IntelliJ IDEA or also with the other IDE’s, like PHPStorm or PyCharm?

  5. Luke says:

    Could you please also explain, on an example, how the pin and recoverUntil work, as well as how to simplify the AST trees by using the extends?

  6. Cemo Koç says:

    Awesome!

  7. Tobias Lidskog says:

    This is great! I noted that the “Lexer and Parser Definition” page is missing a step to create the SimpleFile class that is used in the parser definition. Looking forward to writing my first plugin now.

  8. Joe says:

    @Alberto: It usually works in all IDEs, depends on how you define the metadata.

  9. Yann Cebron says:

    @Alberto: please see http://confluence.jetbrains.net/display/IDEADEV/Plugin+Compatibility+with+IntelliJ+Platform+Products for more information about making your plugin compatible with other IntelliJ based products

  10. Andrey Cheptsov says:

    @Tobias You are right, thanks. Fixed.

  11. Ladislav Thon says:

    +∞ to Luke. The GrammarKit documentation altogether would need some love, but the only important thing I really didn’t understand is pin/recoverUntil. Or actually, I think that I understand them (when a pinned element of the rule is successfully parsed, the rule itself will be considered successfully parsed, and the parser will eat all the following tokens until it reaches some tokens from the recoverUntil set), but I have a really hard time using them correctly. Basically, since I have a C-like syntax, I just settled with a single pinned element in a grammar and a single recoverUntil on ‘;’ or ‘}’. Everytime I try to add some more, the parser stops working and I have no idea why :-)

  12. yole says:

    Ladislav,
    If you have any specific questions regarding the stubs/indexes, please ask, and we’ll update the document to make sure they’re covered.

  13. yole says:

    Konrad,
    Check out lib/src/pycharm-openapi-src.zip in the PyCharm distribution and https://github.com/JetBrains/intellij-plugins/tree/master/pycharm-flask

  14. thomas says:

    Great product! This time its mine 

  15. Sergey says:

    @Ladislav, @Luke Please check out http://goo.gl/l7cVj and the 5 lines below. This is a perfectly working example of recoverUntil concept.

  16. Ladislav Thon says:

    @Sergey Thanks, but I was able to get to something very similar. I have a working rule with pin/recoverUntil in my grammar, just by following the documentation. That, however, isn’t enough to really understand. Once I get back to it, I will probably end up reverse engineering the concept from differences in generated code.

    Oh, and one thing I found just before few minutes. GrammarKit doesn’t support generating classes needed for stub indexes. It looks like it is possible to write all that manually, but it requires working around some GrammarKit bugs: 1. GrammarKit doesn’t like generics (extra interface for each StubBasedPsiElement needed), 2. GrammarKit doesn’t generate proper imports in generated visitor (and for this, there is a pull request with fix for more than 2 months!). And that’s only a beginning for me.

  17. Gregory Shrago says:

    http://goo.gl/19mM0 may shed more light on grammar-kit error recovery

  18. Sven says:

    The FindUsagesProvider-section uses an static word-scanner-instance.
    public WordsScanner getWordsScanner() {
    return WORDS_SCANNER;
    }
    As far as I know this will cause threading issues as the lexer is not thread safe.

  19. Andrey Gayvoronsky says:

    Is any way to extend existing ‘language plugin’ with minimal efforts?

    For example, i need plugin for Squirrel language:
    http://squirrel-lang.org

    Its like slightly simplifier and abit extended version of Java.

  20. Prasan says:

    Thanks for the tutorial, but IntelliJ doesn’t seem to associate the file type specified with my language after following step 2: Language and file type.

  21. Scott Wells says:

    Totally agreed that this is an outstanding tutorial! However, like Luke and Ladislav, I still don’t quite understand how best to use pin and recoverUntil, even after looking at the Erlang grammar. The language I’m implementing is very Java-like, and I’ve made some progress with pins so that I now get “foo expected, found bar” most of the time, but I have a feeling I’m abusing it rather than using it. Also, every time I try to use recoverUntil, primarily by using an expression such as “!(SEMICOLON | RBRACE)”, I just end up tanking the parser.

    I’d be happy to share a few simple rules from my grammar as concrete examples of those help the discussion.

  22. Noah says:

    I’m sorry but while this “tutorial” may be a good start to see what operations must be done in which order, it’s not even close to being a tutorial. It’s a recipe, there is almost no explanation for most of the steps and the reader is just led blindly through it.

    And the example is not practical either. I understand it has to be simple, but like so many examples it just lacks the fundamental links people need in order to build an actual case.

    What for example if I want to build a real parser, it will have to let the grammar get some information from what the lexer is sending (number value, identifier string, …). The JFlex documentation shows how to define a type that contains this information but here it’s all really fuzzy, I gather that somehow it has to be based on IElementType but then what? If I scan a string, where do I put the contents? If it’s an integer, where should the value go?

    This could be a basic grammar, no need for complex stuff, but at least it would show the reader how to do practical things.

    Is there any good example of a language plugin out there which is based on a JFlex lexer and a .bnf grammar definitions? The only ones I found did their own recipe for the grammar part, either with Scala routines or home-made code which seems to amount to ten times the effort (or even trying to squeeze part of the grammar in the lexer, which makes it even worse).

    Thanks for all the work done so far anyway :)

    • vishnu says:

      @Noah I’ve just started with plugin development for a custom language and even I have the same feeling did you get to find any resources or tutorials which could have helped you? Thanks in advance

  23. Noah says:

    And actually it doesn’t work.

    If you take the project from Git, it compiles the first time. Try and generate the parser from the BNF file and it all goes haywire.

    Generated parser:
    import static com.simpleplugin.parser.GeneratedParserUtilBase.*;
    [...]
    builder_ = adapt_builder_(root_, builder_, this, null);

    In the GeneratedParserUtilBase.java file, you realize that the static constructor has a different signature:

    public static PsiBuilder adapt_builder_(IElementType root, PsiBuilder builder, PsiParser parser)

    And by the way, the GeneratedParserUtilBase.java file you provide as a separate link is completely different and also incompatible. It already starts with this:

    package org.intellij.grammar.parser;

    which is actually a different package! What a mess. Why isn’t that file part of the SDK in the first place, as should the GrammarKit, it’s the best way to ensure consistency between those parts.

    • Noah says:

      Trying to change the package to match the com.simpleplugin.parser is not enough. Now the PsiBuilder type is missing the .rawTokenIndex() method, PsiBuilderImpl.ProductionMarker is missing the .getStartIndex() method.

      Trying to change the SDK from 129.1359 to 133.286, regenerating parser. Nope, now PsiBuilder.rawTokenIndex() is there, but PsiBuilderImpl.ProductionMarker is still missing getStartIndex(), plus a new getEndIndex() method…

      I’m giving up for now, the lack of documentation or at least of a relevant example, the inconsistency between those parts, make it too much trouble to use. Back to Eclipse and Xtext I guess, a shame.

      • Noah says:

        Found the bug, apparently this line has to be changed in the BNF, since this file IS indeed part of the SDK and doesn’t need to be included from elsewhere:

        stubParserClass=”com.intellij.lang.parser.GeneratedParserUtilBase”

        It only works with SDK 133.286 though, with the other ones it produces more errors that I will care to report.

        So it finally compiles. Then all the unit tests but one fail in SimpleCodeInsight. The SimpleParsingTest simply crashes…

        • Vektah says:

          For anyone encountering this issue I can run with the current master by NOT using the supplied ‘light-psi-all.jar’ and instead using my current install of intellij.

          java -cp ‘/path/to/grammar-kit.jar:/path/to/intellij/lib/*’ org.intellij.grammar.Main gen src/bnf/MyGrammar.bnf

  24. Shishir says:

    I want to navigate from variable to its declaration etc, but i dint find any anything related to navigation in the tutorial. can somebody please help me out for the same.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">