Nitra goes Open Source!
We’ve been rather quiet about Nitra since our introductory blog post, back in November of last year. But we haven’t been sitting still. We’ve been busy building the foundations to a project that we want to see as the basis for a lot of future work here at JetBrains. Now that we feel we’ve reached a minimum viable feature set, as well as a good architectural base for future development, we’re very happy to take the next steps on our roadmap, and release Nitra as an Open Source project.
The source is currently available on GitHub, released under the Simplified BSD license.
What is Nitra?
As a reminder, Nitra is a framework for the next generation of language tooling.
Traditionally, languages don’t ship with any kind of IDE support. It is up to tools vendors to provide syntax highlighting, outlining, code completion, navigation, find usages, refactorings and so on. Some of these features, such as syntax highlighting and outlining can be crudely implemented in a manner similar to regular expressions, but the more powerful features require a deeper understanding of the language.
Ideally, a language service should be able to parse a file into a form that it can reason about – an abstract syntax tree (AST). With such a deep knowledge of the underlying language, it knows what nodes in the tree are keywords for syntax highlighting, knows where blocks start and end for outlining, can pull out identifiers and their usages for code completion and navigation. Manipulating the tree can reformat or even rewrite the source code, enabling refactorings.
This is what JetBrains does right now, and, as a company, we’re very good at it – ReSharper ships with parsers and ASTs for approximately 20 file formats, and the IntelliJ ecosystem supports many more. As one product manager put it: “We’re JetBrains. We write parsers for breakfast”. But it’s busy work.
Nitra is designed to take away this busy work, allowing us to support more languages, quicker, and to focus on the higher levels in the stack of language tooling, such as analyses and quick fixes.
Nitra is a language for describing grammars, and the abstract syntax trees to represent them. It is also a build tool to compile the grammars into parsers, and generate the classes required to represent and navigate the AST. And it’s also a set of language services that use the grammar to provide automatic syntax highlighting, outlining and brace matching.
What’s more, Nitra’s grammars are designed to be extensible and reusable. A simple example would be the base building block grammars that ship with Nitra, such as Whitespaces and CStyleComments. Both clearly useful in many languages, and can be simply included and reused in a larger grammar.
A better example would be HTML, which allows for embedded “islands” of JavaScript and CSS. The HTML grammar could include the JavaScript and CSS grammars and allow full parsing of a typical HTML file. Take it further, and look at .net’s Razor web templating engine. This is a server-side HTML file that includes not only JavaScript and CSS, but also C#. Build a grammar that includes HTML, JS, CSS and C#, and you’ve got a parser and an AST that can handle Razor files.
Read Hadi’s Introduction to Nitra post to see an example grammar, and check out the wiki for documentation, especially for the syntax of the grammars.
Current Status
While we’re making Nitra Open Source today, this should not be considered an end-user preview. It’s a low-level language toolkit, and if you want to use it, expect to get your hands dirty. Here’s what it currently supports:
- Grammars to define lexer-less parsers and ASTs that can be extended at compile time or runtime, supporting composition of languages, or languages with extensible syntax, such as Nemerle
- A command line tool to compile grammars into an assembly containing the parser and the AST classes (grammars can also be compiled inside Visual Studio)
- A Windows application to load and test grammars
- A Visual Studio language service that uses Nitra to parse files and provide automatic syntax highlighting, outlining and brace matching. Current work is to increase the automatic support here to include go to declarations, find usages and rename refactorings
But this is still a work in progress. There are things that don’t yet work. The source reflects this in places, with incomplete grammars (for example, C++) and some references to internal test tools and file locations that aren’t included in the source code. This just reflects the status of the code as an active project, and will get cleaned up over time.
Building Nitra
Before you can use Nitra, you need to build it. Full instructions for building Nitra can be found here. Here’s an outline:
- Install Nemerle. Nitra is built by the Nemerle team, in Nemerle, so requires Nemerle in order to compile. You can download the latest version from the Nemerle site (there currently isn’t a VS2013 build that supports Nitra. One will be made available shortly. In the meantime, you need to use VS2012)
- Once installed, you need to open a command prompt in the Nitra folder, and run “BuildBoot.cmd”. This bootstraps the build process, building the boot versions of the assemblies required by the main product in order to build. you only need to do this once.
- Build the Nitra.sln solution in Visual Studio
- Set the environment “NitraPath” to point to the root of the source distribution. The Visual Studio integration uses this to locate the “NitraGlobalConfig.xml” file which tells it which grammars to load
Using Nitra
The solution contains several test applications you can use to give Nitra a go.
Firstly, there is Nitra.Visualizer, which is a Windows application that is pre-configured to load test files from the Nitra\Tests\Visualizer folder. It will use the grammars defined in the Nitra solution to load test files and compare the output with a known-good “gold” file. If the output is different, the test fails, and you can see the differences in the “Diff” tab. If the test succeeds, the file is displayed in the right hand pane, with syntax highlighting and outlining applied. You can also navigate the AST under the “Reflection” tab, and see how Nitra will output it in a pretty-printed version.
There are several different tests and languages supported here, including C# and JSON, and a calculator DSL, too.
In fact, there’s also another sample app that uses the calculator DSL directly – Sample.Calc.App. This is a console application written in Nemerle that references the calculator grammar and parses input from the command line. It then pretty prints the expression you enter, outputs it with syntax highlighting, and evaluates it, using a Value() method that is defined as part of the grammar.
And of course, there’s the Visual Studio integration. Once this project builds, you’ll get a Nitra.VisualStudio.vsix file in the bin\Debug folder. Double clicking on this will install it into Visual Studio (you can also set up the debugging command line so you can just F5 the project). Don’t forget to set the “NitraPath” environment variable as mentioned above. The extension adds support for *.json files, and will parse *.csn files as C#, but most usefully, it will parse, syntax highlight and add outlining and brace matching for *.nitra grammar files.
There are other sample projects in the solution, and you can also run the test suite with the “run.cmd” batch file in the Nitra\Tests folder.
Why Open Source?
The plan has always been to release Nitra as Open Source. After all, it’s written in Nemerle, by the Nemerle team, and Nemerle is Open Source. However, we wanted to wait until we had a core platform that was ready to be opened up further. We feel that we have that now, so here we are.
And of course, we’re accepting contributions! If you’d like to get involved and help build the next generation of language tooling, take a look, raise issues, send pull requests.
Please let us know what you think!
Subscribe to JetBrains Blog updates
