JetBrains MPS as a Tool for Extending Java

Vaclav Pech

JetBrains

vaclav@jetbrains.com

Alex Shatalin

JetBrains

alexander.shatalin@jetbrains.com

Markus Voelter

independent/Itemis

voelter@itemis.de

Abstract

JetBrains MPS is an integrated environment for languageengineering. It allows language designers to define new pro-gramming languages, both general-purpose and domain-specific, either as standalone entities or as modular exten-sions of already existing ones. Since MPS leverages the con-cept of projectional editing, non-textual and non-parseablesyntactic forms are possible, including tables or mathemat-ical symbols. This tool paper introduces MPS and showshow its novel approach can be applied to Java development.Special attention will be paid to the ability to modularizeand compose languages.

Categories and Subject Descriptors D.2.6 [SoftwareEngineering ]: Programming Environments - Programmerworkbench

General Terms D.3.2 Extensible languages, D.3.4 Codegeneration, D.3.4 Translator writing systems and compilergenerators

Keywords language extension, DSLs, development envi-ronments, formal methods

1. Introduction

JetBrains MPS is an open-source language workbench basedon a projectional editor. It started as an experimentalproject about ten years ago with the aim to test and vali-date the ideas of Language Oriented Programming (LOP),as summarized in [2]. In essence, MPS attempted to bringcode generation and projectional editing (also known asstructured editing, see [12] for a good definition) to theprogramming mainstream, enabling easy language modular-ization and composition. The approach blurs the distinctionbetween general-purpose (GPL) and domain-specific (DSL)programming languages with an intended positive impacton developer productivity. This paper introduces MPS, fo-cusing on extensibility of languages and IDEs in general andfor Java, specifically.

The term Language Workbench was coined by Mar-tin Fowler [4]. Besides MPS, there are several other lan-guage workbenches including the Intentional Domain Work-

Permission to make digital or hard copies of part or all of this workfor personal or classroom use is granted without fee provided thatcopies are not made or distributed for profit or commercial advantageand that copies bear this notice and the full citation on the first page.Copyrights for third-party components of this work must be honored.For all other uses, contact the owner/author(s).

PPPJ13,, September 1113, 2013, Stuttgart, Germany..Copyright is held by the owner/author(s).ACM 978-1-4503-2111-2/13/09.http://dx.doi.org/10.1145/2500828.2500846

bench [8], MetaEdit+1, Eclipse Xtext 2 and Spoofax 3 (for abrief comparison of some of these tools see Related Work inSection 4). For a general overview of language workbenches,please refer to the Language Workbench Competition4.

Real-world use of MPS As a real-world proof of MPS,JetBrains has built YouTrack 5, a web-based bug trackingapplication. It heavily relies on an MPS-based set of lan-guages for web application development and database map-ping Webr-DNQ. In less than three years YouTrack becamean important part of JetBrains product portfolio, validatingthe LOP idea as well as its implementation in MPS. Moreon how YouTrack benefits from LOP can be found in [9].

The MPS team cooperated with several universities tohelp them adopt MPS as a tool for language design research.For example, a student at the Faculty of Mathematics andPhysics of Charles University in Prague implemented a real-time Java development environment as part of his masterthesis in 2012 [3].

Although MPS is historically bound to Java and theJVM, its principles are universal and can be applied to anytechnological domain. The mbeddr project [11] successfullybuilt a powerful development stack for embedded C develop-ment. They primarily target programmers who write C codefor micro-controllers in robots, avionics or car appliances.

Further plans Inspired by the success of YouTrack andmbeddr, JetBrains has been gradually turning MPS into asolid open-source language workbench that is ready for thetasks in the mainstream of software development. The re-cent 2.5 version can be considered a fully functional, stableproduction-ready tool, targeting language and DSL design-ers as well as programmers. The open source license facili-tates adoption for open-source and academic use.

2. Language Engineering in MPS

Eating its own dog food, MPS, like many other languageworkbenches, offers a set of DSLs for defining various aspectsof languages. These include the structure, editor, type sys-tem, the generator as well as support for sophisticated IDEfunctionality, such as code completion, intentions, refactor-ings, debugger and dataflow analysis.

1 http://metacase.com2 http://eclipse.org/Xtext3 http://spoofax.org4 http://languageworkbenches.net5 http://youtrack.jetbrains.com

You may also check out two introductory screen-casts onprojectional editing in MPS 6 and on language definition7,to get a concrete example of the discussed content.

Abstract Syntax As a first step when defining a newlanguage we specify the structure (aka abstract syntax ormeta model). This resembles object-oriented programming,language concepts are similar to classes. Concepts in MPSrepresent types of AST nodes and define the properties,methods, children and references that instance nodes mayhave. MPS offers three language aspects to define abstractsyntax: the structure aspect defines the concepts, their prop-erties and relationships, constraints restrict the allowed setof values for properties and references, and the behavior as-sociates methods with concepts.

Concrete Syntax The second step is defining the editorfor the concepts. This reflects the projectional nature ofMPS. Since code in MPS is never represented as plaintext (neither on the screen nor on disk), MPS languagesare never parsed and thus no grammar is required. Thisenables the use of non-parseable notations, such as tables ormathematical symbols. Instead of a parser, we define editorsfor language concepts a visual representation for ASTnodes. MPS editor definition capabilities let the languagedesigner define editors for her new language constructs aswell as override editors of concepts from existing languages.Editors can be modularized to support reuse of visual syntaxelements and multiple editors can be defined for a concept,allowing the programmer to choose the notation that fitsbest the task at hands.

The default MPS editor needs some getting used tofor programmers familiar with text editing. Building codefrom predefined language concepts instead of from individualcharacters definitely feels different. However, MPS goes along way towards the editing experience of text editors. Theeditors behavior can be tuned by specifying many of itsdynamic characteristics using, for example, node factories,side transformations, or replacement rules.

IDE Functionality Developers today expect the editorto instantly assist them with code completion, smart navi-gation, intentions, refactorings, hints and code analysis. Be-ing able to quickly build an IDE for a language is one of themost noteworthy advantages of MPS. The editor offers manyof its essential features out-of-the-box, the remainder canbe implemented by the language designer through appropri-ate language aspects. For example, whenever the developerhits Ctrl-Space, the code-completion pop-up menu is auto-matically populated with all type-compatible nodes that arein scope. The language designer can customize the scopingrules of the language, which will be reflected in the contentof the code-completion menu. Also, if the language designerdefines a Dataflow aspect for her language, the user will thenget unreachable parts of the code highlighted as errors.

MPS seamlessly integrates with several version controlsystems (VCS), such as Git and Subversion. Diff and mergeis provided on the concrete syntax level, so that, withoutextra effort on the language designers side, code can beproperly maintained through the projectional editor.

The MPS debugger is also extensible, allowing languagesto be debugged in MPS. The user can set breakpoints and

6 http://tv.jetbrains.net/videocontent/your-first-date-with-jetbrains-mps7 http://tv.jetbrains.net/videocontent/your-second-date-with-jetbrains-mps

inspect values in the high-level language (DSL), avoidingexposure to the actual execution technology used.

Type System The next step is the definition of thetype system. For example, the type of a condition of anIfStatement must be boolean, or a type of a return state-ment must be compatible with the return type of the sur-rounding method. MPS comes with a type system enginethat is capable of evaluating type system rules to assigntypes to language elements (type inference) and to verifycorrectness of types. Typing rules are specified as a set ofequations, such as the following:

infer typeof(ifStatement.condition) :

Figure 1. A template definition that replaces aRoutineDefinition statement with a Java method defini-tion. The COPY_SRC macro replaces the println statementwith the set of statements in the RoutineDefinition.

Java interoperability MPS has historically very strongties to Java. MPS is itself implemented in Java, it runs on theJVM, and a Java dialect named BaseLanguage was the firstlanguage implemented fully in MPS; the language definitionDSLs build on BaseLanguage. It is easy to import existingJava source code into MPS or to use Java libraries in MPSprojects. MPS languages can be packaged as Java librariesand used from within Java IDEs. This all combined makesMPS feel very familiar to Java developers.

Over time MPS has collected numerous BaseLanguageextensions, such as collections, closures, time/date manipu-lation, regular expressions or builders, each enhancing its ca-pabilities far beyond plain Java. Together with the fact thatdevelopers can build their own domain-specific extensions ofBaseLanguage, this makes MPS an interesting solution forJava developers. At the same time, MPS domain of applica-bility is not limited to Java and JVM. The mbeddr project,for example, successfully managed to expand MPS into thedomain of C-based embedded software development.

IDE interoperability The MPS workbench is primar-ily a tool for language definition. Languages and DSLs getdesigned, tested and debugged in MPS and then packagedand distributed to language users, i.e. application develop-ers. Since typical application developers do not need all theMPS language design capabilities and prefer their existingdevelopment environments, such as IntelliJ IDEA or Eclipse,MPS languages can be packaged and distributed as Java IDEplugins (IntelliJ IDEA supported already, Eclipse is plannedfor the end of 2013). The upcoming 3.0 release of MPS willsupport full cross-navigation between Java code in the JavaIDE and code managed by the MPS plugin.

For users of DSLs that are not programmers (system ad-ministrators or business experts), MPS offers the possibilityto generate dedicated Java applications, whose sole purposeis to support editing and maintain code using the packagedlanguages. The YouTrack administrator workflow console isan example of such a dedicated lightweight IDE.

3. Language Composition

Language modularization and composition is the core ideabehind LOP. MPS languages can easily refer, reuse and em-bed one another. Since projectional editing disambiguatesconstructs originating from different languages, MPS doesnot require the construction of a unifying parser or the def-inition of composite grammars. Languages can be extendedwith new constructs and new ways to view and edit them.See [10] for a detailed discussion of language modularization.

In order to successfully build modularized languages, allaspects of language definition must be modularizable andcomposable. We discuss the most important ones in the

Figure 2. A sample use of ParallelFor in Java.

following subsections. We use a parallel for loop to illustratethe approach. Fig. 2 shows how it is used.

Abstract syntax Earlier in the paper we have made theassociation between the language concepts and objects inobject orientation. This analogy also holds for language ex-tension. Concepts can extend by other concepts, and sub-concepts can be used polymorphically. Fig. 3 shows how theparallel-for loop extends the AbstractLoopStatement con-cept, which is defined in BaseLanguage. Since ParallelForcan now be used wherever AbstractLoopStatement is ex-pected, the new extension will seamlessly integrate into theoriginal language. Just like methods on objects in OOP, be-havior methods attached to concepts can be overridden bytheir subconcepts and their constraints may be altered.

The Adapter pattern [5] for language composition usesthis extension mechanism to allow for other types of lan-guage integration reuse, referencing and embedding. Forexample, to embed an existing language of mathematicalformulas into Java, wed have to create an adapter (lan-guage) that wraps the formulas or parts of them into Javalanguage concepts, such as Statement or Expression. Oncewrapped, the formulas can become part of the Java abstractmodel, can be rendered and edited on the screen and alsogenerated, using the reduction rules defined by the wrap-pers. The adapter language would also define scoping rulesfor the formulas so that they could refer to one another orto, for example, Java variables in the surrounding Java code.

Concrete syntax The MPS editor assigns visual nota-tions to language concepts; each concept is responsible for itsown rendering and user interaction. This mechanism worksirrespective of the language the concepts has been definedin. So a math formula will render itself correctly, no mat-ter whether it is part of a Java program or an electricalcircuit simulation model, for example. Subconcepts inheritthe editor of their parent concept, unless they define theirown, more specific editor. Extending languages may supplytheir own editors for inherited concepts and thus overridethe concrete syntax derived from the inherited editor.

IDE functionality It is possible to alter many of theIDE aspects of a language through language extension. Lan-

Figure 3. Structure definition for ParallelFor.

guages may provide additional refactorings, intentions, edi-tor actions and others.

Typesystem Together with new concepts, languages mayintroduce new types and relate them to types of otherlanguages through subtyping rules. All type equations fromall used languages enter the pool of rules that the typesystem engine resolves. This declarative approach makes thetype system easily extensible.

Generator By extending the generator, extensions canalter the semantics of the original language. The MPS gen-erator resolves the generator rules and mapping configura-tions from all languages attached to the project and buildsa global generation plan. The plan specifies the executionorder for the generator rules based on their mutual relativepriorities expressed in mapping configurations. This enableslanguage extensions to inject their own desired generationrules into the most suitable generation phase. Since priori-ties are expressed as a collection of relative ordering betweenmapping configurations, a language extension does not needto know about all other generators involved in the genera-tion of a particular model. Potential (and rare) clashes aredetected and left to the developer to resolve. Once created,the generation plan is used to iteratively invoke the genera-tors, potentially leveraging the parallelism of the underlyinghardware for mutually independent rules.

Providing additional reduction rules is one way to extenda language. Using a generator switch is another option. Ifthe parent language uses a generator switch to choose theright reduction rules, the language extension may extendthat generator switch with its additional logic for picking thereduction rules typically to include new rules contributedby extension languages.

4. Related Work

Parser-based systems Projectional editing makes MPSa rare species, together with the Intentional Domain Work-bench and oomega12. Although it is not a new idea, textediting is more widespread. To name a few workbenchesfrom the parser camp, we should definitely mention Xtextand its related projects - Xbase, Xpand and Active anno-tations. MetaEdit+ and Spoofax are also good examples ofworkbenches built around parsing and text editing.

Text-based workbenches feel more familiar to todays de-velopers, as editing text is the mainstream approach to writ-ing code. Also, integration with the existing developmenttool chain, such as VCSs, code review tools or code shar-ing facilities, is pretty straightforward when the sources aretext. On the other hand, text-based tools restrict the syn-tax of languages to textual notations. Tables, math symbolsor graphical representations cannot easily be included. Forthe same reasons, these tools typically do not let developersswitch between alternative notations. Finally, defining ro-bust composable languages is extremely difficult to achievewith grammars [1, 6]. In a similar vein, domain experts, whotypically are the target audience for DSLs, and who do notnecessarily have a strong background in programming, mayfind projectional editing more convenient, as they do notcarry the baggage of old habits and since projectional edi-tors can be easily made to resemble the tools or notationsthese experts use in their daily practice. Refer to [11] formore details on contrasting the two types of workbenches.

12 http://www.oomega.net/produkt/

Internal DSLs in modern GPLs The raising popularityof alternative JVM languages, such as Kotlin, JRuby, Scalaor Groovy can be understood as an indication of Java, thelanguage, not satisfying todays developers needs. Concisesyntax for collections and maps, closures, data/time manip-ulations, regular expressions are a few examples of wherethe newcomers beat Java in expressiveness and ease of use.One important contribution of these languages is their flex-ibility towards creating internal DSLs [7]. Internal DSLs area useful way to increase expressiveness of a GPL with lit-tle or modest effort. Using the language-specific run-timeor compile-time mechanisms, the language can be extendedto express common idioms in a more concise way. However,these DSLs are limited to various degrees by the hostlanguages syntax and they cannot extend the host languagetype system. Finally, the IDE support for such DSLs is eitherignored completely, or very limited.

5. Summary

MPS is a powerful tool for language development with spe-cial focus on easy language modularization and composition.In this paper we explored the way to extend languages inMPS and to customize language IDE support. The MPSprojectional editor is an important advantage that not onlyoffers notational freedom, but also enables flexible composi-tion of languages. Thanks to its rich set of existing Java ex-tensions and tight integration with Java IDEs, MPS enableseasy interoperability between plain Java, BaseLanguage andDSLs. Despite its relatively steep learning curve, MPS couldhelp the adoption of the LOP principles among softwarepractitioners and move the industry forward by making pro-gramming languages truly customizable.

References[1] M. Bravenboer and E. Visser. Parse table composition. In

R. L. In D. Gasevic and E. V. Wyk, editors, Revised SelectedPapers of SLE 2008, Toulouse, France. LNCS Vol. 5452,2009.

[2] S. Dmitriev. LOP - Language Oriented Programming: TheNext Programming Paradigm. 2004.

[3] T. Fechtner. MPS-based Domain-specific Languages for real-time Java development. 2004.

[4] M. Fowler. Language Workbenches: The Killer-App forDomain Specific Languages?, 2005.

[5] M. Fowler. Domain-Specific Languages. Addison Wesley,2010.

[6] T. Goldschmidt. Towards an incremental update approachfor concrete textual syntaxes for uuid-based model reposito-ries. In R. L. In D. Gasevic and E. V. Wyk, editors, RevisedSelected Papers of SLE 2008, Toulouse, France. LNCS Vol.5452, 2009.

[7] D. Gosh. DSL in Action. Manning Publications co., 2010.

[8] C. Simonyi, M. Christerson, and S. Clifford. Intentionalsoftware. In OOPSLA, pages 451464, 2006.

[9] V. P. Valeria Adrianova, Maxim Mazin. MPS use ofYouTrack case study, 2012.

[10] M. Voelter. DSL Engineering - Designing, Implementing andUsing Domain Specific Languages. Markus Voelter, 2012.

[11] M. Voelter, D. Ratiu, B. Schaetz, and B. Kolb. mbeddr:an extensible C-based programming language and IDE forembedded systems. In Proc. of SPLASH 2012. ACM.

[12] M. Voelter and K. Solomatov. Language Modularizationand Composition with Projectional Language Workbenchesillustrated with MPS. In M. van den Brand, B. Malloy, andS. Staab, editors, SLE 2010, LNCS. Springer.