creating a language editor using dltk
TRANSCRIPT
Building a Custom Language Editor leveraging DLTK
Rupal Patel, Kaniska Mandal.
Overview:
Eclipse base text editing framework:
Eclipse provides rich support for creating programming language editors that operate on text.
The text editing framework follows the same architectural principles as the rest of the Eclipse Platform. The four layers are the model (core), the view (SWT- Styled Text), the controller (JFace - ISourceViewer), and the presentation context (usually the workbench part - ITextEditor).
MVC Collaboration in Eclipse Text Editing Framework
The goal is to create a language editor based upon dynamic language toolkit.
Problem Space:
Tool vendors need a free flow editor for the business analysts and tool developers to code in tool-specific languages. The editor should provide all types of editing, debugging and launching facilities (syntax coloring, auto-completion, content assist, code folding, selection, problem markers, outline view, AST parser, etc.).
Solution Approach:
Eclipse offers a robust language tooling framework.
The Dynamic Languages Toolkit is a set of extensible frameworks for building IDEs for dynamically-typed languages.
Structural Elements of an DLTK tool:
Projects Code folders and code libraries Source modules Types, Functions, Variables Package and import declarations
What makes DLTK an exemplary tool is Type inference:
Without inferring types we can’t build good tools for dynamically-typed languages.
This is a must for:
Content Assistance (Code Completion) Code Navigation (Go To Declaration) Search (Search For References, Read/Write Access, etc) Code Analysis Refactoring
Beyond editing! DLTK a powerhouse of features:
Launch and Debug
Environment configuration based on different interpreter types and their installations Full featured Eclipse-based debugger for all scripting languages provided Remote debugging and launching Debug based on open DBGp protocol Interactive Console Common console protocol Remote console Code completion & assistance All standard features
Views
Script Explorer Outline Call Hierarchies Navigation (packages, types, functions)
Type Hierarchy Quick Outline / Quick Hierarchy
More UI
Project properties Wizards Preference pages Search UI
Leveraging DLTK:
There are three ways, a custom language editor can contribute language-specific things to DLTK :
Language toolkit – parsers, checkers, selection, completion engines, etc. UI language toolkit – label providers, and so on. Other extension points, like documentation providers, view filters, etc.
Creating core plugin:
After creating project, we need to setup dependencies for:
org.eclipse.dltk.core – core of DLTK. org.eclipse.core.filesystem – Eclipse file system. org.eclipse.core.resources – Eclipse Resources, Workspace.
We use dltk to develop an Editor for a hypothetical Business Modeling Language (BML).
Eclipse Nature : com.bml.core.nature
The nature class BMLNature extends ScriptNature class. ScriptNature class has all required stuff for nature management.
<extension id="nature"
point="org.eclipse.core.resources.natures">
<runtime>
<run
class="com.bml.language.core.BMLNature">
</run>
</runtime>
</extension>
BMLNature Class
package com.bml.language.core;
import org.eclipse.dltk.core.ScriptNature;
public class BMLNature extends ScriptNature{
public static final String NATURE_ID = BMLPlugin.PLUGIN_ID + ".nature";
}
** We can set up incremental builder for project.
Creating a simple Language Toolkit:
Most non-UI language-specific things are provided to DLTK by a language toolkit. It's most important thing in DLTK and language specific code interaction.
Language toolkit requires definition of Eclipse project nature,
<extension
point="org.eclipse.dltk.core.language">
<language
class="com.bml.language.core.BMLLanguageToolKit"
nature="com.bml.core.nature">
</language>
</extension>
Primary thing that language toolkit should provide is a source element parser.
This parser builds a source module content, so it could be accessible from DLTK (from script explorer for example).
Other required thing is a source module validation. Basically it could be just checking for a file name extension, in more complex cases it could be checks for file headers and more. UI stuff could be extended gradually as required.
BMLLanguageToolKit implements IDLTKLanguageToolkit, which could extend AbstractLanguageToolkit class for some basic implementation.
BML Language Parser:
Creating a Source Element Parser:
This parser is invoked when the editor is initialized to build the document model. A source element parser
extracts structural and reference information from a piece of source.
<extension point="org.eclipse.dltk.core.sourceElementParsers">
<parser class="com.bml.language.internal.parsers.BMLSourceElementParser"
nature="com.bml.core.nature"
priority="0">
</parser>
</extension>
BMLSourceElementParser which implements ISourceElementParser interface. This class is used to build model for source modules.
public class BMLSourceElementParser implements ISourceElementParser {
private ISourceElementRequestor fRequestor;
public ModuleDeclaration parseSourceModule(char[] contents,
ISourceModuleInfo astCashe, char[] filename) {
try {
sourceParser = (ISourceParser) DLTKLanguageManager
.getSourceParser(BMLNature.NATURE_ID);
} catch (CoreException e1) {
if (DLTKCore.DEBUG) {
e1.printStackTrace();
}
return null;
}
ModuleDeclaration moduleDeclaration = sourceParser.parse(null,
contents, null);
return moduleDeclaration;
}
public void setRequestor(ISourceElementRequestor requestor) {
this.fRequestor = requestor;
}
}
Extending Source Element parser to build correct model
For building model we provide ISourceElementRequestor interface which is passed to the SourceElementParser when building content of source module. It works as visitor. SourceElementParser should call methods to define model elements. i.e types, methods, fields, package declarations.
public ModuleDeclaration parseSourceModule(char[] contents,
ISourceModuleInfo astCashe, char[] filename) {
ISourceParser sourceParser = null;
try {
sourceParser = (ISourceParser) DLTKLanguageManager
.getSourceParser(BMLNature.NATURE_ID);
} catch (CoreException e1) {
if (DLTKCore.DEBUG) {
e1.printStackTrace();
}
return null;
}
ModuleDeclaration moduleDeclaration = sourceParser.parse(null,
contents, null);
moduleDeclaration.disableRebuild();
List statements = moduleDeclaration.getStatements();
try {
fRequestor.enterModule();
namespacesLevel.push("::");
buildModel(statements, TYPE_MODULE, "");
fRequestor.exitModule(contents.length);
} catch (Exception e) {
if (DLTKCore.DEBUG_PARSER) {
e.printStackTrace();
}
}
return moduleDeclaration;
}
Invoking Source Parser :
The Source Element Parser retrieves the Source Parser through DLTKLanguageManager and parses the contents to get the ModeleDeclarations.
ModuleDeclaration moduleDeclaration = sourceParser.parse(null, contents, null);
<extension
point="org.eclipse.dltk.core.sourceParsers">
<parser
class="com.bml.language.internal.parsers.BMLSourceParser"
nature="com.bml.core.nature"
priority="0">
</parser>
</extension>
1. BMLSourceParser passes the contents buffer from the bml file to the SimpleBMLParser.
which generates the BMLScript.
1.a …. script = SimpleBMLParser.parse(contents);
……… CodeScanner reads the file buffer till EOF.
public static BMLScript parse(String content) throws ParseException{
CodeScanner scanner = new CodeScanner(content);
BMLScript script = parse(scanner, false);
return script;
}
1.b. The SimpleParser will create a script and keep adding new BMLCommands till EOF.
1.c. Every BMLCommand in turn keeps gathering the BMLWords till an EOL is encountered.
2. Next, BMLSourceParser reads the BMLScript.
Checks for various Substitutions from every word of the command and respectively forms the expressions.
If this substitution is Quote then a BMLBlockExpression is created , Otherwise if it is just a command (add/delete operators) a BMLExecuteExpression is created.
Everything else is SimpleReference.
3. The newly generated expressions are added to the BML Statement.
4. Finally the statements are added to the BML module declaration which extends from ASTNode.
import org.eclipse.dltk.ast.declarations.ModuleDeclaration;
import com.tibco.cep.ui.rulelanguage.internal.parsers.BMLASTBuilder;
public class BMLModuleDeclaration extends ModuleDeclaration{
public BMLModuleDeclaration(int sourceLength) {
super(sourceLength, true);
}
protected void doRebuild() {
BMLASTBuilder.buildAST(this, getTypeList(), getFunctionList(), getVariablesList());
}
public void rebuildMethods() {
BMLASTBuilder.rebuildMethods(this);
}
}
Sample Code Example
Package com.bml.samples
BizModel {
Declare {
Customer bob=Customer.createCustomer(....);
Account savings = Account.createAccount(….);
}
Get {
String name = bob.getName();
}
Put {
bob.setAccount(savings);
}
}
Completion Engine:
BMLCompletionEngine
<extension point="org.eclipse.dltk.core.completionEngine">
<completionEngine
class="com.bml.language.core.codeassist.BMLCompletionEngine"
nature="com.bml.core.nature">
</completionEngine>
</extension>
BMLCompletionEngine sorts the list of proposals and appends it to CompletionProposals which is displayed on the Popup window.
1. BMLCompletionEngine calls parser to parse and compute the list of proposals.
public class BMLCompletionEngine extends ScriptCompletionEngine {
private BMLCompletionParser parser;
public BMLCompletionEngine() {
this.parser = new BMLCompletionParser();
}
}
BMLCompletionParser extends from BMLAssistParser which implements IassistParser.
public class BMLAssistParser implements IAssistParser {
public BMLAssistParser() {
try {
this.parser = DLTKLanguageManager
.getSourceParser(BMLNature.NATURE_ID);
} catch (CoreException e) {
if (DLTKCore.DEBUG) {
e.printStackTrace();
}
}
}
}
public class BMLCompletionParser extends BMLAssistParser {
/**
* Called when element could not be found.
*/
@Override
public void handleNotInElement(ASTNode node, int position) {
if (node instanceof ModuleDeclaration) {
ModuleDeclaration unit = (ModuleDeclaration) node;
List exprs = new ArrayList();
exprs.add(new SimpleReference(position, position, ""));
BMLEmptyCompleteStatement statement = new BMLEmptyCompleteStatement(exprs);
unit.addStatement(statement);
this.parseBlockStatements(statement, unit, position);
}
}
private static class BMLEmptyCompleteStatement extends BMLStatement {
public BMLEmptyCompleteStatement(List expressions) {
super(expressions);
}
}
public void parseBlockStatements(ASTNode node, ASTNode inNode, int position) {
//when completion proposals are found.
ASTNode nde = new CompletionOnKeywordOrFunction(
completionToken, completionNode, node, keywords);
throw new CompletionNodeFound(nde,
((ModuleDeclaration) inNode).scope, isMethod);
}
}
**
“node” is the ASTNode.
“completionNode” is the Expression from the ASTNode.
“completionToken” is the Name of the comletionnode.
“keywords” are the list of proposals.
Method handleNotInElement is called when an element is not found, and further the ASTNode could be resolved as completionNode and completionToken.
For Example:
“Pack” + Crtl*Space will be the completionNode and the word “Pack” is completionToken. The list of possible proposals would be “Package” for the completionToken “Pack”.
When the list of proposals are found, BMLCompletionParser forms a CompletionOnKeywordOrFunction, ASTNode and throws CompletionNodeFound Exception.
This RuntimeException is caught by BMLCompletionEngine such as,
public class BMLCompletionEngine extends ScriptCompletionEngine {
private BMLCompletionParser parser;
public BMLCompletionEngine() {
this.parser = new BMLCompletionParser();
}
public void complete(ISourceModule sourceModule, int completionPosition,
int pos) {
this.requestor.beginReporting();
boolean contextAccepted = false;
try {
this.fileName = sourceModule.getFileName();
this.actualCompletionPosition = completionPosition;
this.offset = pos;
ModuleDeclaration parsedUnit = (ModuleDeclaration) this.parser
.parse(sourceModule);
if (parsedUnit != null) {
if (DEBUG) {
System.out.println("COMPLETION - Diet AST :");
System.out.println(parsedUnit.toString());
}
try {
this.lookupEnvironment.buildTypeScope(parsedUnit, null);
if ((this.unitScope = parsedUnit.scope) != null) {
this.source = sourceModule.getSourceContents()
.toCharArray();
parseBlockStatements(parsedUnit,
this.actualCompletionPosition);
if (DEBUG) {
System.out.println("COMPLETION - AST :"); / System.out.println(parsedUnit.toString());
}
// parsedUnit.resolve();
}
} catch (CompletionNodeFound e) {
// completionNodeFound = true;
if (e.astNode != null) {
if (DEBUG) {
System.out.print("COMPLETION - Completion node : "); System.out.println(e.astNode.toString());
if (this.parser.getAssistNodeParent() != null) {
System.out.print("COMPLETION - Parent Node : "); System.out.println(this.parser.getAssistNodeParent());
}
}
// if null then we found a problem in the completion
// node
contextAccepted = complete(e.astNode, this.parser
.getAssistNodeParent(), e.scope,
e.insideTypeAnnotation);
}
}
}
if (this.noProposal && this.problem != null) {
if (!contextAccepted) {
contextAccepted = true;
CompletionContext context = new CompletionContext();
context.setOffset(completionPosition);
context.setTokenKind(CompletionContext.TOKEN_KIND_UNKNOWN);
this.requestor.acceptContext(context);
}
this.requestor.completionFailure(this.problem);
if (DEBUG) {
this.printDebug(this.problem);
}
}
} finally {
if (!contextAccepted) {
contextAccepted = true;
CompletionContext context = new CompletionContext();
context.setTokenKind(CompletionContext.TOKEN_KIND_UNKNOWN);
context.setOffset(completionPosition);
this.requestor.acceptContext(context);
}
this.requestor.endReporting();
}
}
private boolean complete(ASTNode astNode, ASTNode astNodeParent,
Scope scope, boolean insideTypeAnnotation) {
//sorts the list of proposals and appends it to CompletionProposals which gets displayed in a //Popup window.
}
}