Plugging into the Java CompilerÉamonn McManus <emcmanus@google.com>Christian Gruber <cgruber@google.com>

JavaOne 2014

Speaker introduction: Éamonn McManus

● At Google since 2011○ Gmail servers initially○ App Engine SDK now○ 20% time on Java Core Libraries (Guava)

● Formerly at Sun then Oracle, on the JDK team● Author of two annotation processors

○ AutoValue (based on an idea by Kevin Bourrillion)○ JavaFX Builder generator

JavaOne 2014

Speaker introduction: Christian Gruber

● At Google since 2009○ “Test Mercenary”○ Mobile testing and development infrastructure○ Java Core Librarian

■ focusing on Dependency Injection (guice, dagger), and testing● Formerly at Sun/JavaSoft (via Lighthouse) and Oracle Consulting● Co-author of

○ Dagger 1 and 2, dependency injection using annotation processors○ auto-common, tools for easing annotation processor development

● Created (with other Googlers) the Truth assertion/testing library● Open-source bigot

JavaOne 2014

Overview

● What is an annotation processor?● Example: AutoValue● Example: Dagger● Write your own annotation processor● Q&A

JavaOne 2014

Reminder: Annotations

@SuppressWarnings(“unchecked”)public class Foo { @SafeVarargs public void bar(Optional<String>... args) {...} public void baz(@Nullable String s) {...}}

JavaOne 2014

What is an annotation processor?

● A way to extend the Java compiler● Standardized by JSR 269 in Java 6

○ javax.annotation.processing, javax.lang.model○ supported by javac (JDK) and ecj (Eclipse)

● Analyze Java code being compiled● Maybe introduce new errors and warnings● Maybe generate new Java code

JavaOne 2014

Exam

ple:

AutoValue

JavaOne 2014

Value Types

● A value type is a class where:○ properties never change (immutable)○ instances with the same properties are interchangeable

● Immutability is good!○ easy to reason about○ thread-safe

● Many uses in Java, for example:○ returning more than one value from a method○ combining values for use in a map key or value

■ Map<Tuple3<String, Integer, Country>, Tuple2<Long, Long>> ?

JavaOne 2014

Value types (ideal simplicity)

public class Address { public final String streetAddress; public final int postCode;

public Address( String streetAddress, int postCode) { this.streetAddress = streetAddress; this.postCode = postCode; }}

JavaOne 2014

Accessors and validation

public class Address { private final String streetAddress; private final int postCode;

public Address(String streetAddress, int postCode) { this.streetAddress = Preconditions.checkNotNull(streetAddress); this.postCode = postCode; }

public String streetAddress() { return streetAddress; }

public int postCode() { return postCode; }}

JavaOne 2014

equals, hashCode, toStringpublic class Address { private final String streetAddress; private final int postCode;

public Address(String streetAddress, int postCode) { this.streetAddress = Preconditions.checkNotNull(streetAddress); this.postCode = postCode; }

public String streetAddress() { return streetAddress; }

public int postCode() { return postCode; }

@Override public boolean equals(Object o) { if (o instanceof Address) { Address that = (Address) o; return this.streetAddress.equals(that.streetAddress) && this.postCode == that.postCode; } else { return false; } }

@Override public int hashCode() { return Objects.hash(streetAddress, postCode); }

@Override public String toString() { return "Address{streetAddress=" + streetAddress + ", postCode=" + postCode + "}"; }}

JavaOne 2014

equals, hashCode, toStringpublic class Address { private final String streetAddress; private final int postCode;

public Address(String streetAddress, int postCode) { this.streetAddress = Preconditions.checkNotNull(streetAddress); this.postCode = postCode; }

public String streetAddress() { return streetAddress; }

public int postCode() { return postCode; }

@Override public boolean equals(Object o) { if (o instanceof Address) { Address that = (Address) o; return this.streetAddress.equals(that.streetAddress) && this.postCode == that.postCode; } else { return false; } }

@Override public int hashCode() { return Objects.hash(streetAddress, postCode); }

@Override public String toString() { return "Address{streetAddress=" + streetAddress + ", postCode=" + postCode + "}"; }}

postCode

JavaOne 2014

AutoValue to the rescue

@AutoValue public abstract class Address { public abstract String streetAddress(); public abstract int postCode();

public static Address create( String streetAddress, int postCode) { return new AutoValue_Address( streetAddress, postCode); }}

JavaOne 2014

AutoValue generates subclass

class AutoValue_Address extends Address { private final String streetAddress; private final int postCode; AutoValue_Address(String streetAddress, int postCode) { ...check streetAddress not null... ...assign fields... } @Override public String streetAddress() {...} @Override public int postCode() {...} @Override public boolean equals(Object o) {...} @Override public int hashCode() {...} @Override public String toString() {...}}

JavaOne 2014

Foo.java

Bar.java

Address.java

@AutoValue ⇒

@AutoValue

AutoValue_Address.java

Annotation processing (1)

generatecompileAutoValueProcessor

JavaOne 2014

compile No furtherannotations

AutoValue_Address.java

Annotation processing (2)

JavaOne 2014

AutoValue_Address.java

generatecode

Foo.class

Bar.class

Address.class

AutoValue_Address.class

Annotation processing (3)

Address.java

Bar.java

Foo.java

JavaOne 2014

Demo

JavaOne 2014

Exam

ple:

Dagger

JavaOne 2014

Dagger

● Dependency-injection framework using JSR 330● Directed Acyclic Graph... of classes and their dependencies.● Dagger 1.x

○ Created by Googlers, and ex-Googlers at Square■ Christian Gruber, Jesse Wilson, Jake Wharton, and others...

○ Open source with contributions from Square, Google and others○ Source code generation and compile-time analysis

● Dagger 2.x○ 100% compile-time, via annotation processing and generated sources○ originated at Google, with design oversight by Dagger 1 contributors.

■ Concept by Greg Kick and Christian Gruber, impl mainly by Greg Kick

JavaOne 2014

Dependency Injection

● Having a class know how to obtain its collaborators is fragile○ Hard to change implementation○ Hard to unit-test

● Pattern described by Martin Fowler:○ http://www.martinfowler.com/articles/injection.html

● Early examples:○ Spring, PicoContainer, Apache HiveMind/Tapestry

● Later examples:○ Guice, CDI, Dagger

JavaOne 2014

Static dependencies crystallized in constructors

public class MailServer { // ... public MailServer() { this.messageStore = new MessageStoreImpl(); this.userService = new UserServiceImpl(); // ... }}

● Cannot test MailServer without invoking MessageStoreImpl, etc.● Cannot swap in alternate messaging, auth, etc.● Shared collaborators (singletons) end up requiring global statics

JavaOne 2014

Instead, declare the dependencies, and pass them in...

public class MailServer { // ...

public MailServer( MessageStore messageStore, UserService userService) { this.messageStore = messageStore; this.userService = userService; // ... }}

JavaOne 2014

Automation from annotation signals - no writing the wiring

public class MailServer { // ...

@Inject public MailServer( MessageStore messageStore, UserService userService) { this.messageStore = messageStore; this.userService = userService; // ... }}

JavaOne 2014

Explicit configuration is defined using annotations...

@Module public class MailServerModule { // Binding an implementation to an interface @Provides MessageStore store(MessageStoreImpl impl) { // MessageStoreImpl itself has @Inject signals. return impl; } // Adapting non-DI-friendly code @Provides UserService userService(DBConnection conn) { // Type is not in our control and no @Inject signals. return new UserServiceImpl.create(conn); }}

JavaOne 2014

Access to the graph defined by @Component

@Component public interface Services { MailServer mailServer(); NotificationServer notificationServer();}

● Annotations provide the signals● Graph analysis begins at annotated interfaces● All the “wiring” or “glue” code for the graph is generated● @Component interfaces’ implementation generated with a builder

for configuration● Plain old java code

JavaOne 2014

Explicit configuration is defined using annotations...

@Module public class MailServerModule { // Binding an implementation to an interface @Provides MessageStore store(MessageStoreImpl impl) { // MessageStoreImpl itself has @Inject signals. return impl; } // Adapting non-DI-friendly code @Provides UserService userService(DBConnection conn) { // Type is not in our control and no @Inject signals. return new UserServiceImpl.create(conn); }}

JavaOne 2014

Why Annotation Processors for D-I?

● Guice, Spring, etc. work well, so why processors and code-gen?

● Performance○ Reflection is expensive in some environments, such as Android○ Graph Validation is work - can affect startup times

● Developer productivity:○ Errors at compile-time vs. load time or even later improves velocity○ Generated code less "magical" and can be seen and reasoned about

● Design○ Knowing the structure of the code lets us build cleaner approaches

than the "magic map" of Injector/Container frameworks.

JavaOne 2014

Demo

Note: Dagger 2 is pre-release, and has some issues, include IDE integration issues

JavaOne 2014

Write

Your

Own

JavaOne 2014

What processors can and cannot see

● Processors can see the structure of your code:○ Class names, inheritance, generics○ Method names, parameter types, return types, generics○ Field names, types, compile-time constant values

● Processors cannot see the contents of the code○ Static initialization blocks○ Method bodies○ Initializer expressions

JavaOne 2014

What processors can and cannot do

● Processors can do quite a few things, such as○ generate new Java source code to be compiled○ generate other files (XML, META-INF/services, arbitrary text)○ perform analysis and emit warnings and errors○ associate the errors with specific source elements

● Processors cannot, however○ modify the code of existing classes

■ including source they generate once written○ introduce new fields or methods into existing classes○ introduce new nested classes

JavaOne 2014

"Annotation" processors

● Annotation processors are usually associated with annotations, obviously

● But, you can also write a processor that analyzes all input classes, whether annotated or not○ Return Collections.singleton("*") from getSupportedAnnotationTypes()

JavaOne 2014

Defining an annotation

import java.lang.annotation.*;

@Retention(RetentionPolicy.SOURCE)@Target(ElementType.TYPE)// @Documentedpublic @interface MyAnnotation { String value() default "";}

// @MyAnnotation class Foo {...}// @MyAnnotation("bar") interface Baz {...}// @MyAnnotation(value = "buh") enum Wibble {...}

JavaOne 2014

Outline of a processor (1)

import javax.annotation.processing.*;import javax.lang.model.*;

@AutoService(Processor. class)public class MyProcessor extends AbstractProcessor { @Override public Set<String> getSupportedAnnotationTypes() { return ImmutableSet.of(MyAnnotation. class.getName()); } @Override public SourceVersion getSupportedSourceVersion() { return SourceVersion.latestSupported(); } ...}

JavaOne 2014

Outline of a processor (2)

import javax.annotation.processing.*;import javax.lang.model.*;

@AutoService(Processor. class)public class MyProcessor extends AbstractProcessor { ... @Override public boolean process( Set<? extends TypeElement> annotations, RoundEnvironment roundEnv) { Collection<? extends Element> annotatedElements = roundEnv.getElementsAnnotatedWith(MyAnnotation. class); handle(annotatedElements); return false; }}

JavaOne 2014

Claiming annotations

● An annotation processor can say that it "supports" one or more annotations (getSupportedAnnotationTypes)

● Its process method will be called only for program elements where those annotations appear

● If it returns true, it has "claimed" the annotations and a later processor that also supports them will not be called

● We recommend never claiming an annotation○ You don't know what that later processor is or whether it should run

JavaOne 2014

API tips: Types and Elements

● A lot of useful functionality is contained in javax.lang.model.util.{Types,Elements}

● If you can't figure out how to do something, check these interfaces to see if they hold the solution

● Get an instance of either in your process method or any method it calls, via the inherited processingEnv field:@Override public boolean process(...) { Types typeUtils = processingEnv.getTypeUtils(); Elements elementUtils = processingEnv.getElementUtils();

JavaOne 2014

API Tips: ElementFilter

@Override public boolean process( Set<? extends TypeElement> annotations, RoundEnvironment roundEnv) { Collection<? extends Element> annotatedElements = roundEnv.getElementsAnnotatedWith(MyAnnotation. class);

List<ExecutableElement> annotatedMethods = ElementFilter.methodsIn(annotatedElements); handle(annotatedMethods); return false; }

JavaOne 2014

API tips: TypeMirror and TypeElement

● The distinction between TypeElement and TypeMirror is subtle● In practice, use whichever one the API gives you, or convert to the

other if the method you need is there● TypeMirror → TypeElement:

Types typeUtils = processingEnv.getTypeUtils();TypeElement typeElement = (TypeElement) typeUtils.asElement(typeMirror);

● TypeElement → TypeMirror:TypeMirror typeMirror = typeElement.asType();

orDeclaredType typeMirror = typeUtils.getDeclaredType(typeElement);

JavaOne 2014

API tips: Upstream errors can cause processor failure

● Processed code may be missing literal elements, be missing imports, be in a broken state, etc.○ Very strange results including core javac types in ClassCastExceptions or

NullPointerExceptions deep within the compiler.○ Upstream compile errors are masked by these exceptions.

● Auto-common provides SuperficialValidation.validateElements(...) ○ Simple sanity check for each Element and contents○ If validateElements() returns false, two options:

■ skip processing supplied elements (if done in a loop)■ return from the processor immediately

JavaOne 2014

API tips: Gotchas

● TypeMirror.equals() is not a reliable comparison. Should use:○ Types.isSameType(mirror1, mirror2) (if available)○ MoreTypes.equal(mirror1, mirror2) (if in a static context)

● TypeMirror instanceof checks are generally incorrect○ MoreTypes.asDeclared(typeMirror) converts correctly or throws IAE○ MoreTypes.asArray(...)... etc. - converters for all TypeMirror kinds.

JavaOne 2014

API tips: Google’s auto-common utilities

● MoreTypes and MoreElements○ Wrappers for equivalence○ Static methods for conversion and comparison

● SuperficialValidation○ For simple sanity checking of elements before processing

● AnnotationMirrors and AnnotationValues○ convenience methods and Equivalence wrappers○ static utilities to handle default values

● Types/Elements have useful methods, but are instances. Auto common utilities provides many static utility method equivalents.

Note: AnnotationValues and AnnotationMirrors pending extraction from Dagger 2.

JavaOne 2014

Generating code: JavaWriter

● Build up your class programmatically.● Refer to elements without using “stringly-typed” references.● Write to any Appendable.

JavaWriter javaWriter = JavaWriter.inPackage("some.package");ClassWriter klass = javaWriter.addClass("SomeType");VariableWriter field = klass.addField(String.class, "foo");field.addModifiers(PRIVATE, FINAL);ConstructorWriter constructor = klass.addConstructor();VariableWriter p0 = constructor.addParameter(Key.class, "key");constructor.body() .addSnippet("this.%s = %s.getFormat();", field.name(), p0.name());javaWriter.write(appendable);

JavaOne 2014

Generating code: JavaWriter

● Get clean, organized and readable code out.

package some.package;

import java.security.Key;

class SomeType { private final String foo;

SomeType(Key key) { this.foo = key.getFormat(); }}

Note: JavaWriter 3 is pending review with Square and extraction from Dagger2. JavaWriter 2 is available.

JavaOne 2014

Generating code: JavaWriter

● Pros:○ Programmatic creation of code○ Reduce errors with cross referencing○ Automatic handling of imports and type shortening○ Very useful in loops where each loop touches many parts of the code○ Some built-in validation based on code structure

■ some erroneous things are simply impossible to write● Cons:

○ Code that generates code can look quite different in shape than output

JavaOne 2014

#foreach ($p in $props) @Override ${p.access}${p.type} ${p}() { #if ($p.kind == "ARRAY") #if ($p.nullable) return $p == null ? null : ${p}.clone(); #else return ${p}.clone(); #end #else return $p; #end }#end

Generating code: templates

@Override public int[] postCodes() { return postCodes == null ? null : postCodes.clone(); }

JavaOne 2014

Generating code: templates

● Apache Velocity is a good choice of template engine○ Supported in all major IDEs (even Emacs)○ Directives clearly distinguishable from Java code snippets

● In AutoValue we do a post-processing step to remove superfluous spaces and blank lines, just so the generated code looks nicer

JavaOne 2014

Testing Processors - Unit testing environment

● Unit testing still requires javac environment, for Types/Elements● Compile-testing has a CompilationRule suitable for JUnit4

import com.google.testing.compile.CompilationRule;

@RunWith(JUnit4.class)public class SomeProcessorTest { @Rule public final CompilationRule compilationRule = new CompilationRule();

private Elements elements; private Types types;

@Before public void setUp() { this.elements = compilationRule.getElements(); this.types = compilationRule.getTypes(); }

JavaOne 2014

Testing Processors - Failing integration tests

● Need to test failing compiles without failing the outer build● Compile-testing has assertions based on the Truth library● These assertions can variously:

○ assert about javac runs success or failure○ be configured to run arbitrary Processor instances○ run against source from files or strings○ execute and store results in-memory○ assert about errors with specific message contents and locations○ assert against contents of generated files○ assert against contents of generated java source, comparing AST

JavaOne 2014

Testing Processors - Failing integration test assertions

JavaFileObject file0 = JavaFileObjects.forSourceLines("test.Foo", "package test", "class Foo {"; ... );

JavaFileObject file1 = JavaFileObjects.forResource("expected/SomeFile.java");

assert_().about(JavaSourcesSubject.javaSources()) .that(ImmutableSet.of(file0, file1)) .processedWith(new BlahProcessor()) .failsToCompile() .withErrorContaining"Invalid use of Annotation @Blah") .in(javaFileObject).onLine(7);

JavaOne 2014

Testing Processors - Succeeding integration tests

● For successful compilation, two approaches:○ Use compile-testing to compare against a golden file, or○ Functionally test that the generated code behaves as it should

● Golden file tests:○ Brittle - small changes can require a lot of change in test files/code○ Useful to demonstrate expected output

● Functional tests:○ Exercise more code paths and use-cases with less verbosity

JavaOne 2014

Testing Processors - Overall strategy

● Use Functional tests to cover the bulk of use-cases

● Use CompilationRule to unit-test the internals of your Processor

● Use compile-testing assertions to test expected errors

● Use compile-testing assertions to test a small number of golden expectation files, for illustration of processor output

JavaOne 2014

Summary

JavaOne 2014

Summary

● Annotation processors are a powerful way to plug in to javac● An ecosystem is evolving to ease writing custom processors

○ JavaWriter, auto-common, compile-testing, @AutoService● Several useful annotation processors exist presently

○ AutoValue, Dagger● Lots of advantages to annotation processors

○ Early error-checking○ Performance improvements○ Viewable generated source code

JavaOne 2014

Annotation Processor Resources

● Check out existing annotation processors:○ Google Auto: https://github.com/google/auto○ Dagger 1: https://github.com/square/dagger○ Dagger 2: https://github.com/google/dagger

● Projects that may help:○ JavaWriter: https://github.com/square/javawriter○ CompileTesting: https://github.com/google/compile-testing○ Google Auto common utilities: https://github.

com/google/auto/tree/master/common

JavaOne 2014

Q&A

JavaOne 2014

Manual IDE configuration

● Make a jar file with your processor and all of its dependencies● Ensure it has META-INF/services/javax.annotation.processing.

Processor○ @AutoService(Processor.class) is the easiest way

● NetBeans:○ Project Properties > Libraries > Processor > Add JAR/Folder

● Eclipse:○ Properties > Java Compiler > Annotation Processing

■ Enable Annotation Processing + Factory Path