The End of the world as we know it?

AKA your last NullPointerException $1B bugs!

Michael @Vorburger.chEclipseCon Europe 2016, October 25, 2016

EclipseCon NullPointerException2

Agenda

1. Topic Intro

2. “How to” (incl. mvn build & IDE config)

3. Some experiences from ongoing work

4. Call to community action++

EclipseCon NullPointerException3

null analysisis not at all by Red Hat!

AFAIK contributions mostly by Stephan Herrmann; as volunteer! #ACK

is a Linux Foundation hosted platform, leading the future of open Software Defined Networking (SDN), based on OSGi (Karaf), backed by a strong multi vendor community with 100s of Devs. Red Hat contributes to ODL, for which I’ve started to use Eclipse null analysis @work, and extended some tooling, and will share some experiences from #dayjob.

BTW: Red Hat is a great company to work

for…

redhat.com/jobs#LifeAtRedHat

EclipseCon NullPointerException4

History

C. A. R. Hoare (Mr. QuickSort et al) jokingly apologized in 2009 for

“adding null reference to ALGOL [=> C => Java], which probably caused

a billion dollars of pain and damage in the last 40 years” (AKA $1B

mistake)

NB: There actually are alt. (JVM) langs with types systems without any

null at all. (Try, for fun.) Meanwhile in the real world, Java has null - and

that ain’t never gonna change anymore…. but still not all hope is lost! ;-)

EclipseCon NullPointerException5

Why NPE Blog Post most viewed?

EclipseCon NullPointerException6

What?Static (compile-time) code analysis re. po. null-ness of method

parameters, local variables, fields.

Why?Aim to prove that your

code is null safe, so NullPointerException

will never occur anymore.

How?Let’s see together...

EclipseCon NullPointerException7

Get started

EclipseCon NullPointerException

<dependency> <groupId>org.eclipse.jdt</groupId> <artifactId>org.eclipse.jdt.annotation</artifactId> <version>2.0.0</version></dependency>

8

@ which?

Let’s get back to more about this (alternatives) in a moment…

EclipseCon NullPointerException

package-info.java, in EACH package, src/main before /test:

@org.eclipse.jdt.annotation.NonNullByDefaultpackage yourpackage;

9

@NonNullByDefault (Recommendation)Activating null analysis via Project Properties alone won’t do much yet.

You’ll need to express what you want analysed with Java annotations.

And occasionally, where you must, use @Nullable in (existing..) code.

EclipseCon NullPointerException10

EclipseCon NullPointerException11

Enjoy your extended coffee break… ;-)

init() close()

Done!Shortest presentation ever.

EclipseCon NullPointerException12

Done?

1. How will a (Maven) build enforce this, same as IDE?

2. How will each project get configured for this in-IDE?

3. Libraries, huh? (#existingcode #nogreenfield)

3.1 Library EEA & the build...

EclipseCon NullPointerException13

This was the most useful I was able to find, and (significantly...) extended and built upon:

● https://github.com/sylvainlaurent/eclipse-external-annotations-m2e-plugin

● https://github.com/sylvainlaurent/null-pointer-analysis-examples

● https://github.com/sylvainlaurent/eclipse-external-annotations

Curious: Sylvain lives literally next door, and we have common friends - what are the odds??

#FLOSS

Research existing attempts to address this#ACK: Others have explored the same space before me - but I hadn’t found a full solution...

EclipseCon NullPointerException

<plugin> <artifactId>maven-compiler-plugin</artifactId> <version>3.5.1</version> <configuration> ... <compilerId>jdt</compilerId> <compilerArguments> <properties>.settings/org.eclipse.jdt.core.prefs</properties>

? ^ ${basedir}/ ? <dependencies> <dependency> <!-- of maven-compiler-plugin, not project --> <groupId>org.eclipse.tycho</groupId> <artifactId>tycho-compiler-jdt</artifactId>

14

Build

1. https://wiki.eclipse.org/Tycho/FAQ#Can_I_use_the_Tycho_compiler_support_in_non-OSGi_projects.2C_too.3F 2. https://github.com/jbosstools/m2e-jdt-compiler 3. https://issues.apache.org/jira/browse/MCOMPILER-123

EclipseCon NullPointerException

https://github.com/vorburger/eclipse-null-eea-augments illustrates 3. in full example project.

https://github.com/vorburger/eclipse-external-annotations-m2e-plugin/commits/individualEEAs is a M2E AbstractProjectConfigurator which copies JDT compilerArguments properties from artifact in compiler-plugin dependency, or compilerArguments properties, into IDE.(TBD ⇒ ? #helpwanted)

15

Project ConfigurationDepends on where you sit on having .settings/** in SCM… ;-) Basically you can either:

1. use workspace wide default preferences (recommended: Oomph!); NOK for gradually.. 2. save .settings/*jdt.core.prefs project preferences properties per project in SCM 3. .settings/ in .gitignore & org.eclipse.jdt.core.prefs in Maven artifact, and:

a. use maven-dependency-plugin to unpack it into target/ for buildpoint maven-compiler-plugin to use that instead of .settings/

b. declare <dependency> and use M2E extension to set into each project (JDT API)

EclipseCon NullPointerException

java.util java.utilOptional.ofNullable(null) /* @Nullable */ V Map.get(Object key)

Null type mismatch: required ...'@NonNull String' but the providedvalue is null

16

The underlying problem is interfacing with existing library code which you cannot modify... or... can we?!

Libraries...You always use external 3rd-party libraries in all of your projects. Even if you think you don’t, you do - JDK! Look at 1st of many you’ll hit:

EclipseCon NullPointerException17

EclipseCon NullPointerException18

External *.eea annotation files “augment” (not in the Star Trek sense..) Eclipse Java Build Path entries with additional @NonNull VS @Nullablable information.

Useful when working with 3rd-party library JARs you cannot easily change - including the JDK RT.

Eclipse External null Annotations (EEA)

EclipseCon NullPointerException19

EclipseCon NullPointerException20

EclipseCon NullPointerException21

https://github.com/vorburger/eclipse-null-eea-augments (TBD ⇒ ? #helpwanted #449653)

Producing EEA JARs as Maven artifacts

slf4j-api-eea ch.vorburger.null:eea-parent

maven-resources-plugin config.

maven-compiler-plugin disabled

maven-surefire-plugin disabled

/pom.xml

/org/slf4j/LoggerFactory.eea

/eea-for-gav

NOT using Maven classifier - just naming convention of artifactId ending in *-eea (for filtering list of dependencies for performance in M2E plugin). Maven Classifiers don’t work great in M2E, and aren’t a fit here anyway - the EEA isn’t really an additional artifact for one (≉ GAV).

EclipseCon NullPointerException

<plugin> <artifactId>maven-compiler-plugin</artifactId> <configuration> <compilerArguments> <annotationpath>CLASSPATH</annotationpath>

<dependencies> <!-- of project, not maven-compiler-plugin --> <dependency> <groupId>ch.vorburger.null.eea</groupId> <artifactId>slf4j-api-eea</artifactId> <version>1.0.0-SNAPSHOT</version> <scope>provided</scope>

22

EEA & the build

1. Granularity not identical to in-IDE, but this works just fine… #pragmatic2. Not REALLY needed on CP, but this approach avoids rebuilding infra. for DL, dist, etc. #itworks

EclipseCon NullPointerException

External Annotations & M2E Maven in IDE

23

I’ve extended M2E Project Configurator(same which sets Compiler Properties) toset External annotations for Maven Dependencies classpath container.

Problem: How to map JAR (or Project) with *.eea to Maven GAVs / JRE?I’ve used a file in JAR (eea-for-gav: org.slf4j:slf4j-api). It has N lines; useful e.g. for org.mockito:mockito-all & :mockito-core & “fat” J.

Supports: Open/Close (=> binary) project - standard Maven Workspace Dependency Resolution (TBD ⇒ ? #helpwanted #457792)

EclipseCon NullPointerException24

In an ideal world, we should all have a common repo of EEA artifacts where we share null annotations...

How-to new Git repo. @ eclipse.org? GitHub PRs how?Or simply a GitHub Org?? (Soft on IP CLA?)

I’m willing to kick something off… Is there anyone here who would be willing to contribute to this effort?

External Annotations repository #helpwantedEclipse.org Bug 449653 and Bug 457792

Some Experiences

&

Wrapping Up

EclipseCon NullPointerException26

How-to move existing product to be NullPointerException free?

1. Set up your product’s org.eclipse.jdt.core.prefs & IDE Setup infra etc.It has to “just work” (OOB)… for Maven-based projects, see prez. part 1! ;)

2. Iterate submitting small incremental changes, project by project, src/main before src/test; @NonNullByDefault package by package:Add package-info.java => lots red, adapt, some @Nullable, etc.

3. Contribute to (my) shared EEA library manually as required as you go

4. If push back, can start with IDE only, enforcement in build later(but do target build enforcement, else quickly regressions from others)

Process (ODL: 60+ .git * 10s .project * 10s .java)

EclipseCon NullPointerException27

1. FindBugs: Oldest, but (supposedly) more limited null analysis.

Still great for other static code analysis (just being ext. intro. in ODL)

2. Checker Framework: Interesting… actually. Extensive external

annotations for JDK (*.jaif); IDE integration (TBC; must have IMHO)

3. JetBrains.com IntelliJ IDEA: Some inference by bytecode analysis,

also external annotations (XML); but no Java 8 generics, and how-to

in build?

Alternatives?

EclipseCon NullPointerException

● org.eclipse.jdt.annotation■ Android’s, Lombok’s…

● javax.annotationfrom com.google.code.findbugs:annotation (NOT :jsr305 BTW...)

● org.jetbrains:annotations■ org.checkerframework

● com.sun.istack.internal (rt.jar)

28

They’re NOT all the same… JDT’s & Checker’s are null type annotations (Java 8 @Target TYPE_USE), for generics! FB/JSR-305 is NOT, others IDK. The semantics differ, too; FB’s @Nullable :( You could also create your own. But simplest to get started is to just use JDT’s.

Yeah, there are several “competing” annotations for (non) nullability…Accept it, pick one as primary for your project with your tool, move on! ;)

@ which @Nullable & @NonNull (NotNull)

EclipseCon NullPointerException29

Gotcha! (Quiz Time)

@Nullable File file() { return null;}

@Nullable java.io.File file() { return null;}

java.io.@Nullable File file() { return null;}

EclipseCon NullPointerException30

Gotcha! (Quiz Time)

@Nullable File file() { return null;}

@javax.annotation.Nullable java.io.File file() { return null;}

java.io.@org.eclipse.jdt.annotation.Nullable File file() { return null;}

https://bugs.eclipse.org/bugs/show_bug.cgi?id=506376

EclipseCon NullPointerException31

FindBugs annotations interop?

● Configure javax.annotation.Nullable & NonNull (used e.g. in Google Guava! and ODL...) as secondary annotations in “Annotations for Null Specifications” preference intro. Since Neon (4.6) M5, save EEA. I’m still

gaining experience with InterOp… diff. only evaluated when reading class files external to the project? That’s inconsistent… see Hello.java prob with fileMayBeNull2.

● Use Eclipse’s null analysis instead of FindBugs’, in both build and IDE (use FB for other code analysis; exclude)

EclipseCon NullPointerException32

As of 2016.10.12:

441 closed VS 135 open bugs in CDT with “[null] …” in summary.(As usual: Not all 135 are true “bugs”; some are unclear, some lingering old “discussions”.)

Sense of “not ideal solution, we had to compromise” in some areas (e.g. Bug #505828)

Filed / found during evaluation:

1. Bug #443146 Code Completion unnecessary @NonNull when @NonNullByDefault2. Bug #506375 Allow NonNullByDefault in external library annotation3. Bug #505828 Inherit stronger than @NonNullByDefault4. Bug #500885 NullPointerException at jdt.internal.compiler.lookup.BinaryTypeBinding5. Bug #489900 annotation 2.1.0 in Maven central

Status & Bugs

EclipseCon NullPointerException33

Annotations & @Override (with EEA…)

@Overridepublic boolean equals(Object obj)

Illegal redefinition of parameter obj,inherited method from Object does notconstrain this parameter

@Overridepublic boolean equals(@Nullable Object obj)

Can be worked around by adding @Nullable - at every use! (Bug #505828 …)

EclipseCon NullPointerException34

● ODL: (0) APPLY.. (1) OMG build on ecj… TBC! ;-) (2) @eclipse & @javax & (3) IntelliJ cohabitation? #me

● EEA: Repo with more contribs, releasing #me&you?Maybe conversion tool of Checker Framework JDK (*.jaif) metadata to EEA ?? Gen. EEA by automatic analysis?

● Plug-in Dependencies CP container? Gradle?? #you?

● Try it all out, contribute eea, provide feedback! #you

Future Ideas (YOU? #helpwanted)

Q & A

and YOUR feedback / experiences?

THANK YOU

plus.google.com/+RedHat

linkedin.com/company/red-hat

youtube.com/user/RedHatVideos

facebook.com/redhatinc

twitter.com/RedHatNews

EXTRA SLIDES

EclipseCon NullPointerException

@NonNullByDefault({}) // deactivate it :-(public class Bean {

private String name;

public String getName() { return name; }

public void setName(String name) { this.name = name; }}

39

Zoom: Java Beans - null by default?

EclipseCon NullPointerException40

Zoom: Java Beans - Collection properties

public class Bean {

private final List<String> names = new ArrayList<>();

public List<String> getNames() { return names; }

// no setNames()

}

EclipseCon NullPointerException

public class Bean {

private Optional<String> name = Optional.empty();

public Optional<String> getName() { return name; }

public void setName(String name) { this.name = Optional.of(name); }

// void unsetName() ?}

41

Zoom: Java Beans - Optional<> properties

public class Bean {

private @Nullable String name;

public Optional<String> getName() { return Optional.ofNullable(name); }

public void setName(String name) { this.name = name; }

// void unsetName() ?}

EclipseCon NullPointerException42

Zoom: Java Beans - mandatory properties Traditionally we don’t think about mandatory (required) bean properties.. if (some) clearly always are (beware..), use final and constructor or Builder pattern which throws Exception.

public class Bean {

private final String name;

public Bean(String name) { this.name = name; }

public String getName() { return name; }

}

public class Bean {

private final String name;

public static Builder builder() { …

public static class Builder { public Bean build() { …

}

EclipseCon NullPointerException43

Zoom: Java Beans - mandatory properties (II)If you would like static compile-time checking of mandatory bean properties instead of (checked or unchecked) exceptions at run-time from the Builder, you can… It’s a PITA to hand-write, but ideal for code generators! Recommendation: Use http://Immutables.org

For an example Builder of a Bean with some mandatory and some optional properties which enforces setting all required at compile time, in fixed order, by returning a particular (inner static) Build class for each (!) mandatory property, AKA “staged Builder”, follow https://github.com/google/auto/issues/370 (originally in Person.java in vorburger/eclipse-null-eea-augments)

EclipseCon NullPointerException44

Zoom: FieldsTODO huh, seriously?! WTF. This is a significant obstacle to migrating existing code! Options?

public int hashCode() { (...) + (field == null ? 0 : field.hashCode());

Potential null pointer access: this expression has a '@Nullable' type

Quick Fix “Extract to local variable (all occurrences)”

public int hashCode() { Type field2 = field; (...) + (field2 == null ? 0 : field2.hashCode());

PS: For hashCode() in particular, use Objects.hash(...) - but problem in general remains.

EclipseCon NullPointerException45

Zoom: Beans’ Fields: hashCode() & equals()

In Beans’ hashCode() and equals() implementations, instead of introducing field indirection, use Java 7’s Objects.hash(...) and equalsHelper (* by yours truly). Or, again, consider using some bean code generator, such as e.g. Google’s AutoValue or http://immutables.github.io, etc. (BTW https://github.com/google/guava/issues/2521)

public int hashCode() { Type field2 = field; (...) + (field2 == null ? 0 : field2.hashCode());

EclipseCon NullPointerException46

● https://www.lucidchart.com/techblog/2015/08/31/the-worst-mistake-of-computer-science/ ● https://wiki.eclipse.org/JDT_Core/Null_Analysis/Options ● ...

Recommended reading