the next generation mop, jochen theodorou, gr8conf 2013

The next generation MOP

Jochen "blackdrag" Theodorou

About blackdrag● Working on Groovy Core since about 2005 ● Almost as long as that, Tech Lead of Groovy

● Currently Employed at Pivotal

● Responsible for most of the technical side of Groovy

Email: blackdrag@gmx.org

About a new MOP

● discussions since 2005

● good-for-all solution always missing

● some ideas open to discussion

● and some cleanup duties

● we insist on you helping us

Why change

Why change● has many inconsistencies

● makes many optimizations impossible

● in theory powerful

● have to know implementation details

● Impossible to extend and spec resistent

● the API just plain sucks

Some Basics and History

The Meta Class● Each class has a meta class

● Saves all the dynamic and static properties/methods for Groovy

● Internal control flow involves exceptions

● Designed for invoking methods through the meta class

The Meta Class● early Groovy meta programming:

● MOP methods invokeMethod and get/setProperty● A custom meta class (foo.metaClass = x)

● later class based automated meta class lookup

● transformed into the meta class creation handle (used by ExpandoMetaClass#enableGlobally())

The Meta Class

● Basic principle: Each instance has a meta class

● More specified: Only every GroovyObject instance(later we changed that with a global map)

● Global registry specifying initial meta class on first use

The Meta Class // myMetaClass some custom metaclass

// meta class in registry differentdef x1 = new X()assert x1.metaClass != myMetaClassx1.metaClass = myMetaClassassert x1.metaClass == myMetaClassdef x2 = new X()assert x2.metaClass != x1.metaClassX.metaClass = myMetaClassdef phantom = new X()def x3 = new X()assert x3.metaClass == x1.metaClassassert x3.metaClass != x2.metaClass

The Meta Class

X.metaClass = x2.metaClassassert phantom.metaClass == ???

Adding Methods/Properties● Standard meta class: MetaClassImpl

● does not support modifications

● New meta class for this: ExpandoMetaClass

● enabled with ExpandoMetaClass.enableGlobally()

● not always equally behaving to MetaClassImpl

More MetaClasses● ProxyMetaClass (intercepting, decorating)● MixinMetaClass (mixins)● ClosureMetaClass (GeneratedClosure)● DelegatingMetaClass (base class)● OwnedMetaClass (related to mixins)● HandleMetaClass (related to mixins)

Plus your own custom meta class

DSL not consistentFoo.metaClass.bar = 1 //defines propertyFoo.metaClass.bar = {1} //defines method

to use a closure as property:

foo.metaClass.bar = nullfoo.bar = {1}

● only for the instance● get metaproperty and set initial value creator

Overriding Super Methodsclass A { def doIt(){two() + ' done.'}def two(){'two'}

} class B extends A {}B.metaClass.two = {'my new two!'}def b = new B() assert b.two() == 'my new two!'assert b.doIt() == 'two done.'

Overriding Super MethodsTo make it work:

class A implements GroovyInterceptable { def doIt(){two() + ' done.'}def two(){'two'}

} class B extends A {}B.metaClass.two = {'my new two!'}def b = new B() assert b.two() == 'my new two!'assert b.doIt() == 'my new two! done.'

Adding Super Methodsclass A { def doIt(){two() + ' done.'}def methodMissing(String name, args){'two'}

} class B extends A {}def b = new B()assert b.two() == 'two'assert b.doIt() == 'two done.'A.metaClass.two = {'my new two!'}assert b.two() == 'my new two!'assert b.doIt() == 'my new two! done.'

Super Methods OverloadDoes not:

class A { def doIt(x){two(x) + ' done.'}def two(x) {'two'}

} class B extends A {}def b = new B()assert b.two('1') == 'two'assert b.doIt('1') == 'two done.'A.metaClass.two = {String s->'my new two!'}assert b.two('1') == 'my new two!'assert b.doIt('1') == 'my new two! done.'

Private Multi Methodsclass A { def doIt(){two() + ' done.'}

} class B extends A {private two(){1}

}def b = new B()assert b.two() == 1assert b.doIt() == '1 done.'

Speaking of privateclass A { private foo=1def b={foo}

} class B extends A {}def b = new B()assert b.b() == 1 //fails

● Information loss thorugh Closure#getProperty

get/setMetaClass

● persistency framework needs to be aware

● transient works for Serialization

● what about other frameworks?

● seamless integration anyone?

Propertiesclass X extends

org.xml.sax.helpers.XMLFilterImpl {def foo

}

● XMLFilterImpl has a get/setProperty

● cannot do new X().foo = bar

● cannot do println new X().foo

invokeMethod

No such conflict known.... but!

● dynamic entry point from Java● as methodMissing ● with GroovyInterceptable (EMC too) as upfront

method

conflicting concepts

What to make better?

… besides fixing those problems

Optimization efforts Lesson:

Java7 with invokedynamic is much better suited for Groovy's dynamic method calls

Reaction:

make Java7 the default (backport); rewrite DefaultGroovyMethods to use indy; throw out a lot of old code

Optimization efforts Lesson:

Hotspot is not happy about invoking target methods in the meta class (mega morphic call sites)

Reaction:

The meta class only gives back something you can call and does not do the call itself.

Optimization efforts Lesson:

Synchronization, Locks, volatiles usages on each method call destroy multithread performance as well as hotspot optimizations. Most applications set up mc changes on startup.

Reaction:

metaclass goes immutable; adding methods creates new meta class; lazy thread update (user level synchronization required)

Hot Swapping Lesson:

Keeping internal state in the class is bad (see timestamp_xxx, timestamp, $callSiteArray)

Reaction:

Removal. CallSiteArray not needed anymore, the timestamps are kept track off by the loader, not in the class

Optimization efforts Lesson:

Garbage collecting and recreating meta classes is very bad.

Reaction:

Keep the base meta class around and reuse everything possible to make new meta classes as lean as possible

API Design Lesson:

Conventions are good, forcing them is bad (GroovyObject)

Reaction:

Don't implement GroovyObject by default anymore.

General Design Lesson:

Too many ways of doing the same thing is no good

Reaction:

Most probably only methodMissing/propertyMissing anymore but easy way to „register“ a method to do invokeMethod.

API Design Lesson:

Having multiple, not equivalent entry points is bad. (MetaClassImpl has 3 for methods, multiusage of invokeMethod, information loss through get/setProperty)

Reaction:

Clean up the API to have only one entry point (removal of MetaClass#invokeMethod)

Possibilities

Internal vs. ExternalInternal usage:class X { def methodMissing(String name, args) {1}}

External usage:class X {}X.metaClass.methodMissing = {1}

Combined:class X { static {this.metaClass.methodMissing = {1}}}

Dynamic Invoke from JavaBefore:

GroovyObject foo = ...;String s = (String)foo.invokeMethod(“bar“, new Object[]{});

After:

Object foo = ...;String s = MopInterface.invoke(foo, “bar“);

● helper class for dynamic interactions with Groovy● similiar for properties

Adding a method from JavaBefore:GroovyObject foo = ...;foo.getMetaClass().registerInstanceMethod(“foo“,

new Closure(null){public Object doCall(){1}};

After:Object foo = ...;MopInterface.addInstanceMethod(foo, “foo“,

new T() {public Object myFancyMethod(){1}});

● Doesn't have to be a Closure or MetaMethod● All declared methods used (lambdas too)

Limited Meta Class ChangesUse Case:

unrolling all changes a GroovyShell or Eval.me did

Before:

● tracking meta class mutations impossible● „unmutate“ the metaclass certainly is● can only to track newly set meta classes

Limited Meta Class ChangesUse Case:

I am a library developer and I don't want intrusive changes from user code to my library classes, changing the way my library is calling methods.

Before:

If the change is not effecting the library, then most probably because of a bug.

Realms

RealmsA realm here defines how the class in the realm

„sees“ the meta classes. Different classes can have different realms.

MyClass(y)

MyOtherClass(x)

x.foo()

using metaClass(x)

Realms

MyClass(y)

MyOtherClass(x)

x.foo()

using metaClass(x) from realm(MyClass)

<Realm defined by MyClass>

Realms

MyClass(y)

MyOtherClass(x)

y.bar()

using metaClass(y) from realm(MyOtherClass)

<Realm defined by MyOtherClass>

Realms

INIT

DEFAULT

realm(A_n)

realm(C_n)realm(A_1)

........

realm(B_1)

realm(C_1)........

realm(D_1)

RealmsINIT does not allow changing the meta class

INIT

DEFAULT

Non isolated

isolated

RealmsDefault gets all the unspecified changes

INIT

DEFAULT

Non isolated

isolated

RealmsIsolated realms don't get unspecific changes

INIT

DEFAULT

Non isolated

isolated

RealmsNon isolated realms can override meta classes in

default without impossing themINIT

DEFAULT

Non isolated

isolated

Isolated Realm@Realm(

marker=SomeMarkerClass, parentRealm=Realm.INIT)

class MyLibraryClass {def foo(){bar()}def bar(){1}

}MyLibraryClass.metaClass.bar = {2}assert new MyLibraryClass().foo() == 1

Non-Isolated Realm@Realm(marker=SomeMarkerClass)class MyLibraryClass {def foo(){bar()}def bar(){1}

}MyLibraryClass.metaClass.bar = {2}assert new MyLibraryClass().foo() == 2

Testing Realm

class X {private foo(){1}}class MyTest extends GroovyTestCase {

@Realm(marker=SomeMarkerClass, allowPrivate=true)

void testPrivateWorking() {def x = new X()assert x.foo() == 1

}void testPrivateNotWorking() {

def x = new X()shouldFail(MissingMethodException) {

x.foo()}

}}

Limited Meta Class ChangesBefore:

Change intrusive, visible to everyone

After:

def foo(){1}assert foo() == 1realm.withSubRealm {

this.metaClass.foo = {2}assert foo() == 2

}assert foo() == 1

Q/A?