designing extensible, domain-specific languages for...

Designing extensible, domain-specific languages for mathematical diagrams

Katherine Ye

with Keenan Crane Jonathan Aldrich

Josh Sunshine

Source: “An Algorithm to Generate Impossible Art?”, Danny Tarlow 1Off the Beaten Track ‘17

Motivation

2

sin ✓ =opposite

hypotenuse

cos ✓ =

adjacent

hypotenuse

3

7

Sometimes people do illustrate mathematics beautifully.

9

Source: Algebraic Topology Hatcher

illustrating groups (with their notations)

Source: Visual Group Theory 10

composing different types of diagrams

Problem:

Drawing good diagrams requires a tremendous amount of effort and expertise.

11

People have very powerful facilities for taking in information visually… and thinking spatially.

On the other hand, they do not have a very good built-in facility for inverse vision, that is, turning an internal spatial understanding back into a two-dimensional image.

Consequently, mathematicians usually have fewer and poorer figures in their papers and books than in their heads.

William Thurston Fields medalist

12





13





14

15

What if mathematicians had more and richer figures in their papers and books than in their heads?

Penrose

Source: “An Algorithm to Generate Impossible Art?”, Danny Tarlow 16

Source: https://sallyarjoon.wordpress.com/2013/04/01/enzymes/

Penrose

17

80% 20%

Casual users

18

“Let A and B be sets, and p be a point in A.”

… …A

Bp

A

B

p

A

B

p

10 seconds

19

80% 20%

Power users

20


… …A

Bp

A

B

p

A

B

p

2 hours

21

Existing solutions for diagramming

22

Factors in choosing a tool:

Programmatic, direct manipulation, domain knowledge

23

The design space

24

more programmatic

more direct

more domain knowledge

Writing a program for a diagram

25

26

K4 =⌦a, b | a2 = b2 = (ab)2 = e

↵

27

K4 =⌦a, b | a2 = b2 = (ab)2 = e

↵

%nodestylecirc/.style={circle,draw=black!60,fill=green!10,minimumsize=7mm}],

%nodes\node[circ](gen_bl){};\node[circ](gen_tl)[above=ofgen_bl]{};\node[circ](gen_br)[right=ofgen_bl]{};\node[circ](gen_tr)[above=ofgen_br,right=ofgen_tl]{};

%\node[circ](gen_bl);%\node[circ](gen_tl)[above=ofgen_bl];%{2}label%\node[circ](gen_tr)[above=ofgen_br,right=ofgen_tl];%\node[circ](gen_br)[right=ofgen_bl];

%drawlinesbetweennodes\draw[-][red,thick](gen_tl.south)--(gen_bl.north);\draw[-][blue,thick](gen_tl.east)--(gen_tr.west);\draw[-][blue,thick](gen_bl.east)--(gen_br.west);\draw[-][red,thick](gen_tr.south)--(gen_br.north);

TikZ

28

K4 =⌦a, b | a2 = b2 = (ab)2 = e

↵





TikZ

node style

29

K4 =⌦a, b | a2 = b2 = (ab)2 = e

↵





TikZ

node style

node positions

30

K4 =⌦a, b | a2 = b2 = (ab)2 = e

↵





TikZ

node style

node positions

line style + positions

31

K4 =⌦a, b | a2 = b2 = (ab)2 = e

↵

The design space

32

more programmatic

more direct


TikZ

33

Creating a diagram via a GUI

Source: Keenan Crane 34

curvature of a polyhedron

35


36


37




5 hours of painstaking clicking and dragging

…

39

40

…

no style reuse!

The design space

41

more programmatic

more direct


Illustrator

Burn it all down and write your own software or DSL?

42

Group Explorer KnotPlot

strid

BayesNet

JaxoDraw

43

Group Explorer KnotPlot

strid

BayesNet

JaxoDraw

44

The design space

45

more programmatic

more direct


Custom DSLs

Penrose

Source: “An Algorithm to Generate Impossible Art?”, Danny Tarlow 46


47


Notation

Set A

Set B

Point p

p 2 A

48


Penrose architecture

49



Substance View Style

50

SetASetBPointpInpA




51


Dimension2CoordinatesNone



52

SetASetBPointpInpA


ShapeSetCircleColorABlueColorBOrangeShapePointCircleLabelAllAuto



53

SetASetBPointpInpA




Intermediate representations

Sampling and optimization


54

SetASetBPointpInpA








… …A

Bp

A

B

p

A

B

p

55

SetASetBPointpInpA








… …A

Bp

A

B

p

A

B

p

56

SetASetBPointpInpA



The design space

57

more programmatic

more direct


Penrose

58

…plus extensibility!

The design space

59

more programmatic

more direct


Penrose

prodirect manipulation

Sketch-n-Sketch (Chugh et al, PLDI ’16)

An example of our vision

60

I’m demonstrating a mockup of a workflow that I hope to demo live in a few months.

61

We have a very simple prototype for set theory, which I won’t demo.

62

Domain: Set theory → Point-set topology

63

64

Phases: Implementing the DSL → using the DSL

Implementing the DSL

65

Declarations

dataSubDecl=DeclObjectdataObject=OSSet|OPPointdataSet=Set’StringSetTypedataSetType=Open|Closed|UnspecifieddataPt=Pt’String

Constraints

dataConstraint=SubsetStringString|PointInStringString

66

Implementing a set theory DSL

Declarations


Constraints


67


Declarations


Constraints


68


Declarations

dataSubDecl=DeclObjectdataObject=OSSet|OPPointdataSet=Set’StringSetTypedataSetType=Open|Closed|Clopen|UnspecifieddataPt=Pt’String

Constraints


69


70


Declarations


Constraints


Declarations


Constraints


71


Declarations


Constraints


72


SetASetBPointpInpA




73

Set theory DSL

Using the DSL

75

Open sets, closed sets, continuous maps

76

An open ball in R2

A

A=Or(p,Rn)

p

r

78

An open ball in R2

A

OpenBallA

79

An closed ball in R2

A

ClosedBallA

80

An open set in R2

OpenSetU

81

An open set in R2, simplified

OpenSetU

U

82

An closed set in R2

ClosedSetU now contains boundary

83

U

ClosedSetU

84

An closed set in R2, simplified

Continuous maps

85

86

Anti-example: discontinuous map

87


function “tears” domain

88


interval in function’s co-domain is open

89


inverse image of open interval is not open!

90

inverse image of open interval is not open!


91

Generalizing the intuition

92

Generalizing the intuition

Picture this!

SetASetBSetR^nSetR^mSubsetAR^nSubsetBR^mMapfABOpenSetUSubsetUBSetf^{-1}(U)Subsetf^{-1}(U)A

Substance

93


Substance

94

declarations


Substance

95

constraints

StyleAllAuto

Substance Style

96


A

Rn

98

A

Rn

B

99

A

Rn

B

Rm

100

A

Rn

B

Rm

f

101

A

Rn

B

U

Rm

f

102

A

f -1(U)

Rn

B

U

Rm

f

103

A

f -1(U)

Rn

B

U

Rm

f

104

StyleAllAutoShapeR^nSquareShapeR^mSquareColorR^nYellowColorR^mYellow

Substance Style

105


A

f -1(U)

Rn

B

U

Rm

f

106

A harder definition…

107

109

Picture this!

110

Picture this!

111

SetASetBSetR^nSetR^mSubsetAR^nSubsetBR^mMapfABOpenSetUSubsetUB

Substance Style

PointpInpAPointf(p)Inf(p)UOpenSetVSubsetVAInpVSetf(V)Subsetf(V)UInf(p)f(V)

underline denotes implicit in notation. f(V) might not be open but is depicted as such


A

Rn

B

Rm

112

A

Rn

B

Rm

f

113

A

Rn

B

Rm

p

f

114

A

Rn

B

Rm

p

f

U

115

A

Rn

B

Rm

p

f

U

116

f(p)

A

Rn

B

V

Rm

p

f

U

f(p)

117

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

118

119

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

How do the two definitions relate?

120

121

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

A

f -1(U)

Rn

B

U

Rm

f

Can prove (1) → (2) via diagram! (omitted)

122

123

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

(2)

Prodirect manipulation: center a point

124


Substance Style



A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

(2)

125

A

Rn

B

V

Rm

f

U

f(p)

f(V)

(2) p


Substance Style



126

A

Rn

B

V

Rm

f

U

f(p)

f(V)

(2)


Substance Style

p


PointpInpAPointf(p)Inf(p)UOpenSetVpSubsetVAInpVSetf(V)Subsetf(V)UInf(p)f(V)

infer code

changes!

127

Penrose takeaways

• Easily implement and extend DSLs • Easily reuse styles • Expertise encoded! Substantial

benefits accrue with each new diagram

• Just throw in notation; get a useful illustration

128

Penrose takeaways




129

Penrose takeaways




130

Penrose takeaways



• DSL users can just throw in notation; get a useful illustration

Language design challenges

131

Penrose provides an extensible visual semantics for mathematical notation.

132


… …A

Bp

A

B

p

A

B

p

133

Where are the semantics?

134

Notation

Set A

Set B

Point p

p 2 A


135

SetASetBPointpInpA





136

SetASetBPointpInpA





137

… …A

Bp

A

B

p

A

B

p

138

Another way to give semantics to mathematical notation…

Once formalized as a procedure, a mathematical idea becomes a tool that can be used directly to compute results.

139

Another way to give semantics to mathematical notation…

Modularity, compositionality

140

Source: Visual Group Theory 141

Dynamic Symbols in Illustrator

142

Interoperability

143

how to design the common IR / substrate / host language?

group theory sub-DSL

permutation sub-DSL

other sub-DSLs

144

1 2

3 4

1 2 3 4

1 2 3 4

1 2 3 4

1 2 3 4

Cayley diagram for K4 Permutation group

* 2

* 2

* 3 * 3

145

Cayley’s theorem

Extensibility

146


…for each component of Penrose




… …A

Bp

A

B

p

A

B

p

147

SetASetBPointpInpA








… …A

Bp

A

B

p

A

B

p

148

SetASetBPointpInpA








… …A

Bp

A

B

p

A

B

p

149

SetASetBPointpInpA








… …A

Bp

A

B

p

A

B

p

150

SetASetBPointpInpA



Our broader vision

151

Automatically parse and visualize mathematics

152

155

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

A

f -1(U)

Rn

B

U

Rm

f

Source: https://www.reddit.com/r/math/comments/5i845f/mathpix_a_phonecamera_powered_3d_grapher 156

https://www.reddit.com/r/math/comments/5i845f/mathpix_a_phonecamera_powered_3d_graphe

Next time you want to illustrate your paper or talk…

157

158

…reach for Penrose!

“Testimonials”

159

(from real people!)

I would certainly be an eager user of such a system.

The current state of the art for my workflow is a lot of copy-pasting and tweaking of TikZ code.

Edward Morehouse category theorist

160

This looks fabulous! (…)

I'd love to use it to rewrite Group Explorer for the web!

Nathan Carter author of Visual Group Theory

161

Thanks! Questions?

162

Gallery

163

164

Source: Visual Group Theory

illustrating a theorem by example

165


illustrating an algorithm

Source: https://arxiv.org/pdf/1610.06204.pdf 166

illustrating an algorithm

https://arxiv.org/pdf/1610.06204.pdf

167

Source: A first course in geometric topology

more geometric diagrams


geometric diagram described by notation

169

Source: A first course in geometric topology

illustrating maps (functions) between objects

170Source: Keenan Crane

illustrating maps (functions) between objects

Source: Algebraic Topology Hatcher 171

function composition

Source: “Understanding LSTM Networks,” Chris Olah 172

circuit-type diagrams for equations

Source: NIST 800-90A 173

circuit-type diagrams for equations

Source: Graphical Linear Algebra 174

math done via diagrams

175


sometimes we don’t know what we want to draw…

FAQ

176

How will you evaluate the effectiveness of Penrose?

How does Penrose compare to Asymptote, TikZ, Ipe, etc.?

How does Penrose compare to Mathematica, Wolfram Alpha, GeoGebra, graphing calculators, etc.?

What other domains of math might you target?

How will Penrose deal with continuous domains?

How much of the system have you actually implemented so far?

Do you plan to put Penrose on the web, or integrate it with TeX?

What rendering backend will you use?

Have you considered <related work>?

Can I use Penrose right now? No? Then when??

177

How much about mathematics does Penrose need to “know”?

Have you considered hooking it up to a system for formal reasoning, such as a theorem prover (e.g. Coq)?

What about programs that represent invalid, incorrect, or inconsistent diagrams?

How can Penrose enable counterfactual or nonconstructive reasoning?

How will you deal with quantifiers (e.g. for all, there exists) and their nesting and ordering?

How will you separate Substance from Style? I don’t find HTML/CSS to be very convincing.

Should all branches of mathematics be illustrated? Some seem more amenable to illustration than others.

How easy would it be for you to extend Penrose to do interactive diagrams, animated diagrams, sequential diagrams, algorithmic diagrams, or parameterized diagrams? What about illustrating theorems and proofs?

178

Bonus slides

179

Runtime and layout challenges

180






… …A

Bp

A

B

p

A

B

p

181

SetASetBPointpInpA



80% 20%

Casual users Power users

182

pinAAandBdonotintersectcurve2isnothomotopicto0Biscontainedinthepreimageoff

Constraints

183

layitoutlefttorightcentersets,butkeepthemawayfromeachotherpositionlabelsclosetotheirobjectbutfarfromothers’smartlinebreaks

Objectives

184

importExpertise

185

Proof by picture, continued

186

What’s the difference?

187

190

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

A

f -1(U)

Rn

B

U

Rm

f

191

A

f -1(U)

Rn

B

U

Rm

f

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

(1)

(2)

(1) -> (2)

A

Rn Rm

p

f

U

f(p)

192

A

f -1(U)

Rn

B

U

Rm

f

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

(1)

(2)

(1) -> (2)

A

Rn Rm

f

U

f(p)p

f -1(U)

193

A

f -1(U)

Rn

B

U

Rm

f

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

(1)

(2)

(1) -> (2)

A

Rn Rm

f

U

f(p)p

f -1(U)

V

open ball centered at p

recall: an open set in R2

OpenSetU

194

195

A

f -1(U)

Rn

B

U

Rm

f

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

(1)

(2)

(1) -> (2)

A

Rn Rm

f

U

p

f -1(U)

f(p)

Vf(V)

open ball centered at f(p)

196

A

f -1(U)

Rn

B

U

Rm

f

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

(1)

(2)

(1) -> (2)

A

Rn Rm

f

U

p

f -1(U)

f(p)

Vf(V)

open ball centered at f(p)

197

A

f -1(U)

Rn

B

U

Rm

f

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

(1)

(2)

(1) -> (2)

A

Rn Rm

f

U

p

f -1(U)

f(p)

Vf(V)

not centered!

198

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

(2)

Prodirect manipulation: center the points

199

A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

(2)

Prodirect manipulation: center the points

200


Substance Style



A

Rn

B

V

Rm

p

f

U

f(p)

f(V)

(2)

201

A

Rn

B

V

Rm

f

U

f(p)

f(V)

(2)


Substance Style

p


PointpInpAPointf(p)Inf(p)UOpenSetVpSubsetVAInpVSetf(V)Subsetf(V)UInf(p)f(V)

infer code

changes!

A

f -1(U)

Rn

B

U

Rm

f

202

animation, interactivity, fuzzing

A

f -1(U)

Rn

B

U

Rm

f

203

animation, interactivity, fuzzing

definition holds for the “edge case” of empty sets

designing extensible, domain-specific languages for...

Documents