kristian kersting, associate professor for computer science, tu dortmund university, germany at...
TRANSCRIPT
AIS Template
Declarative Programming for Statistical ML
Kristian Kersting
Martiin MladenovTUD
Babak AhmadiPicoEgo
Amir GlobersonHUJI
Martin GroheRWTH
Sriraam NatarajanU. IndianaLeonard KleinhansTUD
Danny HeinrichTUDand many more
Pavel TokmakovINRIA Grenoble
Is there a -01 flag for Statistical ML?
Kristian Kersting
Martiin MladenovTUD
Babak AhmadiPicoEgo
Amir GlobersonHUJI
Martin GroheRWTH
Sriraam NatarajanU. IndianaLeonard KleinhansTUD
Danny HeinrichTUDand many more
Pavel TokmakovINRIA Grenoble
Kristian Kersting - Declarative Programming for Statistical ML
There is an arms race to deeply understand dataKristian Kersting - Declarative Programming for Statistical ML
Kristian Kersting - Declarative Programming for Statistical ML Take your spreadsheet
FeaturesObjects
5
Latent Dirichlet Allocation
Matrix Factorization
FeaturesObjects and apply some ML
Gaussian Processes
Decision Trees/Boosting
Autoencoder/Deep Learningand many more Kristian Kersting - Declarative Programming for Statistical ML Support Vector Machines
6
Is it really that simple?Kristian Kersting - Declarative Programming for Statistical ML
What is the probability that the first card of a randomly shuffled deck with 52 cards is an Ace?
Guy van den Broeck UCLAA simple example
A simple example
Guy van den Broeck UCLAcard(1,d2)card(1,d3)card(1,pAce)card(52,d2)card(52,d3)card(52,pAce)
Guy van den Broeck UCLA
card(1,d2)card(1,d3)card(1,pAce)card(52,d2)card(52,d3)card(52,pAce)A simple example
Guy van den Broeck UCLANo independencies. Fully connected. 22704 states card(1,d2)card(1,d3)card(1,pAce)card(52,d2)card(52,d3)card(52,pAce)A simple exampleKristian Kersting - Declarative Programming for Statistical ML
Guy van den Broeck UCLAA machine will not solve the problemcard(1,d2)card(1,d3)card(1,pAce)card(52,d2)card(52,d3)card(52,pAce)A simple example
Positions and cards are exchangable but the machine is not aware of these symmetriesLets use a high-level language e.g.Markov Logic Networks (MLNs)w1:p,x,y: card(P,X),card(P,Y)x=yw2:c,x,y: card(X,C),card(Y,C)x=yand symmetry- and language-aware infernceFaster modellingFaster inference and learningWhat are we missing ?
Kristian Kersting - Declarative Programming for Statistical ML
Symmetry-Aware Message Passing
Compress the modelRun message passing inference on the smaller model[Singla, Domingos AAAI08; Kersting, Ahmadi, Natarajan UAI09; Ahmadi, Kersting, Mladenov, Natarajan MLJ13;Mladenov, Globerson, Kersting AISTATS `14, UAI `14; Mladenov, Kersting UAI15; ...]Big modelsmall model
Kristian Kersting - Declarative Programming for Statistical ML
De Raedt, Kersting, Natarajan, Poole Statistical Relational Artificial Intelligence,2016 the study and design of intelligent agents that act in noisy worlds composed of objects and relations among the objectsStatistical Relational AIScalingUncertaintyLogicGraphsTreesMining AndLearning [Getoor, Taskar MIT Press 07; De Raedt, Frasconi, Kersting, Muggleton, LNCS08; Domingos, Lowd Morgan Claypool 09; Natarajan, Kersting, Khot, Shavlik Springer Brief15; Russell CACM 58(7): 88-97 15]
15
But wait a minute! We want to use some ML, not just graphical models!Kristian Kersting - Declarative Programming for Statistical ML
Latent Dirichlet Allocation
Matrix Factorization
Gaussian Processes
Decision Trees/Boosting
Autoencoder/Deep Learningand many more Support Vector Machines
Lets say we want to classify publications that cite each other
This is a quadratic program. If you replace l2- by l1-,l-norm you get a linear program
Standard ML approach: Support Vector MachinesKristian Kersting - Declarative Programming for Statistical ML [Vapnik 79; Bennett99; Mangasarian99; Zhou, Zhang, Jiao02, ... ]
Write down the problem in paper form. The machine then compiles automatically into algebraic solver form.Statistical Machine Learning via Declarative Programming[Kersting, Mladenov, Tokmakov AIJ15, Mladenov, Heinrich, Kleinhans, Gonsio, Kersting DeLBP16]Logically parameterized variable (set of ground variables)Logically parameterized constraintLogically parameterized objectiveData stored externally
Program1Data1Program2Data2Program3Data3...Kristian Kersting - Declarative Programming for Statistical ML [Kersting, Mladenov, Tokmakov AIJ15, Mladenov, Heinrich, Kleinhans, Gonsio, Kersting DeLBP16]Captures the essence of a problem and can be reused for several problems
MP1DeclarativeProgramMP2MPnData1Data2Datan...Kristian Kersting - Declarative Programming for Statistical ML [Kersting, Mladenov, Tokmakov AIJ15, Mladenov, Heinrich, Kleinhans, Gonsio, Kersting DeLBP16]Captures the essence of a problem and can be reused for several problems
But wait, publications are citing each other. OMG, I have to use graph kernels!REALLY?
Simply program some additional constraints[Kersting, Mladenov, Tokmakov AIJ15, Mladenov, Heinrich, Kleinhans, Gonsio, Kersting DeLBP16]
Kristian Kersting - Declarative Programming for Statistical ML
http://www-ai.cs.uni-dortmund.de/weblab/static/RLP/html/
Kristian Kersting - Declarative Programming for Statistical ML Loops and relations get interwined, and models can refer to each otherDBMS InterfaceUsing a probabilistic programming language we can even get stochastic relational mathematical programs
Finally, the -O1 flag[Kersting, Mladenov, Tokmakov AIJ 2015, Mladenov, Kleinhans, Kersting 2016]Kristian Kersting - Declarative Programming for Statistical ML
Reduce the QP via symmetries Run any solver on the reduced QP
and the -02 flagKristian Kersting - Declarative Programming for Statistical ML
Algebraic Decision Diagrams
Formulae parse treesMatrix Free Optimization()+= Optimization with 60 Millions of non-zeros with 12 minutes per log-barrier iteration and actually sublinear in the number of non-zeros
high-level languages for machine learning and optimization are a step towards the ConclusionsKristian Kersting - Declarative Programming for Statistical ML
Democratization of Machine LearningKristian Kersting - Declarative Programming for Statistical ML Reduces the level of expertise necessary to build optimization applications, makes models faster to write and easier to communicate Facilitate the construction of sophisticated models with rich domain knowledgeSpeed up solvers by exploiting language properties, compression, and compilation
Kristian Kersting - Declarative Programming for Statistical ML
This is all inspired by Turing Award Winner Jim Grays grand challenge of Automated ProgrammingMeeting it requires the help of all of us: Machine Learning, Data Mining, Databases, SAT, Knowledge Representation, Constraint Programming, Game Theory, Graph Theory, Optimization !Thanks for your attention!
Kristian Kersting - Declarative Programming for Statistical ML