graph representation learning
TRANSCRIPT
![Page 2: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/2.jpg)
Overview● Graph-structured data● Graph Neural Networks● Applications
![Page 3: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/3.jpg)
Graphs
![Page 4: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/4.jpg)
Graphs - Where we find them...
Social Media Inter-connected devices on internet Molecular bonds
![Page 5: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/5.jpg)
Graphs● Euclidean graph-data has regular/ periodic structure
○ Eg. Images, videos, time series
● Discrete, Non-Euclidean data structure
Euclidean/regular grid
![Page 6: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/6.jpg)
Graphs● Euclidean graph-data has regular/ periodic structure
○ Eg. Images, videos, time series
● Discrete, Non-Euclidean data structure○ Eg. social networks, molecular structure
Non-euclidean/ non-regular grid
![Page 7: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/7.jpg)
Graphs● Discrete, Non-Euclidean data structure● Key features are;
○ Nodes○ Node features
![Page 8: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/8.jpg)
Graphs● Discrete, Non-Euclidean data structure● Key features are;
○ Nodes○ Node features○ Edges (directed or undirected)
![Page 9: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/9.jpg)
Graphs● Discrete, Non-Euclidean data structure● Key features are;
○ Nodes○ Node features○ Edges (directed or undirected)○ Edge weights○ Edge features
![Page 10: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/10.jpg)
Graphs● Discrete, Non-Euclidean data structure● Key features are;
○ Nodes○ Node features○ Edges (directed or undirected)○ Edge weights○ Edge features○ Edge types etc
![Page 11: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/11.jpg)
Graphs - More FormallyA simple graph is given by:
Where → set of nodes/ vertices, X → node features
→ set of edges weights, A → Adjacency matrix
Assuming a real-valued feature matrix X for each node
Adjacency Matrix -
A simple binary matrix -
Entry is 1 if nodes are connected and 0 otherwise
![Page 12: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/12.jpg)
Knowledge Graphs
Generated by ConceptNetNeo4j
![Page 13: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/13.jpg)
Knowledge Graphs● Fact triplets → head
entity, relation, tail entity● Symmetry is critical (marriage
is symmetric, “is a capital of” is not)
● Usually incomplete → missing links needs to be completed
● We need embedding methods to learn the KG structure and to predict missing links
![Page 14: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/14.jpg)
Main Downstream Tasks
● Node/ Graph Classification - Label nodes/entire graph that were not labeled before in a typical semi-supervised setting
● Link Prediction - Predict links between nodes
● Node Clustering/ Community Detection - Detect clusters of nodes in graph
● Graph Generation - Generate a new graph
![Page 15: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/15.jpg)
What we wantPriori: Learn Good Embeddings for better model
![Page 16: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/16.jpg)
![Page 17: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/17.jpg)
Fundamental QuestionsPriori: Learn Good Embeddings for better model
How to encode the graph structure into machine learning models;
● Leveraging a node’s local or global neighbourhood structure into a feature vector for node classification
● Encoding pairwise information between nodes for link prediction
![Page 18: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/18.jpg)
Classical Rep. Approaches
● Random walks
○ DeepWalk
○ Node2vec
○ LINE (Large-scale Information Network Embeddings)
○ HARP (Random-walk embeddings via graph pre-processing)
● Neighborhood autoencoder and aggregation
○ Deep Neural Graph Representations (DNGR)
○ Structural Deep Network Embeddings (SDNE) etc.
● Transductive learning● No parameter sharing● No feature learned
representation
Hamilton, 2018
![Page 19: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/19.jpg)
Graph Neural Networks1. Simply a family of deep learning/ neural
network methods applied on graphs2. Many tricks and hacks in NN applies● ReLU activation● Graph Pooling● Stacking layers for hierarchical feature
learning● Negative Sampling● Subsampling3. Many idea “Message passing”
![Page 20: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/20.jpg)
Graph Neural Networks● Graph Convolutional Networks● Graph Attention Networks● Graph Recurrent Networks● Graph Autoencoder● Variational Graph Autoencoder
![Page 21: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/21.jpg)
Graph Convolutional Networks (GCN)
● Permutation invariance● Weight shared between layers● Linear complexity O(E)
Kipf & Welling (ICLR 2017)
Illustration from Kipf’s slides
![Page 22: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/22.jpg)
Graph Attention Networks
Pay attention to the most influential nodes
● Computes attention vector that weighs every neighboring node by importance
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio
![Page 23: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/23.jpg)
Graph Attention NetworksPay attention to the most influential nodes
● Computes attention vector \alpha that weighs every neighboring node by importance
● Multi-head attention computes and concatenates k independent attention mechanisms.
● If final layer, we average out
![Page 24: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/24.jpg)
Graph AutoencodersEncoder-Decoder model
![Page 25: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/25.jpg)
Applications
![Page 26: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/26.jpg)
Graph Neural Networks for Social Recommendation
Two parts graph;
● user-item graph (Numbers denotes rating)
● User-user social graph
Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, Dawei Yin
![Page 27: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/27.jpg)
Graph Neural Networks for Social Recommendation
Separately model hidden state;
● User-item interaction - Item Aggregation (IA)
● User-user interaction - Social aggregation (SA)
Feed concatenated state representation from IA and SA into MLP and predict rating
Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, Dawei Yin
![Page 28: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/28.jpg)
Multi-Agent Game Abstraction via Graph Attention Neural NetworkYong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, Yang Gao
Model the complex interactions bet
![Page 29: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/29.jpg)
Bilateral Trade Modeling with GNN
● Data -- Bilateral trade data, Countries profile information
● Tasks○ Node Classification
■ Predict income levels of countries -- High, Upper-middle, Lower-middle, Low
○ Link Prediction■ Predict if any two countries would trade
Kobby Panford-Quainoo, Avishek Joey Bose, Michael Defferrard
![Page 30: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/30.jpg)
OthersGCap:Graph-based Automatic Image Captioning --- Jia-Yu Pan, Hyung-Jeong Yang, Christos Faloutsos, Pinar Duygulu
MolGAN: An implicit generative model for small molecular graphs --- Nicola De Cao Thomas Kipf
Convolutional Networks on Graphs for Learning Molecular Fingerprints - David Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gomez-Bombarelli, Timothy Hirzel, Alan Aspuru-Guzik, Ryan P. Adams
![Page 31: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/31.jpg)
GNN ToolkitsDatasets:
● Cora● Citeseer● PubMed● TU Dataset● Protein interaction dataset
Python Libraries
● DeepGraph Library (DGL)● Pytorch-Geometric ● Pytorch-BigGraph
Open Graph Benchmark:
A collection of datasets, benchmarks and evaluators for machine learning on graphs
https://ogb.stanford.edu/
![Page 32: Graph Representation Learning](https://reader033.vdocuments.us/reader033/viewer/2022053112/62920ac5bad2b220737e46ab/html5/thumbnails/32.jpg)
Reference MaterialsZiwei Zhang, Peng Cui and Wenwu Zhu. Deep Learning on Graphs: A survey
Vijay Prakash Dwivedi, Chaitanya K. Joshi, Thomas Laurent, Yoshua Bengio, Xavier Bresson. Benchmarking Graph Neural Networks
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, Maosong Sun. Graph Neural Networks:A Review of Methods and Applications
William L. Hamilton, Rex Ying, Jure Leskovec. Representation Learning on Graphs: Methods and Applications