![Page 1: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/1.jpg)
Mobile Edge Artificial Intelligence: Opportunities and Challenges
Yuanming Shi
1
ShanghaiTech University
Motivations
![Page 2: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/2.jpg)
Why 6G?
2
Fig. credit: Walid
![Page 3: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/3.jpg)
What will 6G be?
6G networks: from “connected things” to “connected intelligence”
4
6G: connected intelligence 5G: connected things
[Ref] K. B. Letaief, W. Chen, Y. Shi, J. Zhang, and Y. Zhang, “The roadmap to 6G - AI empoweredwireless networks,” IEEE Commun. Mag., vol. 57, no. 8, pp. 84-90, Aug. 2019.
![Page 4: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/4.jpg)
Connected intelligence via AI
Make networks full of AI: embed intelligence across whole networkto provide greater level of automation and adaptiveness
Grid Power
Local Processing
Power Supply
Discharge
Wireless Network
Active Servers Inactive Servers
Cloud Center
User Devices
Edge device
Charge
on-device intelligence
mobile edge intelligence
cloud intelligence
MEC server 18
![Page 5: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/5.jpg)
Success of modern AI
Two secrets of AI’s success: computing power and big data
Computing power: Intel i386, Intel i486, Intel Pentium Intel Core, Nvidia GPU,
Google TPU, Google quantum supremacy,…
Big data: the world’s most valuable resource
is no longer oil, but data
6
![Page 6: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/6.jpg)
Challenges of modern AI
7
model size
sensor
接收器 cloud
transmitter
receiver
speed
energy privacy
![Page 7: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/7.jpg)
Solution: mobile edge AI
Processing at “edge” instead of “cloud”
8
![Page 8: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/8.jpg)
Levels of edge AI
9
Six levels of edge AI based on the path of data
offloading: cloud-edge-device coordination via
data offloading
Fig. credit: Zhou
![Page 9: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/9.jpg)
This talk
Part I: mathematics in edge AI Provable guarantees for nonconvex machine learning
Communication-efficient distributed machine learning
Part II: edge inference process Communication-efficient on-device distributed inference
Energy-efficient edge cooperative inference
Part III: edge training process Over-the-air computation for federated learning
Intelligent reflecting surface empowered federated learning10
![Page 10: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/10.jpg)
Mobile Edge Artificial Intelligence: Opportunities and Challenges
Yuanming Shi
1
ShanghaiTech University
Part I: Theory
![Page 11: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/11.jpg)
Outline
Motivations Taming nonconvexity in statistical machine learning
Communication challenges in distributed machine learning
TwoVignettes: Provable guarantees for nonconvex machine learning Why nonconvex optimization?
Blind demixing via implicitly regularized Wirtinger flow
Communication-efficient distributed machine learning Why gradient quantization?
Learning polynomial neural networks via quantized SGD2
![Page 12: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/12.jpg)
3
Vignettes A: Provable guarantees fornonconvex machine learning
![Page 13: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/13.jpg)
Why nonconvex optimization?
4
![Page 14: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/14.jpg)
Nonconvex problems are everywhere
Empirical risk minimization is usually nonconvex
low-rank matrix completion
blind deconvolution/demixing
dictionary learning
phase retrieval
mixture models
deep learning
…5
![Page 15: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/15.jpg)
Nonconvex optimization may be super scary
Challenges: saddle points, local optima, bumps,…
Fact: they are usually solved on a daily basis via simple algorithms like(stochastic) gradient descent
6
Fig. credit: Chen
![Page 16: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/16.jpg)
Sometimes they are much nicer than we think
Under certain statistical models, we see benign global geometry: nospurious local optima
7
global minimum saddle point
![Page 17: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/17.jpg)
Statistical models come to rescue
Blessings: when data are generated by certain statistical models,problems are often much nicer than worst-case instances
8
Fig. credit: Chen
![Page 18: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/18.jpg)
First-order stationary points
Saddle points and local minima:
9Local minima Saddle points/local maxima
![Page 19: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/19.jpg)
First-order stationary points
Applications: PCA, matrix completion, dictionary learning etc.
Local minima: either all local minima are global minima or all local minimaas good as global minima
Saddle points: very poor compared to global minima; several such points
Bottomline: local minima much more desirable than saddle points
10How to escape saddle points efficiently?
![Page 20: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/20.jpg)
Statistics meets optimization
Proposal: separation of landscape analysis and generic algorithm design
11
landscape analysis (statistics)
generic algorithms (optimization)
all local minima are global minima
all the saddle pointscan be escaped
• dictionary learning (Sun et al. ’15)• phase retrieval (Sun et al. ’16)• matrix completion (Ge et al. ’16)• synchronization (Bandeira et al. ’16)• inverting deep neural nets (Hand et al. ’17)• ...
• gradient descent (Lee et al. ’16)• trust region method (Sun et al. ’16)• perturbed GD (Jin et al. ’17)• cubic regularization (Agarwal et al. ’17)• Natasha (Allen-Zhu ’17)• ...
Issue: conservative computational guarantees for specific problems (e.g., phase retrieval, blind deconvolution, matrix completion)
Fig. credit: Chen
![Page 21: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/21.jpg)
Blind demixing via implicitly regularized Wirtinger flow
12
Solution: blending landscape and convergence analysis
![Page 22: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/22.jpg)
Case study: blind deconvolution
In many science and engineering problems, the observed signal can bemodeled as:
where is the convolution operator
is a physical signal of interest
is the impulse response of the sensory system
Applications: astronomy, neuroscience, image processing, computervision, wireless communications, microscopy data processing,…
Blind deconvolution: estimate and given13
![Page 23: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/23.jpg)
Case study: blind demixing
The received measurement consists of the sum of all convolved signals
Applications: IoT, dictionary learning, neural spike sorting,…
Blind demixing: estimate and given14
low-latency communication for IoT convolutional dictionary learning (multi kernel)
![Page 24: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/24.jpg)
Bilinear model
Translate into the frequency domain…
Subspace assumptions: and lie in some known low-dimensionalsubspaces
where , and
Demixing from bilinear measurements:15
: partial Fourier basis
![Page 25: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/25.jpg)
An equivalent view: low-rank factorization
Lifting: introduce to linearize constraints
Low-rank matrix optimization problem
16
![Page 26: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/26.jpg)
17
Convex relaxation
Ling and Strohmer (TIT’2017) proposed to solve the nuclear normminimization problem:
Sample-efficient: samples for exact recovery ifis incoherent w.r.t.
Computational-expensive: SDP in the lifting space
17
Can we solve the nonconvex matrix optimization problem directly?
: partial Fourier basis
![Page 27: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/27.jpg)
A natural least-squares formulation
Goal: demixing from bilinear measurements
Pros: computational-efficient in the natural parameter space
Cons: is nonconvex: bilinear constraint, scaling ambiguity
18
Given:
![Page 28: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/28.jpg)
Wirtinger flow
Least-square minimization via Wirtinger flow (Candes, Li, Soltanolkotabi ’14)
Spectral initialization by top eigenvector of
Gradient iterations
19
![Page 29: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/29.jpg)
Two-stage approach
Initialize within local basin sufficiently close to ground-truth (i.e.,strongly convex, no saddle points/ local minima)
Iterative refinement via some iterative optimization algorithms
20
Fig. credit: Chen
![Page 30: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/30.jpg)
Gradient descent theory
Two standard conditions that enable geometric convergence of GD
(local) restricted strong convexity
(local) smoothness
21
![Page 31: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/31.jpg)
Gradient descent theory
Question: which region enjoys both strong convexity and smoothness?
is not far away from (convexity)
is incoherent w.r.t. sampling vectors (incoherence region for smoothness)
22
Prior works suggest enforcing regularization (e.g., regularized loss [Ling & Strohmer’17]) to promote incoherence
![Page 32: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/32.jpg)
Our finding: WF is implicitly regularized
WF (GD) implicitly forces iterates to remain incoherent with
cannot be derived from generic optimization theory
relies on finer statistical analysis for entire trajectory of GD
23
region of local strong convexity and smoothness
![Page 33: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/33.jpg)
Key proof idea: leave-one-out analysis
introduce leave-one-out iterates by runningWF without l-th sample
leave-one-out iterate is independent of
leave-one-out iterate true iterate
is nearly independent of (i.e., nearly orthogonal to)24
![Page 34: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/34.jpg)
Theoretical guarantees
With i.i.d. Gaussian design,WF (regularization-free) achieves
Incoherence
Near-linear convergence rate
Summary: Sample size:
Stepsize: vs. [Ling & Strohmer’17]
Computational complexity: vs. [Ling & Strohmer’17]
25[Ref] J. Dong and Y. Shi, “Nonconvex demixing from bilinear measurements,” IEEE Trans. SignalProcess., vol. 66, no. 19, pp. 5152-5166, Oct., 2018.
![Page 35: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/35.jpg)
Numerical results
stepsize:
number of users:
sample size:
26
linear convergence:WF attains - accuracy within iterations
![Page 36: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/36.jpg)
Vignettes B: Communication-efficient distributed machine learning
27
![Page 37: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/37.jpg)
Why gradient quantization?
28
![Page 38: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/38.jpg)
The practical problem
Goal: training large-scale machine learning models efficiently
Large datasets:
ImageNet: 1.6 million images (~300GB)
NIST2000 Switchboard dataset: 2000 hours
Large models:
ResNet-152 [He et al. 2015]: 152 layers, 60 million parameters
LACEA [Yu et al. 2016]: 22 layers, 65 million parameters
29
![Page 39: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/39.jpg)
Data parallel stochastic gradient descent
Challenge: communication is a bottleneck to scalability for large model
30
bigger models
Minibatch 1 Minibatch 2
![Page 40: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/40.jpg)
Quantized SGD
Idea: stochastically quantize each coordinate
31
Update:
is a quantization function which can be communicated with fewer bits
is defined by
Question: how to provide optimality guarantees of quantized SGD for
nonconvex machine learning?
![Page 41: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/41.jpg)
Learning polynomial neural networks via quantized SGD
32
![Page 42: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/42.jpg)
Polynomial neural networks
Learning neural networks with quadratic activation
33
input features:
weights:
output:
![Page 43: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/43.jpg)
Quantized stochastic gradient descent
Mini-batch SGD
sample indices uniformly with replacement from
the generalized gradient of the loss function
Quantized SGD
34
![Page 44: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/44.jpg)
Provable guarantees for QSGD
Theorem 1: SGD converges at linear rate to the globally optimal solution
Theorem 2: QSGD provably maintains similar convergence rate of SGD
35
![Page 45: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/45.jpg)
Concluding remarks
Implicitly regularized Wirtinger flow Implicit regularization: vanilla gradient descent automatically forces iterates to
stay incoherent
Even simplest nonconvex methods are remarkably efficient under suitablestatistical models
Communication-efficient quantized SGD QSGD provably maintains the similar convergence rate of SGD to a globally
optimal solution
Significantly reduce the communication cost: tradeoffs between computation andcommunication
36
![Page 46: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/46.jpg)
Future directions
Deep and machine learning with provable guarantees
information theory, random matrix theory, interpretability,…
Communication-efficient learning algorithms
vector quantization schemes, decentralized algorithms, zero-order algorithms,second-order algorithms, federated optimization,ADMM, …
37
![Page 47: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/47.jpg)
Mobile Edge Artificial Intelligence: Opportunities and Challenges
Yuanming Shi
1
ShanghaiTech University
Part II: Inference
![Page 48: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/48.jpg)
Outline
Motivations
Latency, power, storage
Two vignettes:
Communication-efficient on-device distributed inference
Why on-device inference?
Data shuffling via generalized interference alignment
Energy-efficient edge cooperative inference
Why inference at network edge?
Edge inference via wireless cooperative transmission2
![Page 49: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/49.jpg)
Why edge inference?
3
![Page 50: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/50.jpg)
AI is changing our lives
4
self-driving car smart robots
machine translation AlphaGo
![Page 51: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/51.jpg)
Models are getting larger
5
image recognition speech recognition
Fig. credit: Dally
![Page 52: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/52.jpg)
The first challenge: model size
6difficult to distribute large models through over-the-air update
Fig. credit: Han
![Page 53: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/53.jpg)
The second challenging: speed
7
sensor
接收器 cloud
transmitter
receiver
communication
latencyactuator
long training time limits ML researcher’s
productivity
processing at “Edge” instead of the “Cloud”
![Page 54: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/54.jpg)
The third challenge: energy
8
AlphaGo: 1920 CPUs and 280 GPUs,
$3000 electric bill per game
on mobile: drains batteryon data-center: increases TCO
larger model-more memory reference-more energy
![Page 55: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/55.jpg)
How to make deep learning more efficient?
9
low latency, low power
![Page 56: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/56.jpg)
Vignettes A: On-device distributed inference
10
low latency
![Page 57: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/57.jpg)
On-device inference: the setup
11
weights/parameters
model
training hardware inference hardware
![Page 58: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/58.jpg)
MapReduce: a general computing framework
Active research area: how to fit different jobs into this framework
12
N subfiles, K servers, Q keys
input File
N subfiles
K serversintermediate (key, value)
shuffling phase
Q keys
(blue, )
general framework• Matrix• Distributed ML• Page rank• …
Fig. credit: Avestimehr
![Page 59: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/59.jpg)
Wireless MapReduce: computation model
Goal: low-latency (communication-efficient) on-device inference
Challenges: the dataset is too large to be stored in a single mobiledevice (e.g., a feature library of objects)
Solution: stored files across devices, each can only storeup to files, supported by distributed computing framework MapReduce
Map function: ( input data)
Reduce function: ( intermediate values)
13
![Page 60: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/60.jpg)
Wireless MapReduce: computation model
14
Dataset placement phase: determinethe index set of files stored at eachnode
Map phase: compute intermediatevalues locally
Shuffle phase: exchange intermediatevalues wirelessly among nodes
Reduce phase: construct the outputvalue using the reduce function
on-device distributed inference via wireless MapReduce
![Page 61: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/61.jpg)
Wireless MapReduce: communication model
15
Goal: users (each with antennas)exchange intermediate values via awireless access point ( antennas)
entire set of messages (intermediatevalues)
index set of messages (computedlocally) available at user
index set of messages required byuser wireless distributed computing system
message delivery problem with side information
![Page 62: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/62.jpg)
Wireless MapReduce: communication model
Uplink multiple access stage:
: received at the AP; : transmitted by user ; : channel uses
Downlink broadcasting stage:
: received by mobile user
Overall input-output relationship from mobile user to mobile user
16
![Page 63: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/63.jpg)
Interference alignment conditions
Precoding matrix:
Decoding matrix:
Interference alignment conditions
17
symmetric DoF:
w.l.o.g.
![Page 64: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/64.jpg)
Generalized low-rank optimization
Low-rank optimization for interference alignment
the affine constraint encodes the interference alignment conditions
where
18
![Page 65: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/65.jpg)
Nuclear norm fails
Convex relaxation fails: yields poor performance due to the poorstructure of
example:
the nuclear norm approach always returns full rank solution while theoptimal rank is one
19
![Page 66: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/66.jpg)
Difference-of-convex programming approach
Ky Fan norm [Watson, 1993]: the sum of largest- singular values
The DC representation for rank function
Low-rank optimization via DC programming
Find the minimum such that the optimal objective value is zero
Apply the majorization-minimization (MM) algorithm to iteratively solve aconvex approximation subproblem
20
![Page 67: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/67.jpg)
Numerical results
Convergence results
21
IRLS-p: iterative reweighted least square algorithm
![Page 68: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/68.jpg)
Numerical results
Maximum achievable symmetric DoF over local storage size of each user
22
Insights on DC framework:1. DC function provides a tight
approximation for rank function2. DC algorithm finds better solution
for rank minimization problem
![Page 69: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/69.jpg)
Numerical results
A scalable framework for on-device distributed inference
23
Insights on more devices:1. More messages are requested2. Each file is stored at more devices3. Opportunities of collaboration for
mobile users increase
![Page 70: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/70.jpg)
Vignettes B: Edge cooperative inference
24
low power
![Page 71: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/71.jpg)
Edge inference for deep neural networks
Goal: energy-efficient edge processing framework to execute deeplearning inference tasks at the edge computing nodes
25
modelsmodels
input
output
example: Nvidia’s GauGAN
uplinkdownlink
any task can be performed at multiple APs
pre-downloadedwhich APs shall compute
for me?
![Page 72: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/72.jpg)
Computation power consumption
Goal: estimate the power consumption for deep model inference
Example: power consumption estimation for AlexNet
Cooperative inference tasks at multiple APs:
Computation replication: high compute power
Cooperative transmission: low transmit power
Solution:
minimize the sum of computation and transmission power consumption
26
[Sze’ CVPR 17]
![Page 73: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/73.jpg)
Signal model
Proposal: group sparse beamforming for total power minimization
received signal at -th mobile user:
beamforming vector for at the -th AP:
group sparse aggregative beamforming vector
if is set as zero, task will not be performed at the -th AP
the signal-to-interference-plus-noise-ratio (SINR) for users
27
![Page 74: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/74.jpg)
Probabilistic group sparse beamforming
Goal: total power consumption under probabilistic QoS constraints
Channel state information (CSI) uncertainty
Additive error: ,
Limited precision of feedback, delays in CSI acquisition...
Challenges: 1) group sparse objective function; 2) probabilistic QoSconstraints 28
(maximum transmit power)
transmission and computation power consumption
![Page 75: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/75.jpg)
Probabilistic QoS constraints General idea: obtaining independent samples of the random channel
coefficient vector ; find a solution such that the confidence level of
is no less than .
Limitations of existing methods: Scenario generation (SG): too conservative, performance deteriorates when samples size increases
required sample size
Stochastic Programming: High computation cost, increasing linearly with sample size
No available statistical guarantee 29
![Page 76: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/76.jpg)
Statistical learning for robust optimization
Proposal: statistical learning based robust optimization approximation
constructing a high probability region such that
with confidence at least
imposing target SINR constraints for all elements in high probability region
Statistical learning method for constructing ellipsoidal uncertainty sets
split dataset into two parts
Shape learning: sample mean and sample variance of
(omitting the correlation between , becomes block diagonal) 30
![Page 77: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/77.jpg)
Statistical learning for robust optimization Statistical learning method for constructing
size calibration via quantile estimation for
compute the function value with respect to eachsample in , set as the -th largest value
required sample size:
Tractable reformulation
31
![Page 78: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/78.jpg)
Robust optimization reformulation
Tractable reformulation for robust optimization with S-Lemma
Challenges
group sparse objective function
nonconvex quadratic constraints32
![Page 79: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/79.jpg)
Low-rank matrix optimization
Idea: matrix lifting for nonconvex quadratic constraints
Matrix optimization with rank-one constraint
33
![Page 80: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/80.jpg)
Reweighted power minimization approach
Sparsity: reweighted -minimization for inducing group sparsity
Approximation: ,
Alternatively optimizing and updating weights
Low-rankness: DC representation for rank-one positive semidefinitematrix
where34
![Page 81: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/81.jpg)
Reweighted power minimization approach
Updating updating
The DC algorithm via iteratively linearizing the concave part
: the eigenvector corresponding to the largest eigenvalue of 35
![Page 82: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/82.jpg)
Numerical results
36
Performance of our robust optimization approximation approach andscenario generation
![Page 83: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/83.jpg)
Numerical results
37
Energy-efficient processing and robust wireless cooperative transmissionfor executing inference tasks at possibly multiple edge computing nodes
Insights on edge inference:1. Selecting the optimal set of access
points for each inference task via group sparse beamforming
2. A robust optimization approach for joint chance constraints via statistical learning to learn CSI uncertainty set
![Page 84: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/84.jpg)
Concluding remarks
Machine learning model inference over wireless networks On-device inference via wireless distributed computing
Edge inference via computation replication and cooperative transmission
Sparse and low-rank optimization framework Inference alignment for data shuffling in wireless MapReduce
Joint inference tasking and downlink beamforming for edge inference
Nonconvex optimization frameworks DC algorithm for generalized low-rank matrix optimization
Statistical learning for stochastic robust optimization38
![Page 85: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/85.jpg)
Future directions
On-device distributed inference
model compression, energy efficient inference, full duplex,…
Edge cooperative inference
hierarchical inference over cloud-edge-device, low-latency, …
Nonconvex optimization via DC and learning approaches
optimality, scalability, applicability, …
39
![Page 86: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/86.jpg)
Mobile Edge Artificial Intelligence: Opportunities and Challenges
Yuanming Shi
1
ShanghaiTech University
Part III: Training
![Page 87: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/87.jpg)
Outline
Motivations
Privacy, federated learning
Two vignettes:
Over-the-air computation for federated learning
Why over-the-air computation?
Joint device selection and beamforming design
Intelligent reflecting surface empowered federated learning
Why intelligent reflecting surface?
Joint phase shifts and transceiver design2
![Page 88: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/88.jpg)
Intelligent IoT ecosystem
3
Internet of Things
Mobile Internet
Tactile Internet
Develop computation, communication & AI technologies: enable smart IoT applications to make
low-latency decision on streaming data
(Internet of Skills)
![Page 89: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/89.jpg)
Intelligent IoT applications
4
Autonomous vehicles
Smart health Smart agriculture
Smart home Smart city
Smart drones
![Page 90: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/90.jpg)
Challenges
Retrieve or infer information from high-dimensional/large-scale data
5
limited processing ability (computation, storage, ...)
2.5 exabytes of dataare generated every day (2012)
exabyte zettabyte yottabyte...??
We’re interested in the information rather than the data
Challenges: High computational cost Only limited memory is available Do NOT want to compromise statistical accuracy
![Page 91: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/91.jpg)
High-dimensional data analysis
6
(big) data
Models: (deep) machine learning
Methods:1. Large-scale optimization2. High-dimensional statistics3. Device-edge-cloud computing
![Page 92: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/92.jpg)
Deep learning: next wave of AI
7
image recognition
speech recognition
natural languageprocessing
![Page 93: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/93.jpg)
Cloud-centric machine learning
8
![Page 94: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/94.jpg)
9
The model lives in the cloud
![Page 95: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/95.jpg)
10
We train models in the cloud
![Page 96: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/96.jpg)
11
![Page 97: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/97.jpg)
12
Make predictions in the cloud
![Page 98: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/98.jpg)
13
Gather training data in the cloud
![Page 99: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/99.jpg)
14
And make the models better
![Page 100: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/100.jpg)
Why edge machine learning?
15
![Page 101: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/101.jpg)
Challenges to modern AI
Challenges: data privacy and confidentiality; small data and fragmenteddata; data quality and limited labels
16
Facebook’s data privacy scandal the general data protection regulation (GDPR)
![Page 102: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/102.jpg)
Learning on the edge
The emerging high-stake AI applications: low-latency, privacy,…
17
phones drones robots
glasses self driving cars
where to compute?
![Page 103: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/103.jpg)
Mobile edge AI
Processing at “edge” instead of “cloud”
18
![Page 104: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/104.jpg)
Edge computing ecosystem
“Device-edge-cloud” computing system for mobile AI applications
Grid Power
Local Processing
Power Supply
Discharge
Wireless Network
Active Servers Inactive Servers
Cloud Center
User Devices
Edge device
Charge
on-device computing
mobile edge computing
cloud computing
MEC server
Shannon (communication)meets Turing (computing)
18
![Page 105: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/105.jpg)
Edge machine learning
Edge ML: both ML inference and training processes are pushed downinto the network edge (bottom)
20
Fig. credit: Park
![Page 106: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/106.jpg)
Vignettes A: Over-the-air computation for federated learning
21
![Page 107: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/107.jpg)
Federated computation and learning
Goal: imbue mobile devices with state of the art machine learningsystems without centralizing data and with privacy by default
Federated computation: a server coordinates a fleet of participatingdevices to compute aggregations of devices’ private data
Federated learning: a shared global model is trained via federatedcomputation
22
![Page 108: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/108.jpg)
Federated learning
26
![Page 109: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/109.jpg)
24
Federated learning
27
![Page 110: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/110.jpg)
25
Federated learning
28
![Page 111: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/111.jpg)
26
Federated learning
29
![Page 112: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/112.jpg)
27
Federated learning
30
![Page 113: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/113.jpg)
28
Federated learning
31
![Page 114: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/114.jpg)
29
Federated learning
32
![Page 115: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/115.jpg)
Federated learning: applications
Applications: where the data is generated at the mobile devices and is undesirable/infeasible to be transmitted to centralized servers
30
financial services smart retail smart healthcarekeyboard prediction
![Page 116: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/116.jpg)
Federated learning over wireless networks
Goal: train a shared global model via wireless federated computation
31
System challenges Massively distributed Node heterogeneity
Statistical challenges Unbalanced Non-IID Underlying structure
on-device distributed federated learning system
![Page 117: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/117.jpg)
How to efficiently aggregate models over wireless networks?
32
![Page 118: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/118.jpg)
Model aggregation via over-the-air computation
Aggregating local updates from mobile devices
weighted sum of messages
mobile devices and one antenna base station
is the set of selected devices
is the data size at device 33
Over-the-air computation: explore signal superposition of
a wireless multiple-access channel for model aggregation
![Page 119: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/119.jpg)
Over-the-air computation
The estimated value before post-processing at the BS
is the transmitter scalar, is the received beamforming vector, is a normalizing factor
target function to be estimated:
recovered aggregation vector entry via post-processing:
Model aggregation error:
Optimal transmitter scalar: 34
![Page 120: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/120.jpg)
Problem formulation
Key observations:
More selected devices yield fast convergence rate of the training process
Aggregation error leads to the deterioration of model prediction accuracy
35
![Page 121: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/121.jpg)
Problem formulation
Goal: maximize the number of selected devices under target MSE constraint
Joint device selection and received beamforming vector design
Improve convergence rate in the training process, guarantee prediction accuracy in the inference process
Mixed combinatorial optimization problem
36
![Page 122: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/122.jpg)
Sparse and low-rank optimization
Sparse and low-rank optimization for on-device federated learning
37
multicasting duality
sum of feasibilities
matrix lifting
![Page 123: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/123.jpg)
Sparse and low-rank optimization
38
![Page 124: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/124.jpg)
Problem analysis
Goal: induce sparsity while satisfying fixed-rank constraint
Limitations of existing methods
Sparse optimization: iterative reweighted algorithms are parameters sensitive
Low-rank optimization: semidefinite relaxation (SDR) approach (i.e., droprank-one constraint) has the poor capability of returning rank-one solution
39
![Page 125: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/125.jpg)
Difference-of-convex functions representation
Ky Fan norm [Fan, PNAS’1951]: the sum of largest- absolute values
is a permutation of ,where
40
PNAS’1951
![Page 126: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/126.jpg)
Difference-of-convex functions representation
DC representation for sparsity function
DC representation for rank-one positive semidefinite matrix
where
[Ref] J.-y. Gotoh, A. Takeda, and K. Tono, “DC formulations and algorithms for sparse optimizationproblems,” Math. Program., vol. 169, pp. 141– 176, May 2018.
41
![Page 127: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/127.jpg)
A DC representation framework
A two-step framework for device selection
Step 1: obtain the sparse solution such that the objective value achieveszero through increasing from to
42
![Page 128: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/128.jpg)
A DC representation framework
Step II: feasibility detection
Ordering in descending order as
Increasing from to , choosing as
Feasibility detection via DC programming
43
![Page 129: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/129.jpg)
DC algorithm with convergence guarantees
and : minimize the difference of two strongly convex functions
e.g., and
The DC algorithm via linearizing the concave part
converge to a critical point with speed
44
![Page 130: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/130.jpg)
Numerical results
Convergence of the proposed DC algorithm for problem
45
![Page 131: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/131.jpg)
Numerical results
Probability of feasibility with different algorithms
46
![Page 132: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/132.jpg)
Numerical results
Average number of selected devices with different algorithms
47
![Page 133: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/133.jpg)
Numerical results
Performance of proposed fast model aggregation in federated learning
Training an SVM classifier on CIFAR-10 dataset
48
![Page 134: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/134.jpg)
Vignettes B: Intelligent reflecting surface empoweredfederated learning
49
![Page 135: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/135.jpg)
Smart radio environments
Current wireless networks: no control of radio waves
Perceive the environment as an “unintentional adversary” to communication
Optimize only the end-points of the communication network
No control of the environment, which is viewed as a passive spectator
Smart radio environments: reconfigure the wireless propagations
50
“dumb” wireless “smart” wireless
Fig. credit: Renzo
![Page 136: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/136.jpg)
Intelligent reflecting surface
Working principle of intelligent reflecting surface (IRS): differentelements of an IRS can reflect the incident signal by controlling itsamplitude and/or phase for directional signal enhancement or nulling
51Fig. credit: Renzo
improve spectral and energy efficiency
1. no any active transmit module
2. operate in full-duplex mode
![Page 137: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/137.jpg)
Intelligent reflecting surface
Architecture of intelligent reflecting surface
52Fig. credit: Wu
1. Outer layer: a large number of metallic patches (elements) are printed on a dielectric substrate to directly interact with incident signals.
2. Second layer: a copper plate is used to avoid the signal energy leakage.
3. Inner layer: a control circuit board for adjusting the reflection amplitude/phase shift of each element, triggered by a smart controller attached to the IRS.
![Page 138: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/138.jpg)
Intelligent reflecting surface meet wireless networks
53Fig. credit: Wu
intelligent reflecting surface meets wireless network: • over-the-air computation• edge computing/caching• wireless power transfer• D2D communications• massive MIMO• NOMA• mmWave• …
![Page 139: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/139.jpg)
IRS empowered AirComp
Intelligent reflecting surface (IRS):
overcoming unfavorable signal propagation conditions
improving spectrum and energy efficiency
tuning phase shifts with passive elements
54
IRS aided AirComp system: build controllable wireless environments
to boost received signal power w.l.o.g. assuming
![Page 140: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/140.jpg)
Problem formulation
Received signal at the AP:
w.l.o.g. suppose target function:
Aggregation error:
optimal transmitter scalar:
Proposal: joint design for AirComp transceivers and IRS phase shifts
55
received beamforming vector
![Page 141: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/141.jpg)
Nonconvex bi-quadratic programming
Nonconvex bi-quadratic programming problem
Challenges:
nonconvex quadratic constraints with respect to and
Solution:
Alternating minimization for and
Matrix lifting to alternatively linearize nonconvex bi-quadratic constraints 56
![Page 142: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/142.jpg)
An alternating DC framework
57
Goal: updating receiver beamforming vector with fixed IRS phase shifts
matrix liftingDC programming
DC representation
![Page 143: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/143.jpg)
An alternating DC framework
58
Goal: updating phase shifts with fixed beamformer
matrix liftingDC programming
denoting
DC representation
![Page 144: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/144.jpg)
Numerical results
Convergence behaviors of the proposed alternating DC algorithm
59
layout of AP, IRS and users
![Page 145: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/145.jpg)
Numerical results
Performance of different algorithms with different network settings
60
![Page 146: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/146.jpg)
Numerical results
The power of IRS for AirComp
61
Insights: deploying IRS in AirComp system can significantly enhance the
MSE performance for data aggregation
![Page 147: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/147.jpg)
IRS empowered federated learning system
The power of IRS for federated learning
62training loss prediction accuracy
![Page 148: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/148.jpg)
Concluding remarks
Federated learning over “intelligent” wireless networks
Federated learning via over-the-air computation
Over-the-air computation empowered by intelligent reflecting surface
Sparse and low-rank optimization framework
Joint device selection and beamforming design for over-the-air computation
Joint phase shifts and transceiver design for IRS empowered AirComp
A unified DC programming framework
DC representation for sparse and low-rank functions
63
![Page 149: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/149.jpg)
Future directions
Federated learning
stragglers, security, provable guarantees, …
Over-the-air computation
channel uncertainty, synchronization, security, …
Sparse and low-rank optimization via DC programming
optimality, scalability,…
64
![Page 150: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/150.jpg)
To learn more… Web: http://shiyuanming.github.io/publicationstopic.html
Papers: K. B. Letaief, W. Chen, Y. Shi, J. Zhang, and Y. Zhang, “The roadmap to 6G - AI empowered wireless networks,” IEEE Commun.
Mag., vol. 57, no. 8, pp. 84-90, Aug. 2019.
J. Dong and Y. Shi, “Nonconvex demixing from bilinear measurements,” IEEE Trans. Signal Process., vol. 66, no. 19, pp. 5152-5166,Oct., 2018.
M. C. Tsakiris, L. Peng, A. Conca, L. Kneip, Y. Shi, and H. Choi, “An algebraic-geometric approach to shuffled linear regression,”IEEE Trans. Inf. Theory., under major revision, 2019. https://arxiv.org/abs/1810.05440
K. Yang, Y. Shi, and Z. Ding, “Data shuffling in wireless distributed computing via low-rank optimization,” IEEE Trans. Signal Process.,vol. 67, no. 12, pp. 3087-3099, Jun., 2019.
K. Yang, Y. Shi, W. Yu, and Z. Ding, “Energy-efficient processing and robust wireless cooperative transmission for edgeinference,” submitted. https://arxiv.org/abs/1907.12475
S. Hua, Y. Zhou, K. Yang, and Y. Shi, “Reconfigurable intelligent surface for green edge inference,” submitted.https://arxiv.org/abs/1912.00820
K. Yang, T. Jiang, Y. Shi, and Z. Ding, “Federated learning via over-the-air computation,” IEEE Trans. Wireless Commun., under minorrevision, 2019. https://arxiv.org/abs/1812.11750
T. Jiang and Y. Shi, “Over-the-air computation via intelligent reflecting surfaces,” in Proc. IEEE Global Commun. Conf. (Globecom),Waikoloa, Hawaii, USA, Dec. 2019. https://arxiv.org/abs/1904.12475
65
![Page 151: Mobile Edge Artificial Intelligence: Opportunities and Challengesshiyuanming.github.io/slides/edgeAIGC19.pdf · 2020-05-30 · Mobile Edge Artificial Intelligence: Opportunities and](https://reader034.vdocuments.us/reader034/viewer/2022050107/5f45a294b985c669646430f3/html5/thumbnails/151.jpg)
66
Thankshttp://shiyuanming.github.io/home.html