parallel self-adaptive parallel processing neural networks with irregular nodal processing powers...
TRANSCRIPT
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
1/33
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
2/33
Department of Information Technology
Indian Institute of Information Technology and Management Gwalior Rahul Kala
Problems with Back PropagationProblems with Back Propagation Algorithm Algorithm
Increasing training data
Increasing dimensionality
Increasing Problem Complexity
Limited Computational Power
No Loss of Generality D esired
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
3/33
Department of Information Technology
Indian Institute of Information Technology and Management Gwalior Rahul Kala
The Basic IdeaThe Basic Idea
Divide the computationof Back Propagation Algorithm into many Processing Elements
for higher speed
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
4/33
Department of Information Technology
Indian Institute of Information Technology and Management Gwalior Rahul Kala
Motivations from HumanMotivations from HumanBrainBrain
Parallel Architecture Learning Evolution
Adaptation ComputationalDisparity
Flexibility
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
5/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Basic Partitioning TechniquesBasic Partitioning Techniques
Basic PartitioningTechniques
Layer
D atasetNode
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
6/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
D ataset PartitioningD ataset Partitioningy D ivide data into PEs
y Each PE trains its set of data
y Weights are exchanged andaggregated
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
7/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Basic TechniqueBasic Technique
PE 3 PE NPE 1 PE 2
PE 3 PE NPE 1 PE 2
Data, ANN
Weights
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
8/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
CommunicationCommunication
PE 3
PE 4
PE N
PE 1
PE 2
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
9/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Layer PartitioningLayer Partitioning - -11y D ivide layers into PEs
y Each PE performs its part of computation
y D ifferent PEs store different weights
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
10/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Layer PartitioningLayer Partitioning - -IIIIy Feed forward Basic Equation
y Feed back
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
11/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Basic TechniqueBasic Technique
PE 2PE 2 PE NPE NPE 1PE 1
i1i1
i2i2
inin
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
12/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
CommunicationCommunication
L ayer 3L ayer 2L ayer 1 L ayer N
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
13/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
WorkingWorking
T1
T2
T N
T1
T2
T N
T1
T2
T N
L ayer 1 L ayer 2 L ayer N
Tasks
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
14/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Node PartitioningNode Partitioning - - IIy D ivide nodes into PEs
y Each PE performs its part of computation
y D ifferent PEs store different weights
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
15/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Node PartitioningNode Partitioning - -IIIIy Feed forward
BasicEquation
y Feed Back
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
16/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Basic TechniqueBasic Technique
PE 2
PE N
PE 1i1
i2
in
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
17/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
CommunicationCommunication
PE 1
PE 2
PE N
PE 3
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
18/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
WorkingWorking
T1
T2
T N
T1
T2
T N
T1
T2
T N
L ayer 1 L ayer 2 L ayer N
Tasks
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
19/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Hierarchical PartitioningHierarchical Partitioningy Mixture of three partitioning at different
levels
y Level 1: D ata Sety Level 2: Node or Layer
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
20/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
General techniqueGeneral technique
Dataset1
L ayer 2
L ayer 3
L ayer 1 L ayer 2
L ayer 3
Node 1 Node 2
Node 3
Node 1
Node 2 Node 3
L ayer 1
Node 3 Node 2
Node 1
Dataset2
Dataset3
Dataset
4
Dataset N
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
21/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Self AdaptationSelf Adaptationy Balance computational load among
PEs as per their capability
y Works for data set partitioning
y Computation reallocated with some
frequency
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
22/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Self Adaptation Server/ClientSelf Adaptation Server/ClientModelModel
PE 4
PE N
PE 3
PE 1 PE 2Server
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
23/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
CommunicationsCommunications
Node3
Node2
L ayer3
L ayer2
Node1L ayer1
Server DataSet 1
DataSet 2
DataSet 3
L ayer3
L ayer2
DataSet N
L ayer1
Node3
Node2
Node1
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
24/33
RESULTSRESULTS
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
25/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Speedup v/s No of PEs (InputSpeedup v/s No of PEs (Input1)1)
0
1
2
3
4
5
6
7
8
9
10
2 4 6 8 10 20
Speedup v/s No of PEs
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
26/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Speedups for Input 1Speedups for Input 1
PENo of inputs
Network
Architectur e Iterations
Server
Sync after iterations Time Serial
TimeParallel Speedup
2 500 8-15-1 150000 50000 823737 982521 0.838391
4 500 8-15-1 150000 50000 823737 530306 1.5533246 500 8-15-1 150000 50000 823737 384294 2.143507
8 500 8-15-1 150000 50000 823737 263511 3.126006
10 500 8-15-1 150000 50000 823737 211249 3.899365
20 500 8-15-1 150000 50000 823737 89225 9.232132
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
27/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Speedup v/s No of PEs (Input 1)Speedup v/s No of PEs (Input 1)without self adaptive approachwithout self adaptive approach
0
1
2
3
4
5
6
7
8
9
10
2 4 6 8 10 20
Speedup v/s No of PEs
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
28/33
Department of Information TechnologyIndian Institute of Information Technology and Management Gwalior Rahul Kala
Speedups for Input 1 without self Speedups for Input 1 without self adaptationadaptation
PENo of inputs
Network
Architectur e Iterations
Server
Sync after iterations Time Serial
TimeParallel Speedup
2 500 8-15-1 150000 NA 823737 993483 0.829141
4 500 8-15-1 150000 NA 823737 674358 1.2215136 500 8-15-1 150000 NA 823737 473899 1.738212
8 500 8-15-1 150000 NA 823737 342174 2.407363
10 500 8-15-1 150000 NA 823737 303214 2.716685
20 500 8-15-1 150000 NA 823737 97180 8.476405
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
29/33
Department of Information Technology
Indian Institute of Information Technology and Management Gwalior Rahul Kala
Speedup v/s No of PEs (InputSpeedup v/s No of PEs (Input1I)1I)
0
1
2
3
4
5
6
7
8
9
10
2 4 6 8 10 20
Speedup v/s No of PEs
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
30/33
Department of Information Technology
Indian Institute of Information Technology and Management Gwalior Rahul Kala
Speedups for Input 1ISpeedups for Input 1I
PENo of inputs
Network
Architectur e Iterations
Server
Sync after iterations Time Serial
TimeParallel Speedup
2 500 11-25-1 150000 50000 1733690 1829312 0.947728
4 500 11-25-1 150000 50000 1733690 984654 1.760716 500 11-25-1 150000 50000 1733690 644070 2.691773
8 500 11-25-1 150000 50000 1733690 486311 3.564982
10 500 11-25-1 150000 50000 1733690 429221 4.039155
20 500 11-25-1 150000 50000 1733690 179378 9.665009
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
31/33
Department of Information Technology
Indian Institute of Information Technology and Management Gwalior Rahul Kala
Speedup v/s No of PEs (InputSpeedup v/s No of PEs (Input
1I) without self adaptive1I) without self adaptiveapproachapproach
0
1
2
3
4
5
6
7
8
9
10
2 4 6 8 10 20
Speedup v/s No of PEs
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
32/33
Department of Information Technology
Indian Institute of Information Technology and Management Gwalior Rahul Kala
Speedups for Input 1I withoutSpeedups for Input 1I withoutself adaptationself adaptation
PENo of inputs
Network
Architectur e Iterations
Server
Sync after iterations Time Serial
TimeParallel Speedup
2 500 11-25-1 150000 NA 1733690 1911289 0.907079
4 500 11-25-1 150000 NA 1733690 1175223 1.4752016 500 11-25-1 150000 NA 1733690 683619 2.536047
8 500 11-25-1 150000 NA 1733690 548465 3.160986
10 500 11-25-1 150000 NA 1733690 471293 3.678582
20 500 11-25-1 150000 NA 1733690 196229 8.835035
-
8/14/2019 Parallel Self-Adaptive Parallel Processing Neural Networks with irregular Nodal Processing Powers using Hierarchical Partitioning of Artificial neural Networks
33/33
Department of Information Technology
Indian Institute of Information Technology and Management Gwalior Rahul Kala