(group-theoretic) fast matrix...

(Group-theoretic) Fast Matrix Multiplication

Ivo Hedtke

“Data Structures and Efficient Algorithms” Group(Prof. Dr. M. Müller-Hannemann)

Martin-Luther-University Halle-WittenbergInstitute of Computer Science

30th of January, 2013updated on 11th March, 2013

Ivo Hedtke (University Halle-Wittenberg) (Group-theoretic) Fast Matrix Multiplication 30th of January, 2013 1 / 25

1 Fast Matrix MultiplicationAn Overview

2 Group-theoretic Fast Matrix MultiplicationAn Introduction

3 Properties of the Triple Product Propertyand the Search for Triple Product Property TriplesA Summary


Part1Fast Matrix Multiplication

An Overview


Naive Matrix Multiplication: “Row × Column”

b1,1 · · · b1,j · · · b1,n

b2,1 · · · b2,j · · · b2,n...

......

bp,1 · · · bp,j · · · bp,n

a1,1 a1,2 · · · a1,p...

......

ai,1 ai,2 · · · ai,p...

......

am,1 am,2 · · · am,p

c1,1 · · · c1,j · · · c1,n...

......

ci,1 · · · ci,j · · · ci,n...

......

cm,1 · · · cm,j · · · cm,n

A ∈ Rm×p, B ∈ Rp×n, C := AB ∈ Rm×n ci,j = (AB)i,j =

p∑`=1

ai,`b`,j


Fast 6= Fast

Fast MatrixMultiplication

Fast=Time

specialtypes ofmatrices

shape/form/layout

smallmatrices

sparsematrices

parallelcomp.

computerarchi-tecture

Fast=Complexity

asymptoticruntime

complexityof other

problems


Naive Matrix Multiplication: “Row × Column”

b1,1 · · · b1,j · · · b1,n

b2,1 · · · b2,j · · · b2,n...

......

bp,1 · · · bp,j · · · bp,n

a1,1 a1,2 · · · a1,p...

......

ai,1 ai,2 · · · ai,p...

......

am,1 am,2 · · · am,p

c1,1 · · · c1,j · · · c1,n...

......

ci,1 · · · ci,j · · · ci,n...

......

cm,1 · · · cm,j · · · cm,n

A ∈ Rm×p, B ∈ Rp×n, C := AB ∈ Rm×n ci,j = (AB)i,j =

p∑`=1

ai,`b`,j

for (int i = 0; i < N; i++) {for (int j = 0; j < M; j++) {

aux = 0.0;for (int k = 0; k < P; k++) {

aux += A[i][k]*B[k][j];}Result[i][j] = aux;

}}


Naive Matrix Multiplication: Performance

Ivo Hedtke Matrikel-Nr. 91378 Blatt 03, Gruppe: Mi 8-10

Ich habe auf c0-30 gerechnet!

Aufgabe 4a

Ich habe folgenden Quelltext verwendet:

#include <stdlib.h>#include <stdio.h>#include <sys/times.h>

int main(int argc,char **argv) {

long N, W, L;int i, j, k, w;float s;struct tms t1, t2;clock_t ut;double time_sec, z;

L = strtol(argv[2], NULL, 10);W = strtol(argv[1], NULL, 10);

for ( N = L; N < L+100; N++){

float **A = malloc(N*sizeof(float*));float **B = malloc(N*sizeof(float*));float **C = malloc(N*sizeof(float*));

for (i = 0; i < N; i++) {A[i] = malloc(N*sizeof(float));B[i] = malloc(N*sizeof(float));C[i] = malloc(N*sizeof(float));for (j=0; j < N; j++){

A[i][j]=rand();B[i][j]=rand();C[i][j]=0.0;

}}

times(&t1);

// Wiederholungenfor (w = 0; w < W; w++){

// Matrix-Multiplikationfor (i = 0; i < N; i++){

for (j = 0; j < N; j++){s = 0.0;for (k = 0; k < N; k++){

s += A[i][k]*B[k][j];}C[i][j] = s;

}}

}

times(&t2);

ut = t2.tms_utime - t1.tms_utime;time_sec = ut / 100.;

z = ((2*N*N*N - N*N)*W)/(1000000.0*time_sec);printf("%i %f\n",N,z);

free(A);free(B);free(C);

}}

Folgendes Diagramm konnte ich aus den Daten er-stellen:

0 100 200 300 400 500 600 700 80040

60

80

100

120

140

160

180

200

220

N

MF

LOP

S

Student Version of MATLAB

Wenn wir uns die Kurve anschauen, so würde ichschätzen, dass bei N ≈ 140 =: N0 Teile der Ma-trizen in den L2-Cache ausgelagert werden müs-sen. Dies ergäbe eine Größe für den L1-Cache von:4 · 3 · N2

0 ≈ 235200 Byte. Dies sind ≈ 230kB. Ei-ne gängige Größe sind 256kB, sicher verbraucht dasOS noch einen Teil des L1-Cache. Ich vermute also256kB für den L1-Cache.

Aufgabe 4b

Ich habe folgenden Quelltext verwendet:

#include <stdlib.h>#include <stdio.h>#include <sys/times.h>

// Aufgabe 04, Teil b

int main(int argc,char **argv) {

long N, W, L, m, T;int i, j, k, w, is, js, ks;float s;struct tms t1, t2;

loes03.tex Große Gleichungssysteme Seite 1 von 3

scalar additions n3 −n2

scalar multiplications n3

2n3 −n2 FLOPs

FLOP = Floating Point Operation

FLOPs = Floating Point Operations

FLOPS = Floating Point Operations per Second


RAM Model (random-access machine)1

Instructions are executed one after another, with no concurrentoperations.

Every instruction takes the same amount of time, at least up to smallconstant factors.

Unbounded amount of available memory.

Memory stores words of size O(logn) bits where n is the input size.

Any desired memory location can be accessed in unit time.

For numerical and geometric algorithms, it is sometimes also assumedthat words can be represent real numbers accurately.

Exact arithmetic on arbitrary real numbers can be done in constanttime.

1[D. Ajwani and H. Meyerhenke: Realistic Computer Models. In: M. Müller-Hannemannand S. Schirra: Algorithm Engineering. Bridging the Gap between Algorithm Theory andPractice, LNCS 5971]


Real Architecture2

SpeedSize

Caches

Main Memory

Hard Disk

Registers< 1 KB

< 8 GB

10 ms

< 256 MB

1 ns

10 ns

5-70 ns

> 20 GB

2[D. Ajwani and H. Meyerhenke: Realistic Computer Models. In: M. Müller-Hannemannand S. Schirra: Algorithm Engineering. Bridging the Gap between Algorithm Theory andPractice, LNCS 5971]


Loop Unrolling & Tiling

saxpy = Single-precision real Alpha X Plus Y: α ·~x+~y (BLAS Level 1)

DO 50 I = MP1,N,4SY(I) = SY(I) + SA*SX(I)SY(I+1) = SY(I+1) + SA*SX(I+1)SY(I+2) = SY(I+2) + SA*SX(I+2)SY(I+3) = SY(I+3) + SA*SX(I+3)

50 CONTINUE

C uses row-major storage ⇒ double uses 64 bit per number ⇒ assume ourcache block size is 128 byte ⇒ cache block size of 16 numbers

................... ................ . . .

...

. . . ................ . . .

...................


ATLAS = Automatically Tuned Linear Algebra Software

make[5]: Leaving directory ‘.../tune/blas/gemm’SUCCESSFUL FINISH FOR ./xummsearchBEST USER CASE 241, NB=56: 11302.18 MFLOP

hedtke$ ./Test -O 1024+-----------------------------------------------------------------------+| TIME TEST FOR METHODS OF MATRIX MULTIPLICATION || || C = A*(8A), where A is a (n x n) random matrix with n = 1024 |+-----------------------------------------------------------------------+| method | time (sec) |+-----------------------------------------------------------------------+| NaivStandard 156 MFLOPS | 13.751396 || NaivOnArray | 16.325848 || NaivLoopUnrollingFour | 12.390910 || StrassenNaiv 1.9GB RAM | 4.275675 || WinogradOriginal | 12.824871 || BLAS 8 GFLOPS | 0.254061 |+-----------------------------------------------------------------------+


Strassen’s algorithm3

http://upload.wikimedia.org/wikipedia/commons/e/ef/Strassen_Knuth_Prize_lecture.jpg

A =[

a11 a12

a21 a22

]B =

[b11 b12

b21 b22

]

AB =[

h1 +h4 −h5 +h7 h3 +h5

h2 +h4 h1 +h3 −h2 +h6

]

h1 := (a11 +a22)(b11 +b22) h2 := (a21 +a22)b11

h3 := a11(b12 −b22) h4 := a22(b21 −b11)

h5 := (a11 +a12)b22 h6 := (a21 −a11)(b11 +b12)

h7 := (a12 −a22)(b21 +b22).

3[Volker Strassen: Gaussian Elimination is not Optimal, Numer. Math. 13 (1969),354–356.]







The Exponent of Matrix Multiplication

M(n) := number of field operations in characteristic 0 required to multiplytwo n×n matrices

ω := inf{r ∈R : M(n) =O(nr)

}

19

yields at best an exponent slightly below 2.85 (for n=34). Nevertheless, we will be

able to demonstrate all of the most important techniques for the asymptotic

acceleration of MM by modifying and improving the latter design. We will present

that design in a modified version as the bilinear algorithm (4.1),(4.2) below

although originally it appeared in the equivalent trilinear version and was applied

to the evaluation of one rather than two matrix products. We will end this section

by representing a diagram of the progress in the reduction of the exponent.

Diagram. (~(~) is the best exponent announced by time ~ .)

3,0"

2,8-

2.5-

2.0

coCO ~.-[ST69]

. . . . . . - " t

I ' I

1968 1969

oJ ( r ) [P80]

[P791 ,-q ~ "~z~...- [S80]

[R82] / [(3Wl (P80b]" '

t t

1 I I I I I I I I T ~

1975 1976 1977 1978 1979 1980 1981 1982

In the diagram the arrows indicate the time of the announcement of the new

exponents. For instance, the exponents 2.5167, 2.5161, and 2.496 were announced

during 1980 soon after each other; the exponents 2.548 and 2.522 were announced at

the Symposium 1979 on the Complexity Theory in Oberwolfach, West Germany, on October

24 and 26, respectively. These facts and the whole diagram demonstrate how

extremely rapid the progress was during 1978-1980. It was not clear at that time

how the different successful techniques of devising fast algorithms for matrix

multiplication were related to each other. In some cases the substantial reduction

of the exponents was achieved merely by appropriate combinations of the already

available different techniques. For these reasons, the history of fast MM is not

fully represented by the dates of the publication and even by the dates of the

announcement of the new exponents, so that we also refer the reader to our overview

[Victor Pan: How to Multiply Matrices Faster, LNCS 179, 1984.]


Coppersmith, Winograd and Williams

ω≤ 2.3755

Don Coppersmith and ShmuelWinograd4

ω≤ 2.3727

Virginia Vassilevska Williams5

http://www.ieee.org/about/awards/bios/kobayashi_recipients.htmlhttp://www.cs.berkeley.edu/~virgi/

4[Don Coppersmith and Shmuel Winograd, Matrix multiplication via arithmeticprogressions, J. Symbolic Comput. 9 (1990), 251–280]

5[Virginia Vassilevska Williams, Multiplying matrices faster than Coppersmith-Winograd,Proceedings of the 44th Symposium on Theory of Computing, STOC ’12]


http://www.ieee.org/about/awards/bios/kobayashi_recipients.html

http://www.cs.berkeley.edu/~virgi/

Part2Group-theoreticFast Matrix Multiplication

An Introduction


Cohn and Umans

Henry Cohn and Christopher Umans6

6[Henry Cohn and Christopher Umans, A Group-theoretic Approach to Fast MatrixMultiplication, Proceedings of the 44th Annual Symposium on Foundations of ComputerScience, 11-14 October 2003, Cambridge, MA, IEEE Computer Society (2003), 438–449]


basics: algebraic complexity theory⟨n,p,m⟩ := ⟨n,p,m⟩C⟨n,p,m⟩k : kn×p ×kp×m → kn×m, (A,B) 7→ AB

bilinear complexity

field k, fin. dim. k-VS U , V and W , k-bil. map η : U ×V → Wk-bil. algo.: 1 ≤ i ≤ r: fi ∈ U∗, gi ∈ V ∗, wi ∈ W :

η(u,v) =∑ri=1 fi(u)gi(v)wi ∀u ∈ U ,v ∈ V

(f1,g1,w1; . . . ; fr ,gr ,wr) is called k-bil. algo. of length r for ηR(η) := min. length of all k-bil. algo. for ηk-algebra A: R(A) := rank of its k-bil. multiplication map

restriction of a bilinear map

bil. maps φ : U ×V → W , φ′ : U ′×V ′ → W ′ restriction:∃ linear maps σ : U → U ′, τ : V → V ′, ζ′ : W ′ → W : φ= ζ′ ◦φ′ ◦ (σ×τ)we write “φ≤φ′” if ∃ restriction of φ′ to φFact: If φ≤φ′, then R(φ) ≤ R(φ′).


TPP basics

⟨n,p,m⟩ := ⟨n,p,m⟩C, ⟨n,p,m⟩k : kn×p ×kp×m → kn×m, (A,B) 7→ AB

R(η) rank of bilinear algorithmR(A) rank of algebraR(n,p,m) := R(⟨n,p,m⟩)R(n) := R(n,n,n)R(G) : R(C[G])

(AB)s,u =∑t∈T As,tBt,u ⇒

(∑s∈S,t∈T As,ts−1t

)(∑t̂∈T ,u∈U Bt̂,ut̂−1u

)Q(X) = {xy−1 : x,y ∈ X }

TPP: S,T ,U ⊆ G fulfill the TPP: s ∈ Q(S), t ∈ Q(T), u ∈ Q(u)stu = 1 ⇐⇒ s = t = u = 1

G realizes ⟨n,p,m⟩ :⇔ ∃ S,T ,U ⊆ G, |S| = n, |T | = p, |U | = mand S, T , U fulfill the TPP

In this case we call (S,T ,U) a TPP triple of G. Its size is npm.

TPP (subset) capacity: β(G) = max{npm : G realizes ⟨n,p,m⟩}{di} character degrees of G. r-character capacity: Dr(G) =∑

i dri


Part3Properties of the Triple ProductProperty and the Search forTriple Product Property Triples

A Summary


Properties of the TPP / TPP triples

If S, T and U fulfill the TPP then

[Murthy, arXiv:0908.3671] |S|+ |T |+ |U | ≤ |G|+2.

[Hedtke, arXiv:1101.5598v2] Q(X)∩Q(Y ) = 1 ∀ X 6= Y ∈ {S,T ,U}.

[Hedtke, arXiv:1101.5598v2] |Q(S)|+ |Q(T)|+ |Q(U)| ≤ |G|+2.

[Hedtke, arXiv:1101.5598v2] If S′, T ′ and U ′ fulfill the TPP, then there exists atriple S, T and U with |S| = |S′|, |T | = |T ′|, |U | = |U ′| andS∩T = T ∩U = S∩U = 1 which also fulfills the TPP.

http://www.queens.ox.ac.uk/academics/other-fellows/emeritus-fellows/peter-neumann/

Let G be a group. If (S,T ,U) is a TPP triple of G, then(dSa,dTb,dUc) is a TPP triple for all a,b,c,d ∈ G, too.

basic TPP triple: 1 ∈ S∩T ∩U

[P. M. Neumann, A note on the triple productproperty for subsets of finite groups, LMS J. Comput.Math. 14 (2011), 232–237.]





New Characterizations of the TPP[me & S. Murthy, Search and test algorithms for triple product property triples, Groups Compl. Crypt., Vol. 4 Issue 1 (2012), p. 111-133]

TheoremThree subsets S, T and U of G form a TPP triple (S,T ,U) iff

(i) 1 ∈ S∩T ∩U , (ii) Q(T)∩Q(U) = 1 and (iii) Q(S)∩Q(T)Q(U) = 1.

more general: (S1,S2,S3) is a TPP triple if and only if for all π ∈ Sym(3)

1 ∈ S1 ∩S2 ∩S3, Q(Sπ2 )∩Q(Sπ3 ) = 1, and Q(Sπ1 )∩Q(Sπ2 )Q(Sπ3 ) = 1.

TheoremLet G be a group and (S,T ,U) a basic TPP triple of subsets of G such thateither (i) two members, say S and T, are subgroups of G which permute, or(ii) one member, say S, is a normal subgroup of G. Then |S| · |T | · |U | ≤ |G|.


New Characterizations of the TPP

Let ; 6= C be a finite set and C = {C1, . . . ,Ck} a partition of it. A set X ⊆ C iscalled a subtransversal for C with support suppC(X) = T ⊆ C if for all Ci ∈ C

|X ∩Ci| ={

1 Ci ∈ T ,

0 otherwise.

special case: C = set of cosets of a subgroup S of a group G, then anysubtransversal T for S \ G will be called a subtransversal for S in G.

TheoremLet G be a group, S a subgroup of G, and T, U subsets of G.

1 If (S,T ,U) is a TPP triple of G then T and U are subtransversals for S inG such that

suppS\G(T)∩ suppS\G(U) = {S} (*)

2 If T and U are also subgroups of G, and T and U are subtransversals forS in G satisfying (*) then (S,T ,U) is a TPP triple of G.


Search for TPP triples

http://www2.informatik.uni-halle.de/da/hedtke/tpp/


http://www2.informatik.uni-halle.de/da/hedtke/tpp/

Theoretical insights obtained from experiments

Conjecture 7.4.

For any group G, βg(G) ≤ D3(G).

Conjecture 7.5.

Let D2n denote the dihedral group of order 2n.(i) If n is a multiple of 3, then β(D2n) =βg(D2n) = 8

3 n.(ii) If n is not a multiple of 3, then β(D2n) < 8

3 n and βg(D2n) = 2n.

Conjecture 7.6.

If G is a group with a cyclic subgroup of index 2, then β(G) ≤ 43 |G|.

TheoremThere is no group G that realizes ⟨3,3,3⟩ such that R(G) < 23, or that realizes⟨4,4,4⟩ such that R(G) < 49.


5×5 matrices (techn. rep. in preparation)

TheoremThere is no group G that realizes ⟨5,5,5⟩ such that R(G) < 100.

Suppose G realizes ⟨5,5,5⟩. If G has a subgroup H of index 2, then Hrealizes ⟨3,3,3⟩.If G realizes ⟨5,5,5⟩ and |G| ≤ 72, then G has no abelian subgroups ofindex 2.

If G is group with R(G) < 100 that realizes ⟨5,5,5⟩, then G isnon-abelian and 45 ≤ |G| ≤ 72.

No group G of order 64 fulfills R(G) < 100 and realizes ⟨5,5,5⟩.Search candidates: The final list contains ten groups of order 48 andtwo of order 54.

better search algorithm for ⟨k,k,k⟩ TPP triples


(group-theoretic) fast matrix...

Documents