mosscon 2013, cisco open source talk
DESCRIPTION
These are the slides that I presented at MOSSCon 2013 (slightly edited, because the original slides contained some animations that I morphed to look ok on Slideshare). The general talk is about two things: 1. General philosophy of open source at Cisco. 2. My specific open source work at Cisco. Enjoy!TRANSCRIPT
![Page 1: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/1.jpg)
Cisco Public 1© 2013 Cisco and/or its affiliates. All rights reserved.
Open Source for CiscoHigh PerformanceComputingDr. Jeffrey M. Squyres
![Page 2: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/2.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
My talk today
1. Who am I?
2. Cisco and Open Source
3. My Open Source work at Cisco
![Page 3: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/3.jpg)
Who am I?
Insert photo here
![Page 4: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/4.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Who am I?
Me
Technical Lead at Cisco Systems
Server division, VIC group
![Page 5: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/5.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Who am I?
I am not in marketing
![Page 6: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/6.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Who am I?
I cannot fix
your Linksys
router for you(perhaps you should try
DD-WRT)
![Page 7: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/7.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Who am I?
I write code
Lots of code
All day
Every day
Awwww… yeah
![Page 8: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/8.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
I work in Open Source
![Page 9: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/9.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
I work in these Open Source projects
![Page 10: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/10.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
I work in Open Source
Open MPI
Hardware Locality (hwloc)
OpenFabrics
Linux kernel
Vast majorityof my work
is here
I’ve mademinor contributions
to these other 3
![Page 11: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/11.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
My background story…
![Page 12: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/12.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
My background: research and academia
Undergrad, grad Post doc
![Page 13: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/13.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
My background: research and academia
LAM/MPI
I inherited this
I founded this
![Page 14: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/14.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Open MPIPACX-MPI
LAM/MPI
LA-MPI
FT-MPI
Sun CT 6
Project foundedin 2003,
merging multiple open sourceMPI projects
![Page 15: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/15.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
My own naievete:This is how I thought it would be
PACX-MPI
LAM/MPI
LA-MPI
FT-MPI
Sun CT 6
Me
![Page 16: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/16.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
…but this is what actually happened
PACX-MPI
LAM/MPI
LA-MPI
FT-MPI
Sun CT 6
Me
![Page 17: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/17.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
…and then this happened
PACX-MPI
LAM/MPI
LA-MPI
FT-MPI
Sun CT 6
![Page 18: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/18.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
…and then this happened
Us
![Page 19: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/19.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
What I learned
Differences=
Good
![Page 20: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/20.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
When you write code no one else will see…
You writeto your own
level ofexpectations
![Page 21: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/21.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
When your code is reviewed by your peers…
You writeto theirlevel of
expectations
![Page 22: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/22.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
When your code is reviewed by everyone…
You writeyour best code
![Page 23: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/23.jpg)
Insert photo here
Cisco and Open Source
![Page 24: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/24.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Open Source around Cisco
More than just the 4 projectsI participate in
Much
![Page 25: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/25.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Open Source around Cisco
More than just the 4 projectsI participate in
Much
![Page 26: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/26.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
Latest major open source initiative
After the clouds part,
you are left with…
![Page 27: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/27.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Latest major open source initiative
Major contributionto
![Page 28: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/28.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
OpenDaylight overview:SDN for everyone
![Page 29: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/29.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Pop quiz, hotshot
?Why does Cisco
do Open Source?
![Page 30: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/30.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
Pop quiz, hotshot
Why does Ciscodo Open Source?
IT MAKES GOOD BUSINESS SENSE
• Stand on the shoulders of giants• Become part of the community• Contribute to tools / ecosystem
that we all use• Sell more products
![Page 31: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/31.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
1. Stand on the shoulders of giants
![Page 32: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/32.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
1. Stand on the shoulders of giants
Weare
elevated
Weelevateothers
![Page 33: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/33.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
2. Become part of the community
Circle oftrust
you
![Page 34: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/34.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
2. Become part of the community
Circle oftrust
This is whereyou need
to be
![Page 35: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/35.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
2. Become part of the community
Insert FOSS project
name hereThis is where
you needto be
![Page 36: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/36.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
3. Contribute
Pretend this isa really gross
picture of a leech
![Page 37: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/37.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
3. Contribute
Pretend this isa really gross
picture of a leech
![Page 38: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/38.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
3. Contribute
Pretend this isa really gross
picture of a leech
Just say noto leeches!
![Page 39: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/39.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
3. Contribute
Let’s be clear here…
FOSS is for everyone
EVERYONE(that’s kinda the point, right?)
![Page 40: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/40.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
3. Contribute
Let’s be clear here…
FOSS is for everyone
EVERYONE(that’s kinda the point, right?)
So how can
there be leeches?
![Page 41: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/41.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
3. Contribute
I don’t contribute to everypiece of FOSS I use
Do you?
Of course not
![Page 42: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/42.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
3. Contribute
Those who can,should
![Page 43: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/43.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
3. Contribute
Big companies can contribute
![Page 44: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/44.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
3. Contribute
Individuals can contribute
![Page 45: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/45.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
3. Contribute
Small organizations can contribute
![Page 46: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/46.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
3. Contribute
Those who can,should
![Page 47: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/47.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
3. Contribute
Those who can,should
Submit a patch
Write a new pluginTest test test test testSubmit a GOOD bug report
Write documentation
Answer q
uestions
Suggest feature
s Evangelize
Send cash / beer
Use the software
Make schwag
Write a review
Review code
Obtain / provide funding
![Page 48: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/48.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
4. Sell more products
A giant
![Page 49: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/49.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
4. Sell more products
You, standingon the giant’sshoulders…
![Page 50: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/50.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
4. Sell more products…in thecircle
of trust
![Page 51: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/51.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
4. Sell more products …contributingto the community
![Page 52: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/52.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
Cisco UCSblade server
4. Sell more products
Cisco Nexus7000 router
![Page 53: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/53.jpg)
Insert photo here
My Open Source work at Cisco
![Page 54: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/54.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
Some background first…
![Page 55: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/55.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
High Performance Computing (HPC)
Using supercomputers to solvereal world problems that are
TOO BIGfor laptops, desktops, or individuals servers
![Page 56: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/56.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Generally speaking…
Supercomputer=
(Many) Racks of (commodity)high-end servers
(this is one definition; there are others)
![Page 57: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/57.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
A typical supercomputer
Rack of36 1U
servers
![Page 58: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/58.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
How does that work?
Computational problem
Input Output
Take your computational problem…
![Page 59: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/59.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
How does that work?
…and split it up!
Computational problem
Input Output
![Page 60: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/60.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
How does that work?
Computational problem
Input Output
Distribute the input dataacross a bunch of servers
![Page 61: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/61.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
How does that work?
Input Output
Use the network between serversto communicate / coordinate
![Page 62: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/62.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 62
How does that work?
Input Output
Use the network between serversto communicate / coordinate
![Page 63: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/63.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
How does that work?
Message Passing Interface (MPI)middleware is used for this communication
Input Output
![Page 64: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/64.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Why go to so much trouble?
Computational problem
One processor
hour
1 processor = …a long time…
![Page 65: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/65.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 65
Why go to so much trouble?
Computational problem
One processor
hour
One processor
hour
One processor
hour
21 processors = ~1 hour (!)Disclaimer: scaling is rarely perfect
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
One processor
hour
![Page 66: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/66.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
Coordination requires communication
This communicationmay happen a LOT
It therefore needsto be FAST
![Page 67: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/67.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
Fast communication
Sourceserver
Destinationserver
Network
![Page 68: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/68.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Let’s break it down
Source server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Destination server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Ethernet switch
Port A
Port B
![Page 69: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/69.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Source server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Destination server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Ethernet switch
Port A
Port B
Let’s break it down
200 nanoseconds
299,792,458 m/s c
~8 microseconds(modern hardware) ~8-40 microseconds
Total:~17 – 81
microseconds
~40 microseconds(older hardware)
![Page 70: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/70.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Really? Do we care about 17-81μs?
YES
![Page 71: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/71.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 71
Really? Do we care about 17-81μs?
• Intel Core i7 E5-2690 with turbo boost (3.5-3.8Ghz)“Sandy Bridge” 22nm processor
• LinX v0.6.4 (Linpack v10.3.4.007) benchmarkMeasures floating point operations per second
• 81.34 GflopsThat’s 81,340,000,000 floating point operations per second
17μs = 137,757,800 floating point operations
81μs = 656,375,400 floating point operations
Conclusion: yes, we absolutely care about 17-81μs!
Source: http://www.anandtech.com/show/4503/sandy-bridge-memory-scaling-choosing-the-best-ddr3
![Page 72: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/72.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Network latency matters
HPC apps can do a LOT of computationduring network communication
Latency
![Page 73: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/73.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
How can we reduce network latency?
Hardware is faster than software.
The sooner software canhand off to hardware, the better.
![Page 74: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/74.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Source server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Destination server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Ethernet switch
Port A
Port B
Let’s break it down
200 nanoseconds
299,792,458 m/s 299,792,458 m/s
~8 microseconds(modern hardware) ~8-40 microseconds
~40 microseconds(older hardware)
![Page 75: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/75.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Source server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Destination server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Ethernet switch
Port A
Port B
Let’s break it down
200 nanoseconds
299,792,458 m/s 299,792,458 m/s
~8 microseconds(modern hardware) ~8-40 microseconds
~40 microseconds(older hardware)
Can’t do much about the speed of light
![Page 76: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/76.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
Source server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Destination server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Ethernet switch
Port A
Port B
Let’s break it down
200 nanoseconds
299,792,458 m/s 299,792,458 m/s
~8 microseconds(modern hardware) ~8-40 microseconds
~40 microseconds(older hardware)
Can’t do much about the speed of light
Fastest Ethernet switches today are about 200ns(they’ll probably get a little faster over time)
![Page 77: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/77.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Source server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Destination server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Ethernet switch
Port A
Port B
Let’s break it down
200 nanoseconds
299,792,458 m/s 299,792,458 m/s
~8 microseconds(modern hardware) ~8-40 microseconds
~40 microseconds(older hardware)
Can’t do much about the speed of light
Fastest Ethernet switches today are about 200ns(they’ll probably get a little faster over time)
8-40us is, by far, the biggest chunk of timeReduce this!
![Page 78: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/78.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
Source server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Reducing server-side latency
What if we can skip some of these layers?
Who needs TCP? Raw L2 Ethernet frames, baby!
Who needs the operating system driver?
![Page 79: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/79.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Source server
HPC application
Operating system
MPI middleware
TCP stack
NIC driver
NIC hardware
Reducing server-side latency
What if we can skip some of these layers?
Who needs TCP? Raw L2 Ethernet frames, baby!
Who needs the operating system driver?
Let MPI talk directly to the NIC hardware
![Page 80: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/80.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Hardware usually only seen by the OS
Linux userspaceapplication
Linux kernel
Cisco VIChardware
Can I seethe hardware?
Please?
![Page 81: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/81.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
Hardware usually only seen by the OS
Linux userspaceapplication
Linux kernel
Cisco VIChardware
No.
![Page 82: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/82.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
…except in special cases
Linux userspaceapplication
Linux kernel
Cisco VIChardware
Can I seethe OpenFabrics
hardware?
![Page 83: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/83.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
…except in special cases
Linux userspaceapplication
Linux kernel
Cisco VIChardware
Sure!
![Page 84: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/84.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
…except in special cases
Linux userspaceapplication
Linux kernel
Cisco VIChardware
Can I seethe OpenFabrics
hardware?Yay!
![Page 85: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/85.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
OpenFabrics Linux support
• Coalition of network vendors
• Successfully upstreamed “OS bypass for networking” into Linux
• http://www.openfabrics.org
![Page 86: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/86.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
Source server
HPC application
Linux
MPI middleware
VIC driver
Cisco VIC
Enabling direct MPI communication
Our project: enabling this MPI direct-to-hardware communication on Cisco serverswith the Cisco Virtual Interface Card (VIC)
in Linux.
Everything above the firmware will beopen source.
![Page 87: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/87.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
Kernel
Cisco VIC hardware
TCP / IP stack
Cisco VIC driver
Normal TCP software architecture
UserspaceUserspace sockets library
MPI library
Application
![Page 88: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/88.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
Kernel
Userspace verbs library
Cisco VIC hardware
Cisco USNIC software
MPI library
Userspace
Verbs IB core
Cisco USNIC driver
Bootstrappingand setup
Send and receivefast path
Application
Cisco codehere
![Page 89: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/89.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
…and the 4th project
Hardware Locality (hwloc)
![Page 90: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/90.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Hardware Locality (hwloc)
• Query your server’stopology
• NUMA nodesIncluding memory
• Processor sockets
• L3, L2, L1 cachesInstruction and data
• Cores
• Hyperthreads
• PCI devices
![Page 91: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/91.jpg)
![Page 92: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/92.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
hwloc CLI
• Output formats supported:PDF, JPG, PNG, TIFF, FIG, …
Text (for console windows)
Curses
XML
• Great for feeding into scripts!
![Page 93: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/93.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 93
hwloc-bind: replaces numactl
• hwloc-bind socket:0.core:2 commandBind command to core 2 on socket 0
• hwloc-bind –getPrint a bitmap of your current bindings
• hwloc-bind --get | hwloc-calc -p -H socket.corePrint something more readable than a bitmap
• hwloc-bind --get | hwloc-calc -p -H socket.core.puEven show the hardware threads
• hwloc-ps [-a]Show where processes are bound
![Page 94: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/94.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 94
hwloc C API
• Get a tree data structure representing the topology
• Many API calls for manipulating / traversing the tree
• Typical actions:Get, set processor and memory bindings
React to cache sizes
• …everything you can do in the CLI, and more
![Page 95: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/95.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 95
Why is hwloc useful?
• Verify the internal topology of your serverHow much memory do you have?
Where is that memory?
What processor(s) are local to that memory?
How big are your L1, L2, L3 caches?
• Verify your internal PCI devicesDistinguish ethX devices from each other
![Page 96: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/96.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 96
Why is hwloc useful?
• Bind services to specific coresEnsure related services are on the same NUMA node
Put non-essential services on core 0 (e.g., NTP)
• Bind server-related servicesApache, Bind, NFS, …etc.
Increase performance by not letting them migrate
Keeps memory local, less inter-NUMA-node traffic
NTP etc.
Apache
NFS
![Page 97: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/97.jpg)
© 2013 Cisco and/or its affiliates. All rights reserved. Cisco Public 97
Cisco Public 9797© 2013 Cisco and/or its affiliates. All rights reserved.
“Open source is good.Open source works.
![Page 98: MOSSCon 2013, Cisco Open Source talk](https://reader033.vdocuments.us/reader033/viewer/2022042814/55515170b4c905f2288b558d/html5/thumbnails/98.jpg)
Thank you.