Open Standards andOpen Source in
Datacenter Management
蔡鎮宇 Chen-Yu Tsai <[email protected]>
2014/4/11 OSDC 2014 1
Who am I?• Software Engineer @ CloudMosa, Inc.
• System Administrator for 10+ years starting in college
• Skills: breaking and fixing things
2014/4/11 OSDC 2014 2
Overview
•Monitoring
•Management
•Provisioning
2014/4/11 OSDC 2014 3
- Monitoring -
2014/4/11 OSDC 2014 4
Log Everything!
2014/4/11 OSDC 2014 5
Where to start?
2014/4/11 OSDC 2014 6
MRTG
2014/4/11 OSDC 2014 7
Based on SNMPSupported by most network devices
2014/4/11 OSDC 2014 8
Exports data and metrics
2014/4/11 OSDC 2014 9
Network traffic counters– used by MRTG
2014/4/11 OSDC 2014 10
Known MAC addresses- Map the network
2014/4/11 OSDC 2014 11
2014/4/11 OSDC 2014 12
2014/4/11 OSDC 2014 13
Whatever the device supports
Look up vendor specific MIBs
2014/4/11 OSDC 2014 14
RRDToolTime Series Database
2014/4/11 OSDC 2014 15
MRTG uses it
2014/4/11 OSDC 2014 16
Munin uses it
2014/4/11 OSDC 2014 17
… uses it
2014/4/11 OSDC 2014 18
Write your own!
2014/4/11 OSDC 2014 19
2014/4/11 OSDC 2014 20
2014/4/11 OSDC 2014 21
Munin –Resource Monitoring
2014/4/11 OSDC 2014 22
System is slow…
2014/4/11 OSDC 2014 23
CPU usage?
2014/4/11 OSDC 2014 24
2014/4/11 OSDC 2014 25
Memory usage?
2014/4/11 OSDC 2014 26
2014/4/11 OSDC 2014 27
Disk I/O?
2014/4/11 OSDC 2014 28
2014/4/11 OSDC 2014 29
Web requests?
2014/4/11 OSDC 2014 30
2014/4/11 OSDC 2014 31
Use plugins from standard set
2014/4/11 OSDC 2014 32
Or write Your Own!
2014/4/11 OSDC 2014 33
2014/4/11 OSDC 2014 34
Aggregate DataManual configuration for now
2014/4/11 OSDC 2014 35
2014/4/11 OSDC 2014 36
Others• Monitoring
• Xymon (Hobbit)
• Nagios
• Cacti
• Data collection / Graphing• Graphite
• ZipKin (Twitter)
• Log collection• Scribe (Facebook)
2014/4/11 OSDC 2014 37
Management
2014/4/11 OSDC 2014 38
IPMIIntelligent Platform Management Interface
2014/4/11 OSDC 2014 39
2014/4/11 OSDC 2014 40
Image from Wikipedia
Built into most BMCs
2014/4/11 OSDC 2014 41
Out-of-Bandvs
Side-band
2014/4/11 OSDC 2014 42
Power ControlOn, Off, Reset
2014/4/11 OSDC 2014 43
Serial over LANConsole Access
2014/4/11 OSDC 2014 44
Boot OrderForce PXE boot?
2014/4/11 OSDC 2014 45
SSHSecure Shell
2014/4/11 OSDC 2014 46
SSH Public Key Authentication
Don’t need to input password every time.
2014/4/11 OSDC 2014 47
OmniTTYConsole-based interactive SSH multiplexer
2014/4/11 OSDC 2014 48
Parallel-SSH (pssh)Parallel versions of OpenSSH
2014/4/11 OSDC 2014 49
FabricScriptable, Parallel SSH
2014/4/11 OSDC 2014 50
Provisioning
2014/4/11 OSDC 2014 51
DHCPNetwork Provisioning
2014/4/11 OSDC 2014 52
PXE BootBoot over Network
2014/4/11 OSDC 2014 53
Auto-configurationvia DHCP
Network Switches
2014/4/11 OSDC 2014 54
Kickstart/PreseedAutomatic Install
2014/4/11 OSDC 2014 55
ChefPuppet
Disclaimer: We don’t use them.
2014/4/11 OSDC 2014 56
Custom PackagesPut programs/services/settings
into native packages.
2014/4/11 OSDC 2014 57
Apt-cacher-ngWeb cache for package files
2014/4/11 OSDC 2014 58
Put It All Together
2014/4/11 OSDC 2014 59
2014/4/11 OSDC 2014 60
With the proper hardware/software
2014/4/11 OSDC 2014 61
Datacenters Become Manageable
2014/4/11 OSDC 2014 62
2~3 People2k+ Nodes in4 Datacenters
2014/4/11 OSDC 2014 63
Hands free afterracking and cabling
2014/4/11 OSDC 2014 64
2014/4/11 OSDC 2014 65
10k nodes?
2014/4/11 OSDC 2014 66
100k nodes?
2014/4/11 OSDC 2014 67
Evolve!
2014/4/11 OSDC 2014 68
We are Hiring!
2014/4/11 OSDC 2014 69
Thank You
2014/4/11 OSDC 2014 70