osdc 2014: thomas schend - introduction to synnefo open source cloud stack
DESCRIPTION
This talk wants to introduce you to Synnefo, an open source, scalable and production ready cloud stack. It consists of a vm management layer which is google ganeti and is essentially a cluster manager and delivers an integrated management of compute, network and storage. It also runs on of the shelf hardware and delivers live migration without or with shared storage. Synnefo is the orchestration and presentation layer which talks to ganeti via an API. To the users it presents a simple web UI and also exposes an openstack compatible API for automation called cyclades. Also it is possible to do flexible, L2 isolated networking. Also it provides a “dropbox” like storage services called pithos. It features a sync client for different platforms and a web UI. Synnefo offers all the benefits of an amazon like cloud but is geared towards persistent virtual machines. Also it is perfect as a replacement of the traditional virtualization stack and is easy to setup and use.TRANSCRIPT
04/10/2023 OSDC 20141
Introduction to Synnefoopen source cloud stackThomas Schend, Systems Architect, CompuGroup Medical AG
Open Source Data Center Conference 2014
08. – 10. April, Berlin
04/10/2023 OSDC 20142
Agenda
IntroductionWhat do we need from IaaS?Introducing to SynnefoComponentsGanetiDemoQ & A
OSDC 2014
successfully worldwide in 35 countries for approx. 385,000 customers
10.04.20233
= 19 countries with own offices = additional 16 countries with customers
2012 Guidance 2013Customers 385.000
Group sales € 451 Mio. € 458 – 463 Mio.
EBITDA € 105 Mio. € 97 – 100 Mio.
employees 3.700
Indicators
Best provider of IT solutions for higher quality and efficiency in health care
Worldwide leader for MIT (Medical Information Technology)
High profitability and an average of 30% growth p.a.
Objectives
04/10/2023 OSDC 20144
Bio
Systems Architect, responsible for Infrastructure DesignOSS EnthusiastLPIC-2,MCITP,VCP,CCA,DCIE,…
Contact: [email protected]
04/10/2023 OSDC 20145
Agenda
IntroductionWhat do we need from IaaS?Introducing to SynnefoComponentsGanetiDemoQ & A
04/10/2023 OSDC 20146
What do we need from IaaS?
ReliableScalable (Backend and VMs)Easy to manageLong lived workloads, not all VMs are EC2 instances ;-)
04/10/2023 OSDC 20147
“VMs are not cattle, they are pets”
Vangelis Koukis, Technical Lead, Synnefo
Quote
04/10/2023 OSDC 20148
Agenda
IntroductionWhat do we need from IaaS?Introducing to SynnefoComponentsGanetiDemoQ & A
04/10/2023 OSDC 20149
Synnefo
Open SourceProduction-quality IaaSmade for persistent VMscommodity hardware – No SANLarge-scale Installationssimple UI for the end-useruses Ganeti as a Cluster Manager
04/10/2023 OSDC 201410
Synnefo History
Design started late 2010Production since July 2011~okeanos* public cloud running currently ~8700 VMs
*http://okeanos.grnet.gr
04/10/2023 OSDC 201411
Synnefo Layers
Image provided by GRNET
04/10/2023 OSDC 201412
Synnefo birdseye view
Image provided by GRNET
04/10/2023 OSDC 201413
Synnefo
Build with Ganeti as a backendAstakos: Identity/Account servicesCyclades: Compute/Network/Image/Volume servicesPithos: File/Object Storage servicesnf-manage commandline toolBasic helpdesk toolsnf-burnin / snf-burnin-runkamaki commandline clientCode on github
04/10/2023 OSDC 201414
Agenda
IntroductionWhat do we need from IaaS?Introducing to SynnefoComponentsGanetiDemoQ & A
04/10/2023 OSDC 201415
Synnefo Architecture I
Image provided by GRNET
04/10/2023 OSDC 201416
Synnefo features - Astakos
Identity serviceProvides DashboardTrack Resource usageProject ManagementMultiple Auth methods
Twitter / Google / LinkedIn / LDAP / Active-directory
04/10/2023 OSDC 201417
Synnefo features - Cyclades
Network managementL3 Routed/L2 VLANS/L2 isolation with mac filternfdhcpd
Archipelago storage abstractionceph/NFS/Gluster backendvery fast, thin cloningSnapshots
04/10/2023 OSDC 201418
Synnefo features - Pithos
Object storage serviceEfficient syncingHashing before uploadDeduplication in backendWeb UI written in GWTClients for Windows, Macos and IOSSharing public and private
04/10/2023 OSDC 201419
Synnefo / Ganeti Communication
Image provided by GRNET
04/10/2023 OSDC 201420
written in python / djangoapache/gunicornpostgres databasesnf-gnt-eventd / snf-progress-monrabbitmqsnf-dispatchersnf-vncauthproxybuild for wheezy, can run fully virtualizedspeaks openstack API but does not share any code
Synnefo Architecture II
04/10/2023 OSDC 201422
Secure image deployment of untrusted imagesIsolated from the ganeti nodeSupports Linux,Windows and BSD
customization tasksAssign hostnameDelete SSH keysChange passwordResize FilesystemRun unattended install (windows)Inject files
Synnefo - snf-image
04/10/2023 Central IT Services23
Synnefo snf-image deployment process
gnt-node evacuate -I hail node3
Imag
e pr
ovid
ed b
y G
RNET
04/10/2023 OSDC 201424
Agenda
IntroductionWhat we need from IaaS?Introducing to SynnefoComponentsGanetiDemoQ & A
04/10/2023 OSDC 201425
Ganeti History
Very mature, production readyopensourced since August 30, 2007 by googleuses Mature open source tools LVM,DRBD,python,haskellDifferent hypervisors (Xen,KVM,LXC)Low overhead for cluster management
04/10/2023 OSDC 201426
Ganeti Usage at google
Usage at googlesmall Clusters in Offices (OIB)large Clusters in Datacenters for virtual desktops
mostly using XENNot used for user facing services
04/10/2023 Central IT Services27
Ganeti features I
Scale from 1-200 nodes and beyondLive migration without shared storageNoderoles: master,master candidate,regular nodesNodegroupsOVF import toolBackup with included toolJob queueCommands start with gnt-
04/10/2023 OSDC 201428
Ganeti features II
OS Interface to provision new instancessnf-image secure image deploymentganeti-instance-debootstrap, image, ganeti-os-defsHtools to balance clusters, plan capacity and do maintenance
supported file,plain,sharedfile,ceph,NFS,Gluster Ext storage provider with plugins for Archipelago, HP EVA,,IBM Storwize
04/10/2023 OSDC 201429
Ganeti advanced features
Hooks for every action (pre/post)Tags for every objectExclusion tags to keep machines apart from each otherDeploy flash in every node for high performance (flashcache,bcache,dm-cache)OOB management interfaceGaneti watcher
04/10/2023 OSDC 201430
Ganeti advantages
No need for special hardware (SAN)But SAN/Shared Storage is supportedManage cluster as an entity not single nodes (libvirt)Few dependenciesNo databaseEasy recovery in case of hardware outageNo SPOF which can affect the reliability of VMsVery good documentation
04/10/2023 OSDC 201431
Deployment automation puppet,chef,cfengine,..Monitoring icinga/nagios/check_mk/…Openvswitch LACP + 10GBeAutorepairPackages for all major OS debian,ubuntu,rhel/centos
Ganeti best practices
04/10/2023 OSDC 201432
Ganeti management interfaces
cli with bash completionLUXIRAPI
Different web UIs using the RAPIGaneti web managerGanetimgrSynnefo
04/10/2023 OSDC 201433
Ganeti quick command reference
gnt-cluster init / info / verify / command / copyfile
gnt-node add / list / failover / evacuate
gnt-instance add / info / replace-disks
gnt-job list …
gnt-network …
gnt-backup …
…
04/10/2023 OSDC 201434
Ganeti htools - allocation and placement
written in haskell
hail
hbal
hspace
harep
hinfo
hroller
…
04/10/2023 OSDC 201435
Typical Ganeti Clsuter
Node01 Node02 Node03
gnt-cluster init [-s ip] --enabled-hypervisors=kvm clustergnt-node add [-s ip] node2gnt-node add [-s ip] node3gnt-cluster verifygnt-instance add -t drbd -I hail -o debootstrap+default vm
Veea
m v
irtua
lizati
on st
enci
ls
04/10/2023 OSDC 201436
Ganeti Failover and Repair
gnt-node modify -O yes node01gnt-node failover --ignore-consistency node01gnt-node evacuate -I hail node01
Node01 Node02 Node03
Veea
m v
irtua
lizati
on st
enci
ls
04/10/2023 OSDC 201437
“Ganeti didn't have any generally known meaning before this project. And now it means kick-ass cluster management”
Martin Beauchamp, on the Ganeti Users List
Quote
04/10/2023 OSDC 201438
Agenda
IntroductionWhat do we need from IaaS?Introducing to SynnefoComponentsGanetiDemoQ & A
04/10/2023 OSDC 201439
Login to Astakossyncing and Pithos UI and AppCreate VMs with Cyclades UISnapshot FeatureOOB ConsoleHotplug of NICs and NetworksFloating IPsClean-up
Demo
04/10/2023 OSDC 201440
Agenda
IntroductionWhat do we need from IaaS?Introducing to SynnefoComponentsGanetiDemoQ & A
04/10/2023 OSDC 201441
http://www.synnefo.org
https://github.com/grnet/synnefo
https://code.google.com/p/ganeti/
Try it!
04/10/2023 OSDC 201442
Q & A