the devops journey at bwin.party
TRANSCRIPT
Beginning&the&DevOps&Journey&in&Real&Money&
Gaming&
Kelly&Looney&27.11.2014&
bwin
Partygaming
DevOps in Real Money Gaming Context: 1B € company – Online Sports betting, Online Casino, Poker, other games Two primary technologies combined via a business merger
(turn of the century architecture) • Sports - .Net/SQL Server • Poker, Casino, and “Platform” – Java/Oracle
• Datacenters in Gibraltar, Vienna, and other points in Europe, now in US • 200 people in Ops and Infrastructure
• Development Centers in Vienna, Ukraine, and Hyderabad • Over 700 development team members
3
4
Different&faces/rules&For&different&markets&
Monolithic&App&with&many&single&points&of&failure&
In 2013…the Challenge
Dev&Teams&focused&on&Horizontal&Components&
Totally&separate&Ops,&Maintenance,&and&Dev&teams&
Clashing&cultures&from&Merger/LocaOons/&Code&bases&
Up&24/7&with&Millions&of&€&a&day&wagered&
InThouse&Build&Deploy&Monitoring&…&
What we have done and are doing…new ideas • Global Agile Transformation – classes, coaches, 90 Scrum teams
• Craig Larman, Luke Hohmann (Innovation Games) • DTO (Damon Edwards, Alex Honor) for DevOps principles
• Several Organization Changes • Components -> Features -> Services • Ops -> LeanOps -> Delivery Units
5
What we have done and are doing…tech • Tool changes
• SVN->Git, In house depoy -> Jenkins/Team City, Chef, Rundeck • Bare Metal -> VMWare -> Now headed to Docker/containers • AppDynamics for application monitoring – (amazing!)
• Architectural Principles • Less centralized, fewer failure points • Throw cheap machines, not faster CPUs or bigger DBs at scaling
problems • Use RDBs when needed otherwise avoid
6
Cultural changes we are encouraging • Old style Developers
• Responsibilities: Write code • Focus: Know ONE THING really really well. • Deep expertise = respect
• What we want now is Developers that: • Understand our company goals • Understand requirements and tests • Write, build, integrate, and test code incrementally • Can demonstrate and explain working systems • Maintains his/her code in production • Understands operations
Deep expertise is great, but varied knowledge is just as important 7
Wow, you want developers to do everything…
• First the right attitude…then • Todays Tools and Processes:
1. Scrum provides continuous “customer” access
2. Distributed versioning (typically Git) puts full source control into individual developers hands
3. Continuous Integration isolates mistakes
4. Jenkins-Vagrant-Puppet-Chef-Saltstack pipelines make infrastructure and deployment mostly automatic regardless of complexity
How is that possible?
8
Automated Tests
• It’s not “How many automated tests do I have?”
• We could have easily run days worth or tests whenever we wanted
• It’s “I have the right tests to quickly decide if I can move forward”
• Also BTW “We run Jenkins to do a build every night”
• Does != Continuous Integration…
9
Containers are changing hosting • Virtualization efficiency and cost savings are obvious
• The most interesting issue is the separation of concerns presented
• “developer-land” vs infrastructure 10
Roundabout: The Bwin.Party Data
Center of the Future
Goals:1. Proven Failover and DR
Recent events certainly put this in focus...
2. Frequent Releases with Rollback
Our Agile/DevOps goal for modern development and delivery
Blue/Green/Yellow
— Three full production replicas
Blue = Live
Green = Staging/Rollback
Yellow = Failover
— DB replication across sites
Roundabout Process
1. Live data center hosts our applicationsSwitch to Failover site can be initiated at any time
2. Staging is used to build and do final test on a new release
3. Staging is switched to Live4. Live becomes Rollback.
Roundabout Process cont.
1. The new Live is cloned to Failover - Rollback available if needed
2. When we feel good about Live we can start changing Rollback to a new Staging
Assumption: DB replication supports all this switching without loss
In Conclusion: DevOps for large enterprises is possible!
Just really f-ing hard…