Keep Your Head in the
Clouds Blackboard’s Transition from Enterprise to Cloud Software Through DevOps
David AshmanChief Architect, Cloud Architecture@davidbashman
● The industry leader in educational technology and services
● 17 years young● Privately held● $450M revenue at time of going private● Headquarters in Washington, DC
● 17 year old codebase with roots in Perl● Millions of lines of Java code (Hybrid Java/Perl for
many years)● 7 development/operations offices, worldwide● 700 people between development, testing and
operations (All products)● ~3000 virtual machines across 1000 clients in
hosting● ~8PB of content and data storage
Blackboard Learn
We are a horse.
● 6+ month lead times● Technical debt● High update failure rates● Poor communication paths from development to
operations● Poor feedback loops from operations to
development
And like so many of you...
But through the art of DevOps...
We are a horse.
We are a horse.better
^
● Automation● Cloud Infrastructure● Culture
What changed?
Automation
LOC
# of code commits
Problem
24-36 Hours
Mike McGarr
LOC
# of code commits
Problem
LOC
Wat
# of code commits
Problem
LOC
# of code commits
15-30 Minutes24-36 Hours
But wait... There’s more!Functional Acceptance Testing
Value
5 minutes
CreateTicket
Commit/Build/
Integrate
15 minutes
5 minutes
Problem!
Code Test SuiteAnalyze Failed Tests
4 hours 1 hour 1 hour
5 minutes
36hours
24 hours
70 hours total time
6 hours (value) = 9% Efficient
Waste (wait)
AssignTicket
15 minutes
6 hours
DevelopmentTesting
(2-5 days)
Commit
TicketProject
Management
3-6+ days
100s of Failures
60% script issues(invalid tests, out of sync with functionality)
30% data/environment issues(data left in the database or filesystem)
7% pre-existing issues
3% newly discovered issues
Problem
● Elongated testing cycles (3+ months)● Longer time to market● Reduced visibility to our development team● Long delays between coding and fixing● Too much noise distracting us from product
improvements
The Problem
● TDD● Fully automated acceptance testing pipeline● < 30m acceptance test feedback● New testing approaches (Jasmine, Protractor,
RESTAssured)
Test Automation
6+ month lead time
1-2 week lead time
Cloud Infrastructure
OperationsProduct Development
Development Test Production
!= !=
● Snowflakes - No two environments were alike● Completely different deployment methods
o Manual development buildso Automated installs in testingo Gold master images in production
● Completely different deployment architectureso Windows development machineso Linux VMs in testingo Linux clusters in production
Deployment Environments
Development Operations
● Everything, from the ground up, is automated.● Orchestration in code - stored in SCC, executed
in ALL deployment environments.● On demand provisioning for developers.
o Still develop on their laptops, but can now test in a real deployment environment.
Learn in the Cloud
● OpenStack in our data centers● Abstract cloud fabric to gain benefits in both
clouds● Centralized, standardized Chef automation● Increased visibility for development
o Monitoring, APM, statsd, centralized logging
BbCloud
Culture
“QA is responsible for defining the testing strategy.”
“QA is responsible for checking quality when a feature is done.”
“Unit testing is not enough to verify a feature.”
OperationsDevelopment
Executive Buy-in
X
● Development teams deploying their production environments.
● Developers solving operational issues in production.
● Open feedback loops on operational issues.● Data-driven decisions based on that feedback.
What we’ve achieved
● Understand the impact that cloud computing can have on DevOps even in an “enterprise” company.
● Understand the cost models comparing cloud computing and traditional hosting.
● Frame your pitch to the executive team to make material changes in your company culture.
I might be able to help...
● Effective testing strategies for traditionally manual tests (UI, etc).
● Applying DevOps strategies for shipped-to-premise software.
I’d like to learn more about...
