does14 - david ashman - blackboard learn - keep your head in the clouds

Keep Your Head in the

Clouds Blackboard’s Transition from Enterprise to Cloud Software Through DevOps

David AshmanChief Architect, Cloud Architecture@davidbashman

● The industry leader in educational technology and services

● 17 years young● Privately held● $450M revenue at time of going private● Headquarters in Washington, DC

● 17 year old codebase with roots in Perl● Millions of lines of Java code (Hybrid Java/Perl for

many years)● 7 development/operations offices, worldwide● 700 people between development, testing and

operations (All products)● ~3000 virtual machines across 1000 clients in

hosting● ~8PB of content and data storage

Blackboard Learn

We are a horse.

● 6+ month lead times● Technical debt● High update failure rates● Poor communication paths from development to

operations● Poor feedback loops from operations to

development

And like so many of you...

But through the art of DevOps...

We are a horse.

We are a horse.better

^

● Automation● Cloud Infrastructure● Culture

What changed?

Automation

LOC

# of code commits

Problem

24-36 Hours

Mike McGarr

LOC

# of code commits

Problem

LOC

Wat

# of code commits

Problem

LOC

# of code commits

15-30 Minutes24-36 Hours

But wait... There’s more!Functional Acceptance Testing

Value

5 minutes

CreateTicket

Commit/Build/

Integrate

15 minutes

5 minutes

Problem!

Code Test SuiteAnalyze Failed Tests

4 hours 1 hour 1 hour

5 minutes

36hours

24 hours

70 hours total time

6 hours (value) = 9% Efficient

Waste (wait)

AssignTicket

15 minutes

6 hours

DevelopmentTesting

(2-5 days)

Commit

TicketProject

Management

3-6+ days

100s of Failures

60% script issues(invalid tests, out of sync with functionality)

30% data/environment issues(data left in the database or filesystem)

7% pre-existing issues

3% newly discovered issues

Problem

● Elongated testing cycles (3+ months)● Longer time to market● Reduced visibility to our development team● Long delays between coding and fixing● Too much noise distracting us from product

improvements

The Problem

● TDD● Fully automated acceptance testing pipeline● < 30m acceptance test feedback● New testing approaches (Jasmine, Protractor,

RESTAssured)

Test Automation

6+ month lead time

1-2 week lead time

Cloud Infrastructure

OperationsProduct Development

Development Test Production

!= !=

● Snowflakes - No two environments were alike● Completely different deployment methods

o Manual development buildso Automated installs in testingo Gold master images in production

● Completely different deployment architectureso Windows development machineso Linux VMs in testingo Linux clusters in production

Deployment Environments

Development Operations

● Everything, from the ground up, is automated.● Orchestration in code - stored in SCC, executed

in ALL deployment environments.● On demand provisioning for developers.

o Still develop on their laptops, but can now test in a real deployment environment.

Learn in the Cloud

● OpenStack in our data centers● Abstract cloud fabric to gain benefits in both

clouds● Centralized, standardized Chef automation● Increased visibility for development

o Monitoring, APM, statsd, centralized logging

BbCloud

Culture

“QA is responsible for defining the testing strategy.”

“QA is responsible for checking quality when a feature is done.”

“Unit testing is not enough to verify a feature.”

OperationsDevelopment

Executive Buy-in

X

● Development teams deploying their production environments.

● Developers solving operational issues in production.

● Open feedback loops on operational issues.● Data-driven decisions based on that feedback.

What we’ve achieved

● Understand the impact that cloud computing can have on DevOps even in an “enterprise” company.

● Understand the cost models comparing cloud computing and traditional hosting.

● Frame your pitch to the executive team to make material changes in your company culture.

I might be able to help...

● Effective testing strategies for traditionally manual tests (UI, etc).

● Applying DevOps strategies for shipped-to-premise software.

I’d like to learn more about...

Thanks!dashman@blackboard.com

@davidbashman