[@indeedeng] engineering velocity: building great software through fast iteration
Post on 25-May-2015
2.678 Views
Preview:
DESCRIPTION
TRANSCRIPT
Engineering VelocityBuilding Great SoftwareThrough Rapid Iteration
Dan HellerEngineering ManagerEmployer Applications
Indeed
#1 Job Site Worldwide
More people find jobs on Indeed than anywhere else
Over 100M monthly visitors
Over 3B monthly searches
≤
Rapid IterationLeads to
Better Products
Indeed Product Development Principles
Measure Everything
Test Everything (A/B Experiments)
Iterate Quickly
Build
MeasureLearn
Lean Startup Cycle
Indeed Product Development Cycle
Deliver
MeasureLearn
Indeed Product Development Cycle
Quickly!
Deliver
MeasureLearn
Indeed Product Development Cycle
MeasureLearn MeasureLearn
Deliver
Engineering Velocity
MeasureLearn MeasureLearn
Deliver
Engineering Velocity (n)[en-juh-neer-ing vuh-los-i-tee]
Our ability to quickly deliver implementations of ideas to job seekers & employers
Deliver Developer Productivity
Software Architecture
Processes and Tools
Risk Management
Deliver
Software Architecture
Processes and Tools
Risk Management
Deliver
Service Oriented Architecture
Job SearchWeb App
Doc Retrieval
Company Info
Authentication
Location
Index Search
Job SearchWeb App Index Search
Doc Retrieval
Company Info
Location
Authentication
Doc Retrieval
Company Info
Job SearchWeb App
Authentication
Location
Index Search
Technical Challenges
High Performance● Speed is a feature● 100ms response time
Technical Challenges
Forward & Backward Compatibility● Any client and any service can be updated at
any time
High Performance SOA Framework
Boxcar
Image used with permission of O Scale Trains Magazine (oscalemag.com), photo credit Don McFall of Old Line Graphics
http://go.indeed.com/boxcar
http://go.indeed.com/boxcar
Boxcar: Load Distribution without Middleware
Represent complex data types
Extremely compact and performant
Forwards and backwards compatible
Boxcar: API Compatibility
Boxcar: API Compatibility
Request and response data encoded as Protocol Buffers
Compatible evolution of service interfaces
Reduces release order dependencies
Doc Retrieval
Company Info
Job SearchWeb App
Authentication
Location
Index Search
Boxcar!
Service Oriented Architecturefor
Web Frontends
Each page hasdifferent featuresdifferent operational requirementsdifferent owners
Each page should be its own application
Trends
Company Info
Forums
Salary
Job Search
Indeed
Technical Challenges
Provide a consistent user experience
One implementation of shared functionality
Option: Shared Code Libraries
Reduced flexibility of technology decisions
Requires a release of all projects
Occasional inconsistent user experience
Navshell
Application-level Rewriting HTTP Proxy
Navshell
Request
Web App
HTTP
<head> <title>Johnson & Johnson Careers</title> <link type="text/css" href="common.css"> <link type="text/css" href="comp.css"></head><body> <div id="page-header"> ...... </div> <div id="page-body"> <div id="page-frame"> <div id="comp-header"> ...... </div> </div> </div></body>
<head> <title>Johnson & Johnson Careers</title> <link type="text/css" href="common.css"> <link type="text/css" href="comp.css"></head><body> <div id="page-header"> ...... </div> <div id="page-body"> <div id="page-frame"> <div id="comp-header"> ...... </div> </div> </div></body>
<head> <title>Johnson & Johnson Careers</title> <link type="text/css" href="common.css"> <link type="text/css" href="comp.css"></head><body> <div id="page-header"> ...... </div> <div id="page-body"> <div id="page-frame"> <div id="comp-header"> ...... </div> </div> </div></body>
REQUEST HEADERSCookie, User-Agent, Referer, etcX-Indeed-Shell: 1X-Indeed-Logged-In: [01]X-Indeed-Email: chris@indeed.comX-Indeed-Locale: en_USX-Indeed-Geo: US
Navshell
Web App
RESPONSE HEADERSSet-CookieX-Indeed-Page-Title: Indeed CareersX-Indeed-Include-Js-Footer: /js/app.jsX-Indeed-Include-Css: /style/main.css
Navshell
Web App
Navshell
No duplicate code
Centralized navigation changes
Plain HTTP with strong contract
Decoupled front-ends
Navshell
Requests
Services Web Apps
Web-Based Delivery of Mobile App Features
Mobile
Mobile Browser Mobile App
Mobile
Mobile Browser Mobile App
MobileBrowsers
MobileApps
DesktopBrowsers
Job Search App
● One code base for mobile web, iOS, Android● Simultaneous release of features across
platforms
Shared Code Across Platforms
● No waiting for App Store approval● No waiting for job seekers to upgrade
Release On Our Own Schedule
Easy To Run Experiments
● A/B testing in native apps extremely challenging
● Reuse existing infrastructure for performing and evaluating experiments
Software Architecture
Processes and Tools
Risk Management
Deliver
<Indeed map of the world>
one search. all jobs.
una búsqueda. todos los empleos.
Jedno vyhledávání. Všechna pracovní místa.
Ein Klick. Alle Jobs.
一站搜尋，工作齊全
tek arama. bütün işler.
uma busca. todos os empregos.
één klik. alle vacatures.
あらゆる仕事 まとめて検索
una ricerca. tutti i lavori.
Implement Commit
Deploy to QA
Deploy to ProductionVerify
Before Internationalization
Implement Commit
Translate
Deploy to ProductionVerify
Deploy to QA
CommitExtract
Internationalization Process Overhead
+1 day (20%)
+1 day (20%)
<Indeed map of the world>
<Indeed map of the world><but with only US in orange>
Machine Translation
English
Portuguese
Machine Translation
Low quality translations are better than no translations
But we needed something better
Human Translation Service
Tuesday afternoon: Commit Code Fetch Machine Translations
Human Translation Service
Tuesday afternoon: Commit Code Fetch Machine Translations Request Human Translations
Wednesday morning: Retrieve Human Translations
Implement Commit
Translate
Deploy to ProductionVerify
Deploy to QA
CommitExtract
Localization before Translation Automation
Implement Commit
Deploy to ProductionVerify
Deploy to QA
Extract
Localization with Translation Automation
Retrieve
Software Architecture
Processes and Tools
Risk Management
Deliver
Probability of issue
Duration of issue
Number of users affected
Risk
Releases
Smaller Releases
More releases → less functionality per release
Probability of issue
Monitoring
Duration of issue
Real-time monitoring with Datadog
● Agent reports summarized data
● System and customized business metrics
● Customizable dashboards
Healthcheck Framework
● Applications and services report internal state & health
● Divert traffic away from unhealthy instances● Disable features for failing dependencies
Monitoring and Alerting
● Identify runtime issues● Alert Ops when an issue arises
Proctor
● System for managing A/B experiments● Segment visitors and assign test groups● Manage feature visibility
Proctor
Manage Experiments
Proctor
Manage Experiments → Manage Behavior
Proctor
"rule" : "loggedIn"
"rule" : "${smartphone && 'JP' == country}"
"rule" : "${(ua.OS.family == 'ios' && ua.OS.majorVersion >= 6)}"
etc.
Proctor
if (proctor.isNewThingEnabled()) { newThing.generateResults();} else { oldThing.generateResults();}
Feature Toggles
Probability of issue
"rule" : "fn:endsWith(confirmedUserEmail, '@indeed.com')"
Decouple Features from Releases
Duration of issue
Rollback features and releases independently
Number of users affected
Small allocation changes
0% → 1% → 5% → 50% → 100%
Software Architecture
Processes and Tools
Risk Management
Deliver
Increased impact on products
Increased chance of success
No Failed Projects
No Failed Projects
Fail Fast
No Deadline Crunches
There’s always another release in a few days
More Transparent Schedules
Large changes hidden behind feature toggles
Incremental progress validated in production
environment
We built all of this incrementally, to meet our
own challenges
Yours are different
v = d / t
With rapid iteration & quick delivery, we can
cover more ground
Q&A
top related