all things api presentation - gordon weakleim [homeaway]
TRANSCRIPT
HomeAway’s Core APIOr how I learned to stop worrying and love the abstraction layer…
My History
Gordon Weakliem Professional dev for 21 years Mostly in the travel industry CORBA, COM, SOAP, REST… I started with VRBO.com in February,
2007.
HomeAway’s History
Founded in 2005
Went public in June 2011 (NASDAQ:AWAY)
20+ Acquisitions Mostly full-stack “listing sites” Diverse tech stacks and teams Old code bases World-wide dev teams
Complex Marketplace + Products
VRBO’s History
Started in 1995 advertising our founder’s ski condo.
By 2006, we were a mix of ASP and ASP.NET in a roughly 3 tier architecture talking to SQL Server.
At the time of the HomeAway acquisition in 2006, there were 4 full time developers. This grew to a manager, 8 dev + 2 QA in the next couple years.
VRBO had 65,000 active listings and was growing 100% YOY.
Growth through acquisition
We’re a long way from advertising a ski condo from a basement server.
How do you integrate all these companies?
Platform Evolution
Pub/Sub – initial consolidation strategy Solves distribution – enabled
growth
Evolution to platform Apps need single (logical) SoR per
resource type Apps need a clean abstraction Must interop with 200+ existing
apps
Growing Pains
During the pub-sub era: The architecture was baroque. It was really painful to figure out why things went
wrong. Listing sites had to implement missing
infrastructure themselves – administration, caching. Feature set wasn’t unified across brands.
There were some shared services (inquiries, reviews) but each one implemented their own provisioning, interface standards, and API conventions.
HomeAway’s Core API
REST API Unified Architecture XML/JSON over HTTP (polyglot access) Many teams contributing
Façade over existing systems
Java/Spring/Jersey/Mybatis/MS SQL
REST API!
Why? HTTP methods – standard verbs Resources – standardize nouns URLs – standardize references Layered – enterprise features globally Focused on interface/contract, not on
implementation
api.homeaway.com
API Architecture
API endpoint
API endpoint
API endpoint
Legacy app
Legacy svc
Legacy svc Legacy app
app app app
cacheoauth
versioningetc…
Access routed through a single hostname at the
load balancer
Endpoints service a set of nouns, routed by URL path
Enterprise features
layered in across the API
API endpoints coexist with legacy apps and services
The “Grand Vision” of Platform
We designed the API we wanted but couldn’t build (right away)
We built it anyways As a façade to the existing systems
New App development benefits Minimized Risk for Legacy Sunsets
API -> Platform Refactoring
API
Application
ApplicationApplication
REST – Constraints (a refresher)
Client-Server
Stateless
Cacheable
Layers
Uniform Interface!! Urls Manipulation mime-types HATEOAS
Code on Demand
REST - Representation
Urls/{entityCollection}/{systemCode}/
{entityUuid}
Resource Oriented Noun centric (/listings, /accounts)
REST - Representation
HATEOAS / Strongly Linked
REST - Representation
Resource representations are XML / JSON
Defined by an XSD .NET, ruby, python, javascript bindings
Declarative Validation
REST - Representation
Versioning X-HomeAway-ApiVersion: 1.81 Version “ratchet” via bi-directional
XSL chain
REST - Representation
Version=1.2
Version=1.3
XSL
XSL
REST – State Transfer
Etags Versioning Etag: “34310538”
Etags Optimistic Concurrency If-Match: “34310538”
REST – State Transfer
Caching Conditional GETs
If-None-Match: “34310538” Expires: Thu, 29 Aug 2013 04:45:54 GMT
Last-Modified: Wed, 28 Aug 2013 18:00:00 GMT
Cache-Control: max-age=60, must-revalidate
Authentication
OAuth2-based Authentication
Authorization / ACL
Self-serve provisioning for credentials Access to certain features requires a
review
Implemented as a Filter (layered!)
api-framework
Large company 250+ developers
APIs existed before our “grand vision”
We could not have got this far without a good foundation.
api-framework
Rails, Sinatra, Django, web.py, JAX-RS are general purpose web frameworks
Teams develop unique philosophies Organization URL Structure Media
api-framework
“There should be one – and preferably only one – obvious way to do it.”
Consistency
API developers can easily work with(and on) any API endpoint
URL Structure
Entity Collection
System Code
Entity UUID
SubResource
URL Structure
/listings
/listings/0000 /listings/0000/af6ffb31-ec1b-47b4-8f55-
9cc33d0341dd
/listings/0000/af6ffb31-ec1b-47b4-8f55-9cc33d0341dd/hitCounter
/listings/legacy?id=vrbo/980 -> an entity
URL Structure
Opaque – “thou shalt not reason about a URL’s structure”
Provides an obvious pattern of organization
Confined vocabulary directly related to features in the framework
Interface Contracts
There’s a fundamental client/server mismatch Clients reason about APIs at the HTTP
level Developers are deeply nested in
implementation
You could just scrape request vars…
But it’s better to define an interface
Mismatch
Easy to make to assumptions that are not shared with clients
Leads to backwards incompatibilities
Design By Contract
Force API developers to think about the “contract of invocation” Preconditions / Post conditions Fine grained requirements
Statically declared contracts
Developers do not enforce contracts
Design By Contract
Contract tightly coupled with the interface
Change is not “free” Forces developers to think about
backwards compatibility
Contract Documentation
Humans like this form…
HomeAway API Stats
8,266,885,627 calls week ending 2015-03-09 From 264,570,522 week ending 2013-
03-04
99.3% of all calls < 250ms
100+ internal clients
Monitoring for errors/SLA across API
Adoption
Operational Concerns
We rely on this API Outages are a big deal We need to be able to develop new features
quickly
A lot of people are working on this It can’t be a free-for-all
REST isn’t the end-all. Sometimes it’s inefficient Polling Following links
Filters
HomeAway ops filter
Jersey Servlet
CachingVersioningOAuthAuditing
AuditingVersioningCaching
api-frameworkcode code app
Endpoint developers code java methods
here
App request/response goes through many layers of functionality
Analytics
We use Splunk, heavily
Framework log format is the same for everyone. Splunk queries work for all core services.
HomeAway has an entire Big Data team.
Big Data is kind of a big deal at HomeAway
Caching
We handle a LOT of traffic
Connecting consumer endpoints to core API is operationally challenging.
In-memory caching on each box
Elastic Search to support distributed caching
Developers must enable caching per endpoint
Eventing (AMQP/RabbitMQ)
POST/PUT/DELETEs are events
API Endpoints publish events to AMQP topic
Consumers bind Queues to the topic
Entity Search Service
Before, developers wrote /findByXXX methods to support search functions
ESS – based on Elastic Search Dev annotates indexable Indexer job automatically builds indexes Index exposed under /search No free lunch – we have to maintain strict
policies around adding new indexes
Search Examples
https://api.homeaway.com/reservations/[email protected] https://api.homeaway.com/reservations/search?
q=contact.email=%[email protected]%22
https://api.homeaway.com/reservations/findByDate?url=/druids/0060/ef151f3b-ad64-423d-9102-10fccac084fa&start=2015-01-01&end=2015-03-31 https://api-stage.homeaway.com/reservations/search?
q=reservationDates.beginDate=2015-01-01%20AND%20reservationDates.endDate=2015-03-31%20AND%20propertyProfile.druid.rel=%22/druids/0060/ef151f3b-ad64-423d-9102-10fccac084fa%22
Tool Support
We have a dedicated Developer Tools team Manages things like github, Team City, Jira,
deployments
End to end support, from starting the project to deploying to production. Central repository for application
registration Developers can self-serve deployment all
the way to production.
BizOps
Our platform needs are evolving
BizOps are the next level above the domain API
Intended an an intermediate integration point for services that need to combine data from several core API entities.
JSON is the default format because the primary client is an AJAX client – these are consumed by our consumer-facing sites.
Summary
Domain-specific framework provides a consistent organizational pattern
Layers permit adding functionality without changing code in multiple projects
Summary
First-class contracts reduce risk of backwards incompatibility
Derive documentation from contracts
Dedicated Tools & Data teams
Operational concerns drive certain compromises