building consistent restful apis in a high-performance environment
DESCRIPTION
This is one of two presentations given by LinkedIn engineers at Java One 2009. This presentation was given by Brandon Duncan, Director of Engineering, and Yegor Borovikov, Software Architect at LinkedIn.For more information, check out http://blog.linkedin.com/TRANSCRIPT
Building Consistent RESTful APIs in a High-Performance Environment
Yegor Borovikov, Software Architect
Brandon Duncan, Director of Engineering
LinkedIn Corporationhttp://blog.linkedin.com/
2
Topics We’ll Cover
> Examples of RESTful APIs What’s missing? Variety versus Uniformity
> Domain Model as Foundation Provides Uniformity Allows Flexibility
> Examples of LinkedIn APIs> Using Incentives to Scale
Some LinkedIn production APIs metrics> Q & A
3
Examples of RESTful APIsWhat if you want to get a person’s profile?
> Use one of these…http://social.yahooapis.com/v1/user/{guid}/profilehttp://api.linkedin.com/v1/people/{guid}http://www.orkut.com/social/rest/people/{guid}/@self<?xml version="1.0" encoding="UTF-8"?><person> <id>111222</id> <first-name>Gertrude</first-name> <last-name>Stein</last-name> <headline>Author of Tender Buttons</headline> <connections total="76"> …</person>
4
Examples of RESTful APIsWhat’s Missing?
> Ability to get exactly what you need (variety) If you need more, may require multiple
API calls (if they exist) If you need less, resources are wasted
> Consistency of responses (uniformity) “Same” object returned by different APIs
may have different structure Once in production, hard to get consistent later
5
Examples of RESTful APIsMultiple Calls to Get What You Need
> Want to get user’s friend’s profile? Do this… http://social.yahooapis.com/v1/user/123/connections
<connections yahoo:start="0" yahoo:count="1" yahoo:total="1"> <connection yahoo:uri="http://social.yahooapis.com/v1/user/123/connection/456?view=usercard"> <guid>456</guid> <contactId>4</contactId> </connection></connections>
6
Examples of RESTful APIs Multiple Calls to Get What You Need
> … then make second call to get friend’s profile:http://social.yahooapis.com/v1/user/456/profile
<profile yahoo:uri="http://social.yahooapis.com/v1/user/456/profile"> <guid>456</guid> <birthdate>3/3</birthdate> <created>2008-08-4T17:13:56Z</created> ...</profile>
Latent, redundant data Optimization requires stickiness
7
Typical SolutionVariety versus Uniformity
> Solution: introduce another call
> Desire for variety of responses undermines uniformity of requests
> Leads to RPC-like REST APIs
> Many APIs + Great Documentation =Lots of Reading + Lack of Automation
8
Domain Model as Foundation Sample Domain Model /people : Person[] // collection of Person resources /id : string // primary key /name : string /email : string // unique key /photo : url /best-friend : Person /friends : Person[] /jobs : Job[] // collection of Job resources /company : Company /title : string /start-date : date /end-date : date … /companies : Company[] /name : string /ceo : Person …
9
Domain Model as Foundation Follow request URL to navigate through your model
> To get a person’s profile:http://api.linkedin.com/v2/people/123
<person uri=“urn:linkedin:v2:people/123” key=“123”> <id>123</id> <name>Reid Hoffman</name> <email>[email protected]</email> <best-friend uri=“urn:linkedin:v2:people/456”/> …</person>
Conventional URL in request Default representation in response
/people[/id=123] /id /name /email /photo /best-friend /friends /jobs /company /title /start-date /end-date … /companies /name /ceo …
10
Domain Model as Foundation Fine-grained Request
> What if you only need certain fields(e.g., name and photo)?http://api.linkedin.com/v2/people/123:(name,photo)
<person> <name>Reid Hoffman</name> <photo>http://media.linkedin.com/photos/123.jpeg</photo></person>
/people[/id=123] /id /name /email /photo /best-friend /friends /jobs /company /title /start-date /end-date …
11
Domain Model as Foundation Fine-grained Request
> To get names and photos of one’s friends and their best friends:…/v2/people/456/friends:(name,photo,best-friend:
(name,photo))<friends total=“66” start=“0”> <friend uri=“urn:linkedin:v2:people/123” key=“123”> <name>Reid Hoffman</name> <photo>http://media.linkedin.com/photos/123.jpeg</photo> <best-friend uri=“urn:linkedin:v2:people/456” key=“456”> <name>Brandon Duncan</name> <photo>http://media.linkedin.com/photos/456.jpeg</photo> </best-friend> </friend> <friend>…</friend></friends>
/people[/id=456] /id /name /email /photo /best-friend /friends /123 /id /name /email /photo /best-friend /name /photo /jobs …
12
Domain Model as Foundation Fine-grained Request
> Allows client to construct custom calls
> Better than digging for the closest matching API:http://social...com/v1/user/123/profilehttp://social...com/v1/user/123/profile/usercardhttp://social...com/v1/user/123/profile/tinyusercard
> Allows optimization on the backend
13
Domain Model as Foundation Benefits
> Provides a frame for both request and response semantics
> Still allows for flexible syntax Requests – path, query params, matrix params… Responses – JSON, XML, POJOs, protobuff…
> Helps to unify and automate many development tasks on both ends Request / response creation, parsing, marshalling Code (and documentation) generation Discovery services
14
Examples of LinkedIn APIs HTTP GET - Read
…/people/[email protected]/friends?sort=name
…/people/123/friends;sort=name:(name,jobs;sort=start-date)
…/people:(id,name,photo)?name=page&company=google
…/people::(123,456)…/people::(123,456):(name,photo)
15
Examples of LinkedIn APIs HTTP PUT - Update
> Set the user’s name:PUT http://api.linkedin.com/v2/people/123/name<name>Reid Hoffmann</name>
> Update the user’s profile - change name and best-friend and remove photo:
PUT http://api.linkedin.com/v2/people/123<person> <name>Reid Hoffman</name> <best-friend uri=“urn:linkedin:v2:people/999”/> <photo xsi:nil=“true”/></person>
/people[/id=123] /id /name /email /photo /best-friend …
/people[/id=123] /id /name /email /photo /best-friend …
16
Examples of LinkedIn APIs HTTP POST - Create
> Add a friendPOST http://api.linkedin.com/v2/people/123/friends<friend uri=“urn:linkedin:v2:people/888”/>
201 CreatedLocation: http://api.linkedin.com/v2/people/123/friends/888
/people[/id=123] /id /name /email /photo /best-friend /friends /456 /888 …
17
Examples of LinkedIn APIs HTTP DELETE - Remove
> Remove a friendDELETE http://api.linkedin.com/v2/people/123/friends/456
> Delete a companyDELETE
http://api.linkedin.com/v2/companies/exchange=NYSE&ticker=GM
> Delete two companiesDELETE http://api.linkedin.com/v2/companies::(664,665)
18
Use Standard Headers
> Content-Type> Last-Modified> Accept> Vary> Authorization> Cache-Control> Content-MD5> Location> Warning
19
Incentive System
> Multiple ways to get at the same data
> Partner can ask for exactly what they need
> Associate cost with resources, system of accounting creates incentives for partners
> Throttling by resource rather than API
20
Real-World Example Xobni Toolbar
> Xobni makes ~20 million profile API calls per week> Default representation is ~2k on average> Using in-line filter brings average to ~1.5k
25% reduction in response size ~11000 Mbits savings per day
11k Mbits out of LinkedIn datacenter 11k Mbits into Xobni datacenter Saves both companies money
