punctuated equilibrium, celestial navigation, and apis
DESCRIPTION
In evolutionary theory, punctuated equilibrium refers to a period of significant environmental stress resulting in rapid, dramatic changes among species. It is a powerful model for understanding the changes in technology, business models and data over the last few decades - changes that have given rise to the age of APIs. In this talk, we'll look at three key themes in the API economy: •The Evolution of Business Models: From 1st Party to Partner to Platform •The Evolution of Architecture: From Mainframe to Mobile •The Evolution of Data: From Silos to SocialTRANSCRIPT
Punctuated Equilibrium,Celestial Navigation,and APIs
Competing through dynamic adaptation
Sam Ramji, Apigee @sramjiDan Jacobson, Netflix @daniel_jacobsonMichael Hart, Netflix @michaelhart
PUNCTUATEDEQUILIBRIUM
Darwin formulated his theory of evolution about 150 years ago
Based on observations he made in the Galapagos Islands 15 years earlier
A wild diversity of creatures existed in a new environment
Starting from an ancestor which looked like this
Geospiza Fulginosa
Finches evolved that looked like this
Geospiza Fortis
and this
Camarhynchus Pallidus
and this
Camarhynchus Pauper
and this
Geospiza Conirostris
and this
Certhidea Olivacea
and this
Geospiza Scandens
For many years the belief was that this change happened slowly and gradually.
In 1972, Stephen Jay Gould and Niles Eldredge proposed a new idea
that evolution is not slow and gradual
but sudden and violent.
Applying this view to the observations of finches
Geospiza
C. Parvulus
C. Pauper
Camarhynchus
C. Psittacula
C. Heliobates
C. Pallidus
G. Fortis G. Fulginosa
G. Magnirostris G. Scandens
G. Conirostris
Certhidea
C. Olivacea
“Thus, from the war of nature, from famine and death, the most exalted object of which we are capable of conceiving, namely the production of higher animals directly follows.”
Charles DarwinOn Origin of Species
So while it may look slow and gradual in hindsight
Evolution is experienced in punctuated bursts.
If you’re living in a punctuated burst of evolution
it feels like a revolution
CELESTIALNAVIGATION
Exploration
like evolutionary change
only looks smooth in hindsight
Living through it is usually chaotic
Karen JamesThe Beagle Project Blog
To navigate, you need a map and instruments
Maps exist for transferring knowledge
and they too have evolved over time.
They started as oral traditionand were written down in a form called a periplus
Periplus of HannoCourtesy of Heidelberg University
Periplus of HannoCourtesy of Cornell University
Far less efficient knowledge transfer than a modern map of the same journey
Map of Hanno’s JourneyCourtesy of Bourrichon/Wikipedia
Exploration was dramatically held back for want of a map
In the two thousand years between Hanno’s journey on a Phoenician trireme
And the Mediterranean caravel of the 15th century
Maps had only evolved to be graphical descriptions of coastlines
That was a map published a few years beforeColumbus crossed the Atlantic to find India
After his crossing, his expedition shared their knowledgein a new map
Still far from perfectbut much improved.
The biggest challenge in this kind of explorationwas determining their location on the Earth
Instruments for measuring latitude had beenused and improved for centuries
Longitude was the hard problem.
You needed to know not just the angle of the sun and stars
you also needed to know the precise time.
Regardless of your sailing technologywithout the proper measurementyou were lost
We are not promising a perfect map of the new world
But it should be more like this
than this
Periplus of HannoCourtesy of Heidelberg University
and we will show you what we know how to measure.
APIS
There are more niches today than we’ve seen before, so we need to borrow from nature
If we start with an API we can explore all the niches around our business
Visualization by Apigee
The leaders of today’s Internet
clearly understand this mechanism.
They understand that the distribution model for value has changed in the Internet era.
ConsumerRetail StoreProducer
Packaged Goods
Internet Services
ConsumerDeveloperProvider App
Developers took their APIs and explored the niches for them
The providers and the developers both benefited from this adaptation
Suddenly this seems obvious to everyone.
Data from Programmable Web
2005 2006 2007 2008 2009 2010 20110
1000
2000
3000
4000
5000
6000
Open APIs from 2005-2011
And developers are racing to pack the niches.
Data from Wikipedia
0
100,000
200,000
300,000
400,000
500,000
600,000
0
2000000000
4000000000
6000000000
8000000000
10000000000
12000000000
App Store Growth 2008-2011
Apps AvailableTotal App Down-loads
This is a sudden, material shift in competition.
It only looks gradual if you’re losing.
BUSINESS MODELS
From 1st Party to Partner to Platform
We’ve seen punctuated equilibrium in business models over the last hundred years
Direct Sales
Specialty Store
Department Store
Indirect Sales
National Chain
Big Box Retailer
App Developer
Mobile App
Web Catalog
Device App
Web Retail
What’s the environmental stress driving the current rapid change?
The first wave of the Internet demonstrated the economic impact of web-based business models.
Previous eras of business showed a normal distribution for revenue, with most firms getting most of the revenue. In the later half of the 20th century, business model innovations focused revenue in the 2nd standard deviation above the mean. The “80:20” rule became conventional wisdom.
The HTML-driven Internet showed new business models that focused revenue in the 3rd standard deviation (examples: Amazon, EBay). Reality reflected a “95:5” rule where 5% of companies dominated the transactions and profits.
The API-driven Internet is demonstrating the next concentration of power and is reflecting a “99:1” distribution (examples: Twitter, Facebook) due to the high switching costs and effective lock-in through software.
80:20
95:5
99:1
The next wave of the Internet is demonstrating the economic impact of API-based business models.
Hardt’s Theorem: The Internet Power Law
But you need to tackle it in a way that fits your business
Platform Partner
1st Party
Open
Open Open
These are complementary and distinct.Open is different for each one.
1st Party Apps
Partner Apps
Platform Ecosystem
1st party is about offering direct access to your core business via apps that you make or contract out
1st party
Here open means all the business is accessible to internal developers and contract specialists
1st party
Partner is about enabling directed development of apps that extend your business model towards your business partners
partner
Here open means existing partners have access to your business via APIs and can innovate asynchronously
partner
Platform is about enabling unknown developers to build brand new apps and businesses that will surprise and inform you
platform
Here open means enabling business models and allowing developers to support each other at massive scale
platform
Open is attractive
open
Open is Biz Dev 2.0
open
Platform Partner
1st Party
Open
Open Open
Open lets you navigate across the possible business models when your first model doesn’t work as planned
open
Platform Partner
1st Party
Open
Open Open
To get your API strategy properly grounded
John MusserProgrammable Web
“
But how?
Let’s break it down
Establish Target Segments
Engage Developer Channel
Set Industry Goal
An API should extend your core businessinto a new part of the market
target segments
Your core business already has key performance indicators
target segments
So apply your KPIs to the new market segment you’re targeting with your APIs
target segments
What is the market impact you need to create in order to succeed as a business?
target segments
What does the target segment need that it is not getting from you today?
target segments
The answer will be the foundation of your API strategy.
target segments
In most cases the channel for your API will be developers, but what do they need?
developer channel
A profit motive.
developer channel
Here are the leading profit models for developers today
developer channel
In-app purchases
Affiliate royalty
Your advertising spend
Market awareness of their offering
developer channel
App sales
If you don’t know where you’re going, you definitely won’t get there
industry goal
Partnerships and platform businesses are very different things.
industry goal
Partnerships are formed to serve a known set of entities
industry goal
A partner API should be traceable to each partner’s relationship
industry goal
And support end-to-end business processes
industry goal
A platform exists to create massive and unpredictable opportunities
industry goal
All your technology, support, and community decisions will be about surviving the scale of adoption
industry goal
That’s the strategy dimension.
The execution dimension is what you already know.
Planning.
Management.
Organization.
Putting all this in context gives us a map for our API strategy
Planning Management Organization
Target Segment(s)
Define market segment in detail including size and
user persona; specify API profile needed to satisfy top use cases for each
target segment
Establish KPI targets, traceability and dashboard
Business-led
Segment-oriented workstreams
Engage Channel
Specify business model and marketing driver for
the channel that will reach each target segment
Establish developer adoption targets,
developer marketing and channel actions
(community site, events, and communication)
Channel-led
Community, developer, and business development
workstreams
Industry Goal
Specify roadmap of API deliverables, mechanics, integration, and business
process to meet target segment needs
Implement API roadmap, adjust and report on iteration cycle, and
establish alpha developer team
Engineering-led
API, infrastructure, and developer support
workstreams
The instruments will be your KPIs and your core API metrics: performance and adoption
ARCHITECTURE
From Mainframe to Mobile
Computing
Mainframe
Minicomputer
Integrated
PersonalComputer
Smartphone
Connected Devices
Website
Client/Server
Web App
DCOM
Distributed
CORBA
N-tier
Chris AndersonWired Magazine
“The Web is Dead. Long Live the Internet.
The Web is Dead. Long Live APIs!
Twitter traffic distribution shows what he means
Twitter Traffic in 2010
Twitter APITwitter.com
Netflix traffic distribution is nearly the same.
The majority of Netflix traffic comes from API-driven connected devices.
Like Columbus, Netflix started with a map of the coastline
Build an open API as a platformand let a thousand flowers bloom
But they had left the coastline far behind
And the instruments indicated that there were fewer flowers than expected
Netflix API Requests by Segment
Netflix Devel-opersOpen API De-velopers
But partners started building apps for connected devices and the business took off
XBox
PS3
Wii
LG TVs
Apple TV
iPad
iPhone
Roku
Samsung TVs
Architecture should reflect the business model
So Netflix has drawn the following map
XBox
PS3
Wii
Google TV
Apple TV
iPad
iPhone
Roku
LG TVs Samsung TVs
Instruments show that API traffic has grown tremendously in a short time
Growth of Netflix API
Jan-10
Feb-10
Mar-1
0
Apr-10
May-1
0Jun-10
Jul-10
Aug-10
Sep-10
Oct-10
Nov-10
Dec-10
Jan-110
5
10
15
20
25M
onth
ly R
eque
sts
in B
illio
ns
20,000,000,000 API requests per month.
Is that a cause for celebration?
Or for concern?
When you’re navigating uncharted waters,speed is not your friend.
Perhaps it’s time to slow down and avoid risking unknown reefs.
Navigating this growth challenge for Netflix means that the next API revision will focus on reducing overall traffic.
Part of this redesign is reviewing conventions
Punctuated Equilibrium: REST
Data sourced fromProgrammableWeb
2005 2006 2007 2008 2009 2010 20110
500
1000
1500
2000
2500
3000
3500
4000
4500
RESTSOAP
REST seems obvious but assess what makes sense for your business.
Tiered architecture helps you navigate different problems with agility
Recommendations
User Info
Similar Movies
Movie Metadata
Ratings
Viewing History
DataNormalization
&Resiliency
User Service
R12n Service
Similar Movie Service
USER API
iPhone Wrapper
Wii Wrapper
Xbox Wrapper
PS3 Wrapper
Roku Wrapper
Apple TV Wrapper
iPad Wrapper
PC / Mac Wrapper
TiVo Wrapper
Source Data Layer
API Repository Layer
API Layer Wrapper Layer
App LayerWeb Service Layer
SHARED
API INTERFACES
Shared Layer
Model Controller View
UN
IFIED
LIST/TITLE API
Recommendations
User Info
Similar Movies
Movie Metadata
Ratings
Viewing History
DataNormalization
&Resiliency
User Service
R12n Service
Similar Movie Service
USER API
iPhone Wrapper
Wii Wrapper
Xbox Wrapper
PS3 Wrapper
Roku Wrapper
Apple TV Wrapper
iPad Wrapper
PC / Mac Wrapper
TiVo Wrapper
Source Data Layer
API Repository Layer
API Layer Wrapper Layer
App LayerWeb Service Layer
SHARED
API INTERFACES
Shared Layer
Flexible Stable Agile
UN
IFIED
LIST/TITLE API
Server architecture should support both crests and troughs of the waves of demand
Instance Architecture Based on Specialization
List CreationDependency
Service
API METADATA CACHING LAYER
METADATA SERVICE
MetaData
Dependency Service
ELASTIC INSTANCE LAYER
Instance Architecture Based on Specialization
List CreationDependency
Service
API METADATA CACHING LAYER
METADATA SERVICE
MetaData
Dependency Service
ELASTIC INSTANCE LAYER
Handles Request/Response
Caches Dependency Data
Populates and Manages Cache
Map out your usage patterns and cache your data accordingly
Vertical Caching
Vertical Caching: Netflix Full Movie Data
Horizontal Caching
Horizontal Caching: Netflix Basic Data
Combining horizontal and vertical caching may be the best approach when building for multiple geographies
Two-Dimensional Caching
Design for where you’re going
Not for where you are
You may be starting here
Growth of Netflix API
Jan-10
Feb-10
Mar-1
0
Apr-10
May-1
0Jun-10
Jul-10
Aug-10
Sep-10
Oct-10
Nov-10
Dec-10
Jan-110
5
10
15
20
25M
onth
ly R
eque
sts
in B
illio
ns
But you must design for here
Growth of Netflix API
Jan-10
Feb-10
Mar-1
0
Apr-10
May-1
0Jun-10
Jul-10
Aug-10
Sep-10
Oct-10
Nov-10
Dec-10
Jan-110
5
10
15
20
25M
onth
ly R
eque
sts
in B
illio
ns
You don’t need to implement for massive scale
But you must design for it or you will follow your successful ocean crossing with a massive shipwreck.
Navigation also means constantly adjusting your course to ensure you arrive at your final goal
Sometimes adjusting course on an API means you must change your version
Versioning means supporting multiple applications, all of which basically do the same thing
Versioning1.0
1.5
2.0
Today
3.0?
4.0?
5.0?
If possible, go versionless
Version-less API1.0
1.5
2.0
versionless
Today
Here there be dragons
Rules for going versionless in your production APIs
Extend your API by extending data types
Addition is not version-worthy
Better to be incomplete than inaccurate
Withhold implementation if you are unsure
With APIs emerging, we need better tools help us navigate
The service level agreement is your sextant
Set reasonable service level agreements
What is reasonable will vary by API and use casebut you must communicate them to your users.
Measure average latencies, error rates and types,and respond when SLAs are broken
Having visibility into performance means you can tack immediately rather than after your users threaten to mutiny.
Longer-term navigation requires higher-level metrics
Request-based metrics such as what endpoints were called and what parameters were passed can show you what aspects of your API are popular.
Response-based metrics such as what was delivered, how quickly, and whether the response was valid can show you what aspects of your API need work.
System trace metrics that track what underlying systems were called and how they responded can show you where you need to evolve your internal architecture.
Business performance metrics such as how much revenue or how much customer engagement is occurring through the API show you how close you are to delivering on the business plan.
DATA
From Silos to Social
Data
Flat file
Mainframe
Silos
Caching DBs
Domain-specificData APIs
RDBMS
Data APIData
Warehousing
Shared
Private Cloud DBs
Punctuated equilibrium in data sharing
Mainframe Databases Middleware APIs
App Org Cross-org Cross-business
So we are evolving to cross-business sharing
There are a few challenges in making this journey
Sharing your data via an API entails a different set of considerations than APIs which expose your services
loss of control
only recourse is legal
enforcement is expensive
That said, it may be time to get overyour control issues
Be honest about the value you could get from sharing your data outside your corporation
instead of just the costs and risks of sharing it.
In opening up its movie data warehouse
Netflix found that the cost was the same as any API,
the risks could be managed through rate limits and access control,
but that now others could build great movie discovery experiences that led to increased Netflix viewing hours.
Sharing data as a service means making a few course corrections on your API
enable larger downloads for fewer queries
more liberal retention policies means less API traffic, higher performance, and less cost
push incremental updates
limited access to richer queries
Looking forward, how are we going to work through this wave of shared data?
Two different dimensions are apparent
First, no one wants all of your data, just some of it.
Just the right parts of it for their particular need.
Where have we seen this before?
Mainframe Databases Middleware APIs
App Org Cross-org Cross-business
Last time the right solution was a query language
Perhaps it’s the right solution now
Instead of a static REST call, we could pose a query like “what are the highest rated movies from the 1980s?”
http://odata.netflix.com/Catalog/Titles?$filter=ReleaseYear le 1989 and ReleaseYear ge 1980 and AverageRating gt 4 &$expand=Awards
Caveat structor does apply
This is new ground and we haven’t seen anyone do this at massive scale
Second, people want just the right parts of your data for their particular need
But they need the right parts of other people’s data to have a complete context
Where have we seen this before?
Mainframe Databases Middleware APIs
App Org Cross-org Cross-business
Last time the right solution was middleware for distributed queries
Perhaps it’s the right solution now
But the rules have changed a bit since the data is laying all over the Internet
[{ "id": null, "name": null, "type": "/dining/restaurant", "/business/business_location/address": [{ "street_address": [], "citytown": { "id": "/en/toronto" } }], "cuisine": [{ "/dining/cuisine/region_of_origin": [{ "!/film/film/featured_film_locations": [{ "id": "/en/the_italian_job" }] }]
Instead of a single domain query, we could ask for a list of “Toronto restaurants with cuisine from a filming location of ‘The Italian Job’”
Caveat structor still applies.
This is very new terrain indeed.
The bigger question than what should you be sharing out
Is what should you be sharing in
What data APIs should your business be using?
What could you offer your customers if you knew which of them were friends with each other?
What does that logo mean to you today?
Perhaps it’s time to think differently.
Facebook is a massive data API that lets you correlate your customers with their true context
You could move beyond a flat-earth view where all your customers are their own islands of data
And each of them were connected in ways that makes your business more valuable to them
What could you offer your customers if you knew where they like to go?
The more of your customer’s context that you can understand
The more time you can save them
And that makes your business more valuable to them
The good news is that there are already data APIs to get this context
Now you need to focus on sharing in
In
CLOSING
We are going through a period of rapid change in business models, architecture, and data
Navigating based strictly on the stories of others
Periplus of HannoCourtesy of Heidelberg University
Will not give you the clear map that you need
Develop instruments around your APIthat help you understand where you’re going
so that you can correct your course and beat your competition in the race for the future.
THANK YOUQuestions and ideas to:
@michaelhart@daniel_jacobson@sramji