Download - The DNA of a great API
The DNA of a GREAT API
Ciprian Sorlea, CTOOctober 2015
Building API’s that developers will love
04.11.2015 2
What’s an API:
Any API:• allows Software components, systems and subsystems to interact.
• hides the complexity of the underlying architecture and functionality
• focuses on separation of concerns
• allows callers/consumers to access resources (ex. CRUD operations) or to perform various operations on top of the resources
04.11.2015 3
Qualities of a good API:
A good API must be:• domain specific, simple and easy to use• consistent (naming, parameters, etc.)• stateless• transparent & clear in terms of error trapping• accessible (from different platforms, languages, etc.)
04.11.2015 4
From Good To Great:
A great API must be:• well documented and should provide many working samples• fast, secure, scalable and reliable• forward and backwards compatible (versioning)• “self aware”: usage statistics, logs/traces
04.11.2015 5
From Good To Great:
A great API must go the extra mile
04.11.2015 6
Who are the consumers?
An API can be consumed by:- A web application- A desktop application - A mobile application- Another API- An integration test- A 3rd party integration- Internet Of Things- A home appliance, a wearable device, a smart gadget
04.11.2015 7
Now, let’s remember….
Who is your target audience?
Developers
04.11.2015 8
A big question & challenge:
How do we handle all these uses cases, while also making it easy for
Developers?
04.11.2015 9
Things to consider:
- Transport: how do we exchange data?- Resource models: what kind of data do we exchange?- API Architecture patterns: how do we design our API?- Error Handling: how do we handle errors & failure?- Testing: how do we make our API bullet proof?- Security: how do we protect our data (or ourselves)?- Manageability: how can we manage our API and support it operation-wise?- Evolution: how do we ensure our API is future-proof?- Adoption: how do we make sure our API will be adopted?
API Transporthow do we exchange data?
04.11.2015 11
API Transport
REST API:- Recommended for external integrations and applications, 3rd party apps and
plugins- Use JSON & XML over HTTP(s)- Use established Security mechanisms (OAuth2, Basic Authentication, etc.)
04.11.2015 12
API Transport
RPC / Binary Protocol:- Recommended for internal usage or high throughput requirements, IOT and
devices with limited networking capabilities / bandwidth- Use Apache Thrift, Google Protobuff or alternatives- Use simplified authentication and authorization model to minimize overhead
– assume working in safer environment
Note: For now, we’ll focus on
REST API’s
API Resource Modelswhat kind of data do we exchange?
04.11.2015 15
API Resource Models
Resources are the heart and soul of the API
Resources represent the abstraction of any information we exchange and operate on within an API
Domain Model ~= Resource Model
04.11.2015 16
API Resource Models
Resource classification:
- Singleton – can be part of a collection or can be stand-alone- ServerInfo – Server Information can be exposed as a standalone singleton resource- User – User information can be exposed as a resource, part of a collection
- Collection – a list of so called documents belonging to the same domain- Users- Products- Orders- Documents
04.11.2015 17
API Resource Models
Minimal Resource Model Attributes:
- URI: Unified Resource Identifier- represents the unique URL by which a resource can be accessed- HATEOAS takes this to the extreme level- Ex: http://api.domain.com/v1/serverInfo, http://api.domain.com/v1/users/1, http://
api.domain.com/v1/projects/
- ID:- required when the resource is a singleton as part of a collection- a numeric ID (ex: 1029) or a GUID (ex. c366728d-ca93-4b23-a4b9-ae360600b86a)- can be the database specific primary key (ex. auto generated) or can use a custom field with
random generated values, to enhance security
04.11.2015 18
API Resource Models
Important consideration:
- Ensure your resource models can be properly represented in any transport models (ex. JSON, XML) and without significant effort (Ex. Don’t try to use complicated bit level data in your models)
- Follow the Open Close principle. This will make your resources future-proof
- Ensure the API resource model doesn’t carry with it the DB optimizations and complexity
API Architecture Patterns(archetypes)
how do we design our API?
04.11.2015 20
API Archetypes
The de-facto API Archetypes:• Document – a singleton resource• Collection – a collection of resources• Repository (also known as Store) – a collection of immutable resources• Controller – an action dispatcher ***
*** Should be avoided. Explanation later.
04.11.2015 21
API Archetypes
Pseudo API Archetypes:• Queue – asynchronous processing of a long operation – can also be a batch operation
or a transaction, but not limited to those• Should provide status and progress information• Ex: File processing, order processing, etc.
• Transaction – multiple resource operations to be processed in a ALL-OR-NONE manner• The request fails if any of the operation fails.• Any simple action (see Controller) can be also seen/handled as a transaction
• Batch – a sequence of resource operations to be processed in sequence• Any failure is reported to the consumer, but doesn’t stop the processing of the batch • Should provide status and progress information• Ex. Batch create, batch update, etc.
04.11.2015 22
API Archetypes: Document
• Exposes functionality specific to a singleton resource.
• Some standalone documents (ex. ServerInfo), can be read-only
• Some collection-owned documents are mutable (can be updated, deleted, etc.) -> parent archetype is a Collection
04.11.2015 23
API Archetypes: Document – Partial Retrieval
Partial retrieval allows consumer to limit the transferred information. This leads to better performance and improved customer experience.
Limiting the fields:• Specify the fields to be included, as a query string parameter• Specify the fields to be ignored, as a query string parameter• Specify the fields to be expanded (sub-details), as a query string parameter• Specify a predefined view (predefined or custom definition of data retrieval
model)
04.11.2015 24
API Archetypes: Document – Partial Retrieval
Examples:http://api.domain.com/v1/users?fields=id,firstName,lastName,email,addresses(city)
http://api.domain.com/v1/users?excludedFields=email
http://api.domain.com/v1/users?view=minimalView
http://api.domain.com/v1/users?expand=addresses,previousEmails
04.11.2015 25
API Archetypes: Document – Partial Update
Allows consumers to update only those fields that need to be changedThis provides faster and better user experience.
Two ways:- Specifying the fields to be changed with the new value (partial documents)- Using operation maps
04.11.2015 26
API Archetypes: Document – Partial Update
Using partial documents:
PATCH /v1/users/1029 HTTP/1.1
Content-Type: application/json
X-API-PUBLIC-KEY: D6DB835A-35B1-42B3-8805-8F00F5344BB9
X-API-REQUEST-ID: 5BF75F7E-0A55-40DF-9E79-354B9B162971
{
“emailAddress”: [email protected],
“companyName”: “NordLogic”
“phoneNumbers” : [“+4074ILOVEAPIS”]
}
04.11.2015 27
API Archetypes: Document – Partial Update
Advantages of using partial documents:- Request very clean and minimal- Requires very little extra effort to implement
Disadvantages:- Hard to add items to lists / arrays, remove values from maps, etc.
04.11.2015 28
API Archetypes: Document – Partial Update
Using operation maps:
PATCH /v1/users/1029 HTTP/1.1
Content-Type: application/json
X-API-PUBLIC-KEY: D6DB835A-35B1-42B3-8805-8F00F5344BB9
X-API-REQUEST-ID: 5BF75F7E-0A55-40DF-9E79-354B9B162971
{
“emailAddress”: { “set”: “[email protected]”},
“companyName”: { “set” : “NordLogic”},
“phoneNumbers” : { “add” : [“+4074ILOVEAPIS”] }
}
}
04.11.2015 29
API Archetypes: Document – Partial Update
Advantages of using operation maps :- More powerful control over what operations are made- Doesn’t require pre-fetching of documents prior to update to get the actual
map/list content
Disadvantages:- Requests tend to be more verbose, even though just slightly- More effort is needed to implement support for this approach (usually)
04.11.2015 30
API Archetypes: Document – Partial Update
PRO TIP: You can implement Partial Update with either approach, or you can combine
them and allow patching in either way.
04.11.2015 31
API Archetypes: Document – Partial Retrieval
PRO TIP: Combining Partial Update with Partial Retrieval, APIs can offer extreme
flexibility for a wide variety of consumers & devices.
04.11.2015 32
API Archetypes: Document – Partial Retrieval
PRO TIP: You can implement transactions for updating fields that require additional
details on change:
Ex. Change status + comment
04.11.2015 33
Document Archetype: URI & HTTP Verbs
• Base URI: http://api.domain.com/{version}/{documentID} or http://api.domain.com/{version}/{collection}/{documentID}
HTTP Verb DescriptionGET Retrieves the document (complete or partial)PUT Updates a document with the content provided. Only for Collections.PATCH Updates a document fields specified in the content, or applying the
specific update operations. Only for Collections.DELETE Deletes the document from the locationHEAD Verifies if the given document id exists. OPTIONS Lists the available operations / HTTP Verbs
04.11.2015 34
API Archetypes: Collection
Collections provide access to multiple document resourcesOperations are usually CRUD
Collection capabilities:- Pagination - Filtering: using simple filtering or a query language)- Sorting: simple (based on single field) or complex (based on multiple fields)- Partial field retrieval: identical to document level partial field retrieval
04.11.2015 35
API Archetypes: Collection Query String
Query String parameters are used to configure collection functionality (ex: sorting, filtering, etc.)Rules & recommendations:- Use same QS parameter names across your entire API, and document them
- Ensure you don’t introduce un-necessary constraints by choosing improper names
- As long as you don’t confuse your consumers, you can allow aliases for some parameters
Ex: - Page & PageSize work for a basic pagination but not for continuous scrolling – improper naming
- Start & limit work for any data pagination scenario
- Offset/From and count can be aliases to start & limit
04.11.2015 36
API Archetypes: Collection Query String
Pagination
Ex: http://api.domain.com/v1/users?from=100&count=10
http://api.domain.com/v1/users?count=10 (uses a default offset/start)
04.11.2015 37
API Archetypes: Collection Query String
Sorting- can be simple (single field + direction) or complex (list of fields, with specific
sort orders)- can be applied on 1st level fields or on details (expanded content)
Ex: http://api.domain.com/v1/users?sortBy=email,sortOrder=DESC
http://api.domain.com/v1/users?orderBy=name:ASC,email:DESC,contactInfo.phone
04.11.2015 38
API Archetypes: Collection Query String
Filtering:- Simple filters (all filters are of type EQUALS):
http://api.domain.com/v1/[email protected]://api.domain.com/v1/users?companyName=Nordlogic&role=developer
- Advanced filters (specify the operation: EQ, NE, GT, LTE, etc.)http://api.domain.com/v1/users?companyName=LIKE:*logic&role=NE:qa
- Query (use a pseudo query language):http://api.domain.com/v1/users?query=companyName like ‘*logic’ and role not eq ‘QA’
http://api.domain.com/v1/users?query=name eq ‘John’ or name eq ‘Bill’
04.11.2015 39
Collection Archetype: URI & HTTP Verbs
• Base URI: http://api.domain.com/{version}/{collection}/
HTTP Verb DescriptionGET / Retrieves a list of paged, filtered and sorted documents, complete or
partialGET /{docID} Retrieves the document (complete or partial)POST / Creates a new document in the collection
HEAD Can be used to check if data exists given the provided filters and pagingDELETE Purges the collection. Recommended for Development only.OPTIONS Lists the available operations / HTTP Verbs
04.11.2015 40
Collection Archetype: Response
Few tips:
- Always echo the pagination details (ex. offset & count), to provide context
- Give more details about the rest of the data (is there more?). Ex: maxRows
- Don’t include empty/null fields and ensure (through documentation) your consumers handle that. (This applies to single documents as well)
04.11.2015 41
API Archetypes: Repository
Repositories provide access to multiple immutable document resourcesOperations are usually CRD (Create, Retrieve, Delete)Updates to documents within Repositories MIGHT happen behind the API level
Nice to have capabilities:- Capped repositories: limit data to most recent X records- Auto Expiration: automatically delete data older than X days/hours/minutes- Tailing options: retrieve latest X records
04.11.2015 42
Repository Archetype: URI & HTTP Verbs
• Base URI: http://api.domain.com/{version}/{repository}/
HTTP Verb DescriptionGET / Retrieves a list of documents with limited capabilities (ex. Tailing over
the records in an audit/trace log). Nov very common.GET /{docID} Retrieves the document (complete or partial).POST Creates a new document in the repositoryHEAD When GET is also implemented, this can be used to check if data exists.DELETE Purges the repository. Recommended for Development only.OPTIONS Lists the available operations / HTTP Verbs
04.11.2015 43
API Archetypes: Controller
Allows execution of application specific actions that cannot be mapped with any of the CRUD methods (specific to collections or repositories).
Should be avoided as much as possible. Actions should be reconsidered as one of the proposed pseudo-archetypes: batch, queue, transaction.
This would allow execution to be more manageable and traceable.
04.11.2015 44
Controller Archetype: URI & HTTP Verbs
• Base URI: http://api.domain.com/{version}/{param1}/{param2}/{controllerAction}
HTTP Verb DescriptionPOST Executes the specific action, with the given parameters
04.11.2015 45
API Archetypes: Queue
Allows execution of application specific actions (jobs) in an asynchronous way, while allowing proper management and traceability of the action.
Each action, once queued, has an identifier and a status. Therefore each action becomes a Resource.
Upon completion, the job document can embed the job result or can provide a link to the actual result.
04.11.2015 46
Queue Archetype: URI & HTTP Verbs
• Base URI: http://api.domain.com/{version}/{queue}/
HTTP Verb DescriptionGET /{id} Retrieves the queued job status, progress information and execution
result – if available.POST Submits a new job into the queue and retrieves the job identifier.DELETE /{id} Cancels a job (if the status allow the operation to be canceled)OPTIONS Lists the available operations / HTTP Verbs
04.11.2015 47
API Archetypes: Transaction
Allows execution of application specific actions in transactional manner.
Transactions can be asynchronous (see queues) or asynchronous.
In case of synchronous transactions, response contains the transaction result.
04.11.2015 48
Transaction Archetype: URI & HTTP Verbs
• Base URI: http://api.domain.com/{version}/{queue}/
HTTP Verb DescriptionGET /{id} Retrieves the transaction status, progress information and execution
result – if available.POST Submits a new transaction to be processed.DELETE /{id} Stops/cancels a transaction (in case of asynchronous/queued
transactions)OPTIONS Lists the available operations / HTTP Verbs
04.11.2015 49
API Archetypes: Batch
Allows execution of application specific actions over multiple records / bulk processing.
Batches must always be asynchronous, and the behavior is identical to Queues.
Progress information is mandatory in case of batches.
04.11.2015 50
Batch Archetype: URI & HTTP Verbs
• Base URI: http://api.domain.com/{version}/{batch}/
HTTP Verb DescriptionGET /{id} Retrieves the status, progress information and execution result – if
available.POST Submits a new batch job to be processed.DELETE /{id} Stops/cancels a batch jobOPTIONS Lists the available operations / HTTP Verbs
04.11.2015 51
Pseudo Archetypes
PRO TIP: You can combine Queues with Transactions and Batches in any way you
want.
Just make sure things make sense
04.11.2015 52
Pseudo Archetypes
PRO TIP: Sometimes you might want to have, for each Pseudo Archetype, a
validation equivalent.
Helps your applications/integrations cause less frustrations.
Error Handlinghow do we handle errors & failure?
04.11.2015 54
API Error Handling
Use specific HTTP Error Codes where appropriate for application logic specific errors. Ex:
- 401 Unauthorized: errors with authentication / authorization
- 404 Not Found: document does not exist
- ….
04.11.2015 55
API Error Handling
Response body should always contain:- Error code – what happened, coded for analytics
- Error message – human readable message, localized to user’s profile
- Error details – the context on the error.
Nice to have:- Error ID – id of the error, generated by the lowest level module where the error
occurred. Should also propagate and find itself in the log files.
- Flow ID – if possible, for cross-app tracking
- ….
04.11.2015 56
Pseudo Archetypes
PRO TIP: Certain operations can generate multiple errors (ex. multiple validation
errors). If applicable, provide a list of errors in the response instead of providing just the
first error encountered.
04.11.2015 57
Pseudo Archetypes
PRO TIP: Don’t provide too much details in the error messages. Consider privacy and
security before anything else.
Testinghow do we make our API bullet proof?
04.11.2015 59
API Testing
Option 1:
- Develop API Client along the actual API Service- Build Automated tests using the API Client, using the same programing
language
04.11.2015 60
API Testing
Option 2:
- Build Automated tests using a dedicated API Testing framework
• http://frisbyjs.com/• https://github.com/mkotsur/restito• https://github.com/rest-driver/rest-driver• https://github.com/dareid/chakram
04.11.2015 61
API Testing
Option 3:
- Use dedicated 3rd Party apps or Browser Plugins
• https://www.getpostman.com/• http://www.soapui.org/rest-testing/getting-started.html
Securityhow do we protect our data (or
ourselves)?
04.11.2015 63
Securing Rest API
Simple rules:
- Use disposable authentication tokens- Implement authentication mechanisms good for your target audience
Ex: OAuth2 works very well for web and mobile, but does it work for enterprise?
- Use granular roles & access rights for each resource model- Take the multi-tenancy into account from start- Use & require HTTPS whenever sensitive data is in transit- Encrypt any sensitive data at rest
04.11.2015 64
Securing Rest API
Preventive & Enterprise Ready Security:
- Audit any failed authentication / authorization- When using sessions, ensure the expiration time can be configured- Lock/disable authentication tokens after a configurable number of failed
authentication attempts- Use a whitelist / blacklist for request filtering- Implement overall rate limits as well as application / token specific limits- Validate any data coming IN. Never take things for granted.
04.11.2015 65
Securing Rest API
Preventive & Enterprise Ready Security:- Require (configurable) password strength- Use password expiration - Log Remote IP, User-Agent (and Application ID, if possible) with any request- Pass authentication tokens in the request headers, not in URL/Query String- Filter unaccepted query string parameters from URL’s- Escape/strip any textual information accepting user input
04.11.2015 66
Pseudo Archetypes
PRO TIP: NEVER, EVER, EVER, EVER
TRUST YOUR USERS.
Manageabilityhow can we manage our API and
support it operation-wise?
04.11.2015 68
Managing Rest API & Supporting Operations
Simple rules:- Log everything, almost everything.- Implement usage analytics and metrics- Time any operation, build performance metrics and alerts- Build operational dashboards, enable error notifications - Use change logs & audit trails for sensitive data- Require “4 eyes check” rule on any major change performed by admins- Use infrastructure as code
Evolutionhow do we ensure our API is future-
proof?
04.11.2015 70
Future proof your API
Simple rules:
- Version your resource models and archetypes from the beginning- Any new version of the resource model should extend the previous versions- Establish a support plan from first public release- Classify your API’s as public, private & unsupported to ensure phased
adoption
Adoptionhow do we make sure our API will be
adopted?
04.11.2015 72
Ensuring APIs adoption
- Make it easy to get started - developers should be 1 CURL request away from their first API call.
- Provide good and clear documentation.
- Provide SDK’s / client libraries and samples for most popular development platforms and languages of your target audience.
04.11.2015 73
Ensuring APIs adoption
- Ensure your business logic & rules don’t have to be guessed and duplicated by developers in their own apps (ex. validation rules).
- Eat your own dog food. Use your API to build your other products / integrations.
- Don’t introduce breaking changes. Plan to support your public API releases.
04.11.2015 74
Ensuring APIs adoption
PRO TIP: Deliver VALUE, not just JSON or XML.
Thanks for watching.