livingsocial/api-design

Name: api-design

Owner: livingsocial

Description: LivingSocial API Design Guide

Created: 2017-07-28 14:53:10.0

Updated: 2018-05-10 21:11:20.0

Pushed: 2017-07-28 14:56:10.0

Homepage: null

Size: 26

Language: null

GitHub Committers

UserMost Recent Commit# Commits

Other Committers

UserEmailMost Recent Commit# Commits

README

LivingSocial API Design Guide

Version 1.0.0 - Published 2017 Jul 28

Internal version published 2016 Mar 10

Contents
Purpose

The purpose of the LivingSocial API Design Guide is to provide standards and best practices that all new LivingSocial APIs (both internal and external) should follow. This guide is to help developers and architects design and implement consistent and well-documented APIs across our enterprise.

A basic knowledge of REST, HTTP, and JSON is assumed. We will provide links to resources for those who want to learn more about the fundamentals of these technologies.

Overview

API Design has come of age, and has become a first-class citizen in the enterprise. The principles and practices that we follow determine the usability and overall quality of our APIs. We've seen web sites and books on the topic, but why do we need an API Design Guide, and what are the essentials of good API design?

Why Have an API Design Guide?

An API Design Guide provides us with the following:

As we begin to externalize our APIs, they reflect on our company, and become an extension of the LivingSocial brand and strategy.

This is a living document and subject to change and discussion. Please file PRs against this repo to discuss changes and enhancements.

Guiding Principles

Here are the guiding principles for designing an API and determining the merit of each design practice:

API Design Essentials

The remaining sections of this page cover the key areas to consider when designing and implementing an API.

Requests
URIs/Paths

The URI (Uniform Resource Indicator) / Path is the path to a resource exposed by an API. Here are the key principles:

Use Plural Resource Names

Resource names should be plural, for example if we're exposing Inventory Items, then the Base URI should look like:

entory_items

unless the resource is a singleton, for example, the overall status of the system might be /status.

Use Nouns, Not Verbs in the Base URI

Never put a Verb in the Base URI. Rather than something like /get_inventory_by_id, we use the following URI with an HTTP GET:

entory_items/4

We'll cover appropriate HTTP Verb usage in the HTTP Verbs section below.

Shorten Associations in the URI - Hide Dependencies in the Parameter List

Resources are related to one another, and eventually they are stored in a database with associations between the tables. In the early days of REST, there were URIs that looked like:

omers/1/orders/54/inventory_items/5900

This is bad practice because:

Instead, shorten the association in the URI and hide the nested data dependencies in the parameter list. Most URIs should go no deeper than the following:

ource/{identifier}/resource

With this structure in mind, the new path now looks like:

omers/1/orders?order_id=54&inventory_item=5900
Uniform Interface with HTTP Verbs/Methods

The following table shows the standard HTTP Verbs/Methods to act on resources. Use this approach rather than cluttering the URI with verbs.

| HTTP Verb / Method | Action | |:——————-|:——————-| | GET | Read | | POST | Create | | PUT | Update (full) | | PATCH | Update (partial). | | DELETE | Delete |

Resource Identifiers

Resources should use Sequential UUIDs (SQUUIDs).

We chose SQUUIDs because they limit fragmentation of indexes and eliminate performance problems in MySQL. SQUUIDs are stored as binary(16)

SQUUIDs should be represented in string form as lowercase. This is so that they can easily be joined across tables in the data warehouse.

Do Not Use Database Table Row IDs as Resource IDs

Auto-incrementing integer database row identifiers should not be used as Resource IDs or exposed in any way through the API.

Exposing DB table “auto-increment” row IDs as Resource IDs leaks implementation details outside the API. Row IDs couple the resource too closely to the underlying persistence engine and schema, and can cause headaches when the owning service might need to migrate away from a particular persistence engine or structure, or when clustering of several DBs becomes necessary. IDs not generated by autoincrementing column IDs allow for more flexibility.

Sample SQUUID Generator
squuid
is is basically the implementation of SecureRandom.uuid ...
y = SecureRandom.random_bytes(16).unpack("NnnnnN")
. but we replace the high-order 32 bits with the current time.
y[0] = Time.now.to_i
y[2] = (ary[2] & 0x0fff) | 0x4000
y[3] = (ary[3] & 0x3fff) | 0x8000
08x-%04x-%04x-%04x-%04x%08x" % ary

Versioning
Why Versioning?

API Versioning is an important aspect of API design because it informs the consumer about an API's capabilities and data. Consumers use the version number for compatibility.

Versioning Methods

Here are 2 of supported methods of API versioning:

We currently have some APIs that use Header-based versioning, and some that use URI-based versioning, in addition to deployed mobile apps that invoke URI-versioned APIs. Here is the direction we would like to take:

Version in the HTTP Accept Header

entory_items/4
pt: application/vnd.livingsocial.v1+json

Here are the pros when putting the version in the HTTP Accept Header:

Here are the cons:

Version in the URI

Many APIs put the version in the URI. Here's an example:


inventory_items/4

Here are the pros when putting the version in the URI:

Here are the cons:

Versioning Schemes

A pragmatic versioning scheme completes the overall API versioning strategy and addresses how to format the version number, when to upgrade and when to retire a version:

Breaking Changes

A Breaking Change is any change to an API that could break a contract and cause consumer/client invocations of the API to fail. When changing an API, there are several things to think about when determining if a new version is needed or if the current version is still going to work for the client:

Non-Breaking Changes

A Non-Breaking Change is a change to the API that does not break a contract nor does impact the consumer/client. Here are some examples:

Resources for Versioning Schemes

Here are a few resources for dealing with the above issues:

API Versioning Summary

Based on the research and discussions above, here are the high-level recommendations for versioning an API:

Pagination
Why Pagination?

An API must be able to control/gate the amount data returned in the response so that the Consumer is able to handle the volume of data. If an API returns all instances of a given resource (e.g., inventory items, etc.) it could easily overrun the memory and processing capacity of the consumer. Pagination helps control the volume of data returned and makes it easier for the Consumer to process.

Pagination - Summary

Here are the high-level recommendations for API Pagination:

Pagination Method

For our APIs, the offset is semantic - it could be a UUID, ID, a Date, etc. that is sortable. This semantic offset approach gives the API Producer (i.e., the developer who creates the API) a chance to choose an optimal way to do an offset into a result set based on a sortable field/column based on the most efficient way to access the resource's data. But the consumer will not provide the offset and limit; instead, the API itself provides this information to the Consumer.

Pagination - Leading the Consumer

An API should lead the consumer through the API invocations to help them reach a goal. In this case, we want to lead the consumer forward and backward through the result set rather than making them guess or keep track of where to go next. Each API should provide hyperlinks that provide URIs to the next and previous pages. This is an example of HATEOAS (Hypertext as the Engine of Application State), and here's what these hyperlinks might look like in a JSON response:


entory_items?offset=100&limit=25

_links": {
 "self": {
   "href": "/inventory_items?offset=543&limit=25"
 },
 "next": {
   "href": "/inventory_items?offset=768&limit=25"
 },
 "prev": {
   "href": "/inventory_items?offset=123&limit=25"
 }
,
items": [
 {
   "name": "Washington Nationals Game",
   "type": "Sporting Event",
   "price": "$35.00"
 }
 ...


In the above example, the "_links" property provides links to the consumer to help them navigate through the result set:

This is a very simple implementation of HAL (JSON Hypermedia API Language). For further information, please visit the HAL site.

Client Use of Pagination URIs

For client apps to use paginated APIs, it is necessary to embed the appropriate data from _links into the client views. For example, when the client displays a view containing paginated data, the next link must be embedded into the view so that the client web app knows where to fetch the next page of API data. Clients should avoid trying to construct new URLs to the API, but rather depend on the next and previous links returned by the API.

It is also recommended to encrypt the pagination URIs embedded in client views to prevent bad actors from making arbitrary calls to our backend services.

Given the example results in Leading the Consumer, the client view might generate a links in its view like this ERB template example:

ref="/results?page=<%= encryptor.encrypt_and_sign("/inventory_items?offset=543&limit=25") %>">Next Page</a>

The controller that renders the /results view can use the value of the page parameter to fetch the next page of data from the API.

API-Only Sub-Domains

If your API will be publicly accessible, contact Architecture to have a discussion about hosting it under api.livingsocial.com, as there are many potential benefits to doing so.

Filter/Sort/Search

Here are some general guidelines:

Content Negotiation

Content Negotiation, part of the HTTP specification, is a mechanism that enables an API to serve a document/response in different formats (e.g., XML, JSON, etc.). Based on current industry best practices, we prefer JSON.

An API consumer can specify that they expect JSON in the response by using the HTTP Accept Header or by adding an extension to the URI. Here's how to specify the JSON MIME type in the HTTP Accept Header:

pt: application/json

If JSON wasn't specified in the HTTP Accept Header, here's how to do this with a .json URI extension (Rails provides this by default).

cities/nearby/zipcode/1234.json

The HTTP Content-Type Header is the other header involved in Content Negotiation, and works as follows:

Here's how to specify the JSON MIME type in the HTTP Content-Type Header:

ent-Type: application/json
Security

Each API must address the following security concerns:

Transport

Always use HTTPS to communicate with and between APIs. A best practice is to use TLS by default.

For internal APIs hosted in our IAD data center, this usually adds a performance penalty and can be dropped. As we move to AWS hosted services in 2016 and beyond, be sure to watch for this.

The goal here is confidentiality by hiding/encrypting sensitive information so that unauthorized 3rd parties cannot see the textual data transmitted between an API Producer and Consumer.

Authentication/Authorization

The purpose of Authentication is to validate a service consumer (i.e., subject/identity), which could be a:

Authorization ensures that a subject/identity has permission to use/access services and resources in a system. Authorization usually occurs as part of or after Authentication. In this case, Authorization ensures that a service consumer has the right to access and use an API.

CORS (Cross-Origin Resource Sharing)

Web and mobile applications can only make HTTP requests to the site (i.e., domain) they're currently displaying. For example, if the UI is running on www.myapp.com, it can't perform an HTTP request against www.yourapi.com. There are 2 ways to handle this:

CORS defines a mechanism in which an API and its Consumers can collaborate to determine if it's safe to allow the cross-origin request by leveraging HTTP Headers.

The Origin is the only relevant CORS-related HTTP Request Headers, and it indicates the origin (i.e., domain) of the cross-site access request. Upon receiving an HTTP Request, an API can check the Origin header's value (i.e., the originating domain - myoriginatingdomain.com, for example) against a whitelist of allowed domains. If the originating domain is not found in the whitelist, then the API should return a 403 (Forbidden) Status Code.

Otherwise, if the originating domain is found in the whitelist, then the API performs the request and populates the following CORS-related HTTP Response Headers:

| HTTP Response Header | Description | Example | | ——————– | ———– | ——- | | Access-Control-Allow-Origin | Indicates whether a resource can be shared based by returning the value of: the Origin request header, *, or null. Even though * is allowed by the CORS spec, don't use it because this implies that every domain is acceptable. Use the value of the Origin request header instead. | myoriginatingdomain.com | | Access-Control-Allow-Credentials | Indicates whether the response to a request can be exposed when the omit credentials flag is unset. When part of the response to a preflight request it indicates that the actual request can include user credentials. | true | | Access-Control-Allow-Headers | Indicates which HTTP headers are safe to expose to the API of a CORS API specification. | Authorization | | Access-Control-Allow-Methods | Indicates, as part of the response to a pre-flight request, which HTTP Methods can be used during the actual request. | GET, POST, PUT, DELETE |

CORS Implementations

There are a couple of good implementations:

To have common/reusable CORS functionality in a single place across our APIs (i.e., the DRY Principle), we should consider:

Security References
Responses

When a Consumer invokes an API, one of three things can happen:

Each API needs to handle errors and convey an appropriate status and error message in the HTTP Response.

HTTP Status Codes

Here is a list of the most common HTTP Status Codes, their usage, and contextual meaning.

| HTTP Status Code | Applicable HTTP Verb / Method | Meaning | |:—————–|:——————————|:———————————————————————————————————————————————————————————————————-| | 200 | GET / PUT / DELETE | OK. The API processed the request properly without error. | | 201 | POST | Created. API successfully created the Resource. The URI of the newly created Resource should go in the Location Header of the Response. | | 202 | POST / PUT / DELETE | Accepted. The Request has been been received, has not been completed, and will be processed later. | | 204 | POST / PUT / DELETE | No Content. API successfully processed the request, and is not returning data. | | 301 | GET / PUT / DELETE | Moved Permanently. This and all future requests should be directed to the new URI given in the Response. | | 304 | GET | Not Modified. The resource hasn't changed since the previous GET request. Consumer must provide the following Headers: Date, Content-location, ETag. | | 400 | ALL | Bad Request. Due to Malformed URI or invalid data in the request parameters or body. | | 401 | ALL | Unauthorized. Indicates that the Consumer has not provided appropriate credentials or that the supplied credentials are invalid. | | 403 | ALL | Forbidden. Similar to 401 (Unauthorized), but the API chose not to respond. This could be due to the fact that the HTTP Verb for the Request URI is not allowed. | | 404 | GET / PUT / DELETE | Not Found. The API couldn't find the resource specified by the Request URI. | | 409 | PUT | Conflict. The Request can't be processed due to an edit conflict on a resource. This could be caused by a race condition on the data to multiple updates from different consumers on the same resource. | | 410 | ALL | Gone. The resource is unavailable and will not be available again. | | 500 | ALL | Internal Server Error. The request couldn't be processed due to a server-side processing exception. | | 501 | ALL | Not Implemented. The request is valid, but functionality has not been implemented for the HTTP Verb and Request URI. | | 503 | ALL | Service Unavailable. The API (i.e., server) is temporarily unavailable. The server could be overloaded or down for maintenance. |

Error Handling / Messages

Error Handling / Messaging requirements are different for Internal APIs and Consumer-Facing APIs.

Internal API Error Handling

HTTP Status Codes are sufficient for expressing statuses and error conditions for APIs that are consumed by other APIs. Internal APIs could handle errors (especially uncaught exceptions) and return the information in the JSON response as follows:


essage": "Error message high-level description.",
xception": "[detailed stacktrace or error message]"

In production, we shouldn't have to put error messages in the response.

Consumer-Facing API Error Handling

Consumer-facing applications (both web and mobile) - that consume an API and display error information to the end user - need more contextualized information is needed for consumer-facing applications. In this case, errors that include highly contextualized (localized, relevant to the action they're taking, etc) error messages would provide a much better experience to the user when there is an error.

Based on the JSON API Error Formatting Guidelines, APIs used by consumer-facing applications could provide the following fields in the error response:

Here's an example error message:


rrors": [
{
  "code": "unrecoverable_error",
  "title": "The flux capacitor disintegrated",
  "details": "Hold on, the end is nigh.",
  "user_message": {
    "default": "OMG, panic!",
    "en-GB": "Keep a stiff upper lip",
    "de-DE": "Schnell, schnell!!"
  }
}


Response Validations

The ls-api_validation gem automates response validations. The gem looks at either your Swagger documentation or published JSON Schema documents to ensure your responses live up to their promises. In development ls-api_validation will raise an exception if the schema is invalid and in production it will validate a percentage of responses (1% by default) and log statuses. These actions are also configurable so you can set these to match your use cases.

Error Handling References

Here are some error handling examples from other API Design Guides:

Response Envelopes and Hypermedia

Core RESTful principles are great at describing which HTTP verb to use and how to design manageable URIs. But there are two problems that need to addressed in the Response:

HAL Specification

Of the several Hypermedia specifications we reviewed (see Hypermedia Specifications Considered), we prefer HAL because:

Here's an example:

ET /orders/523 HTTP/1.1
ost: example.org
ccept: application/hal+json

TTP/1.1 200 OK
ontent-Type: application/hal+json


 "_links": {
   "self": { "href": "/orders" },
   "next": { "href": "/orders?page=2" },
   "find": { "href": "/orders{?id}", "templated": true }
 },
 "_embedded": {
   "orders": [
     {
       "_links": {
         "self": { "href": "/orders/123" },
         "basket": { "href": "/baskets/98712" },
         "customer": { "href": "/customers/7809" }
       },
       "total": 30,
       "currency": "USD",
       "status": "shipped"
     },
     {
       "_links": {
         "self": { "href": "/orders/124" },
         "basket": { "href": "/baskets/97213" },
         "customer": { "href": "/customers/12369" }
       },
       "total": 20,
       "currency": "USD",
       "status": "processing"
     }
   ]
 },
 "currentlyProcessing": 14,
 "shippedToday": 20

In the above document:

Please note that the _links object is not required by each order object - it is optional in that context.

Here are the requirements for a simple valid HAL document:

HAL is simple and works well, and we should use it for every JSON response from our APIs. Here are a couple of example use cases for HAL:

Developer Libraries for working with HAL

Here's a list of developer libraries for working with HAL. The most promising libraries are:

The above list is a small subset of the available HAL libraries, so individual development teams should research and try out several options to find what works best for them.

Hypermedia Specifications Considered

There are several Hypermedia specifications that enable developers to link API responses and standardize API response formats:

We reviewed the above options for a consistent envelope that standardizes:

In general, we found that all of the above met these needs, but there were issues with the Data and Metadata structures required by most of these specifications:

Hypermedia References Response Meta Data

Occasionally there will be a need to return meta-data about a response that doesn't belong in the JSON for the individual objects in the response. This data should go in a hash in a top level meta attribute. For instance, catalog-service can return offers using one of many different 'representations'. To include the name of the representation used for this particular response we might return something like this:

/offers/{guid}.json

eta": {
"representation": "tile",

ffer": {
"id": 123,
... other attrs ...



/offers.json?some=search_query

eta": {
"representation": "tile",

ffers": [ ..list of offers... ],
.

JSON

Our APIs and Kafka messages use JSON because it is the format of choice for most modern Web APIs. JSON provides interoperability and cleanly converts to programming constructs (i.e., objects, arrays, key/value pairs) in most development languages.

Data Formatting Dates and Times

We use RFC 3339 to represent Dates and Times in UTC (Coordinated Universal Time) format for interoperability. Here's an example of an RFC 3339-compliant date/time:

-09-08T22:47:31Z-05:00

Here's an example of the date in a JSON document:


.
reated_at": "2008-09-08T22:47:31Z-05:00"
.

Here are the formatting guidelines:

The Relationship Between RFC 3339 and ISO 8601

RFC 3339 and ISO 8601 both provide a standard for representing dates and times using the Gregorian Calendar. Per Wikipedia and other sources:

UTF-8

Each API should use UTF-8 when it returns a JSON response because:

Just add a charset notation in the HTTP Content-Type response Header:

ent-Type: application/json; charset=utf-8
Currency and Fixed-Point Values

Monetary (i.e., Currency) information (e.g., the price of an offer (option)) should follow the ISO 4217 - Currency Code standard:

Example:


rice": [
{?currency_code": "USD", "value": 1000, "exponent": 2 },
{"currency_code": "JPY", "value": 200, "exponent": 0 }


In the above example:

Fixed-point values (e.g. discount percentage) should follow a similar format:


ax_rate":
{"value": 875, "exponent": 2 }

In the above example:

Third Parties and XML

Sometimes third parties (e.g., San Diego Zoo, SeaWorld/Busch Gardens) use XML, and our applications would simply consume the XML and convert it to JSON. The use of XML should be limited to data exchange with the third parties that still use XML.

Consumer Tooling
Ruby Consumer Tools

Client gems are a bit of an anti-pattern that can lead to service logic being duplicated client-side. Moving forward we'd like teams to rely on libraries like ls-api_client and ls-api_model to integrate their new services into applications. These libraries are fairly stable but improvements are always welcome.

Artifacts

Each API should come with the following artifacts to help consumers use the API:

Documentation

Each API should have human-readable documentation so that consumers know how to use the API.

All of our APIs should use Swagger to generate documentation. Many of our APIs already use it successfully. With Swagger, you use a JSON to specify an API's endpoints, and then generate the documentation as HTML.

Our internal Swagger documentation lists all published swagger docs for all implementing services.

Here are some resources for getting started with Swagger:

Executable Examples
Performance

Both the API Consumer (i.e., the Client) and Producer (i.e., the Server) play a role in improving API performance.

Consumer-Side

The fastest API is call is one that isn't made in the first place. The API Consumer should consider caching if it can live with data that is slightly out of date. Please see SOA Series Part 4: Caching Service Responses Client-Side for details on how to implement this technique.

Producer-Side

There are several ways to improve the performance of an API on the Producer/Server side:

Condense JSON Response with HTTP Compression

HTTP Compression improves data transfer speeds and reduces bandwidth usage. There are 2 common compression schemes:

We should consider leveraging GZip to compress JSON Responses. Many implementations have seen a 60-80% reduction in payload size. Here are a couple of implementation suggestions:

Conditional GET with Caching and ETag

Caching improves scalability by reducing calls to retrieve requested data, either from databases or other services. Each API should include an ETag HTTP Header in all responses to identify the specific version of the returned resource. The following table describes the other HTTP Request Headers that work with the ETag header:

| HTTP Request Header | Description | Example | |:——————–|:——————————————————————————————————————————————————-|:————————————————–| | Cache-Control | The maximum number of seconds (max age) a response can be cached. However, if caching is not supported for the response, then no-cache is the value. | Cache-Control: 360 or Cache-Control: no-cache | | Date | The date and time that the message was sent (in RFC 1123 format). | Date: Sun, 06 Nov 1994 08:49:37 GMT | | Pragma | When Cache-Control is set to no-cache, this header is also set to no-cache. Otherwise, it is not present. | Pragma: no-cache |

The following table describes the related HTTP Response Headers:

| HTTP Response Header | Description | Example | |:———————|:———————————————————————————————————————————————————————————————————————————————————————————————————|:————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————–| | Cache-Control | The maximum number of seconds (max age) a response can be cached. However, if caching is not supported for the response, then no-cache is the value. | Cache-Control: 360 Cache-Control: no-cache | | Date | The date and time that the message was sent (in RFC 1123 format). | Date: Sun, 06 Nov 1994 08:49:37 GMT | | | | ETag | Useful for validating the freshness of cached representations, as well as helping with conditional read and update operations (GET and PUT, respectively). Its value is an arbitrary string for the version of a representation, often a Hash that represents the value of the underlying domain object. | ETag: "686897696a7c876b7e" | | Expires | If max age is given, contains the timestamp (in RFC 1123 format) for when the response expires, which is the value of Date (e.g. now) plus max age. If caching is not supported for the response, this header is not present. | Expires: Sun, 06 Nov 1994 08:49:37 GMT | | Last-Modified | The timestamp that the resource itself was modified last (in RFC 1123 format). | Last-Modified: Sun, 06 Nov 1994 08:49:37 GMT | | Pragma | When Cache-Control is set to no-cache, this header is also set to no-cache. Otherwise, it is not present. | Pragma: no-cache |

Here's an example set of HTTP Response Headers in response to a GET request on a resource that enables caching for one day (24 hours):

e-Control: 86400
: Wed, 29 Feb 2012 23:01:10 GMT
-Modified: Mon, 28 Feb 2011 13:10:14 GMT
res: Thu, 01 Mar 2012 23:01:10 GMT

In the above example, the API would:

Rather than duplicating the Cache-Control header vs Date header in each API, factor it out to a common place:

Performance References
Testing

Here's what most developers test for:

Ruby Test Tooling

Here's a typical Rails-based API test environment:

Mobile Test Tooling

The Mobile teams do something similar to VCR. They created VCRURLConnection, an iOS and OSX API to record and replay HTTP interactions. Any API to be consumed by a mobile app should use VCRURLConnection as part of their test environment.

In the future, the Mobile teams would like to have something similar to Mock Mode in their test suite.

Improve API Test Performance

A test can pull from 3 data sources when exercising an API:

Consider the following techniques to improve the performance of API testing:

API Developer Guides - Implementing the API Design Guide on our Platforms

This document only covers what an API should look like, but not how to implement it. To see best practices for implementing APIs that fit with this Design Guide, please refer (and contribute) to the following pages:

Why Do We Have Separate API Developer Guides?

The Developer Guides are separate from the API Design Guide to group issues/concerns at the right level. The goal here is to maintain a clear focus in each document. Furthermore, we have 2 development platforms at LivingSocial, each of which has its own set of unique challenges and implementation concerns.

Other API Design Guides

In addition to our experience, we've drawn on the the following external guides to help in the development of this API Design Guide:

JSON References

This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.