Background
Once API guidelines are published, the next natural step is to ensure compliance of newly developed APIs with the published guidelines. There are 2 parts to validating guidelines compliance:
a. Static or design-time validation – Validating the API specification document (OpenAPI, WADL, RAML, Blueprint) for compliance with sections of guidelines applicable to interface definition.
b. Dynamic or run-time validation – Validating the API implementation for compliance with sections of guidelines applicable to runtime characteristics of the API. Some examples of such validation are:
a. Request/response payload, URL, headers are consistent with the interface specification
b. Response times are complying to ranges specified in guidelines
c. Security model consistent with guidelines (for example use TLS 1.2 or above)
The approach for static validation is straight-forward and involves using a software program like Spectral to take the API specification as input and checking for the compliance by parsing and processing the specification with code or configurations representing the guidelines.
The approach for dynamic validation is more complicated and is accomplished by one of the following approaches:
a. Instrument the validation into the service implementation (for example as a middleware) and for every API call received perform the processing. While feasible, the clear disadvantages of the approach are the overhead for every API provider to implement it, do it correctly and completely and performance impact due to additional processing. Further governance is a challenge because the validation is implicit to the service and requires code review to get visibility into it.
b. API Gateway – If the service implementation sits behind API Gateway, then a policy can be introduced in the gateway to perform validation of the input payload against the API specification and guidelines. The advantage is that once such a policy is implemented and instated, all APIs behind the gateway can make use of it without having to redo it. The implementation of the validation has to be reviewed only once. The disadvantage is that this assumes an API gateway in place and also incurs the performance cost of validation processing for every API call. The performance penalty can be avoided with an offline processing approach described below.
c. Log processing – In this approach every API call log is logged either through instrumentation in code or through an API Gateway policy if there is one. Then an offline processing step extracts the logs from the storage and runs the validation processing on it. This approach doesn’t have a performance cost since the validation processing is offline. However, the processing relies on completeness of information in the logs – for ex. URL, headers, payload, correlation between request/response for it to function correctly. Sometimes it is not feasible to log all information due to security or size restrictions. Further it needs programmatic access to retrieve the logs from store for processing which can be a challenge. This approach is complicated to implement.
The common characteristic of all the above approaches for dynamic validation is that validation is performed on the service side and that does not take into account the client-side view/experience of the API.
Client-side Validation
The method proposed solves the problem by turning the focus around to the client-side of the API equation. The core of the idea is described below.
For performing any dynamic validation of an API, it is necessary to have that API invoked. During testing (functional, regression, coverage, contract, performance, stress) of APIs invariably the APIs are exercised in a complete way (depth and breadth). Further, the API testing apparatus (scripts, infrastructure etc.) is under the control of the API provider and is used during the development, testing, staging and production phases of the API delivery as necessary. By performing the following steps when the regular API testing is done, dynamic validation can be seamlessly accomplished.
Setup
Steps
1. On client machine which executes the API tests install a packet capture/web proxy tool (Fiddler, Charles proxy, Wireshark, tcpdump etc.) if it is not already installed.
2. Start the packet capture/web session recording in the tool to capture packets for the duration of tests.
3. Enable filter in the capture based on API endpoint URL (scheme, hostname, resources).
4. Run the API test cases as you normally would.
5. After tests complete, stop the packet capture tool and export/extract the capture output to a file for subsequent processing.
6. Since each tool has a capture format specific to the tool, use a tool-specific plugin to convert the output to a normalized format (for ex. json with a standard schema for API request/response, response times, connection handshake etc.).
7. With the normalized data in above step and the API specification as inputs run the compliance tests implemented as code/config representing the run-time aspects of the API guidelines specification.
8. Generate validation report based on compliance tests and store it (in file, database, object store etc.) for further consumption.
Flowchart
Advantages
1. There is no dependency on having an API gateway on the service side. There is no dependency on logging instrumentation in the service implementation and ability to get complete logs from the store.
2. The is no performance impact on the API processing because the data is collected non-intrusively on the client-side.
3. All the data for the API call is available because packet capturing tools capture all data flowing on the wire. Further the data is tamper-proof because it is collected at the source by the tool unlike a logging or middleware-based approach where there is scope for modification.
4. Approach is totally agnostic to API implementation language/runtime which is the case for a logging or middleware-based approach.
5. Implement once and use many times approach which can work with any/all REST/HTTP API implementations.
PS For the record:
-
we didn’t go down the path of implementation due to changed priorities
-
PoC was done using Fiddler and Charles Proxy as interceptors