# Test-Drive Your API with a Validated OpenAPI Spec

## Background

When I worked for Arity (an Allstate startup), Allstate had contracted with Pivotal Labs to facilitate its agile transformation. Pivotal's model was one of enablement - they would augment a cross-functional product team with their own cross-functional staff, instill processes and principles onto the core team, then gradually peel away, leaving the core team newly capable of best (or better) agile practices.

During this time I was introduced by Pivotal engineers to consumer-driven contract testing. This is an API development methodology in which API consumers drive the development of new and existing API contracts. Here is an example workflow:

1. API consumer desires a new endpoint
    
2. API consumer writes a new contract test in the API codebase asserting that the new endpoint exists, as well as accepts and returns the desired data
    
3. API consumer opens a pull request with the new contract test. This test immediately fails, as it's not implemented
    
4. From the new and existing contract tests, stubs are generated for use in creating a mock server
    
5. API consumer uses new stubs as a basis for new work. For example, a new view in a mobile app requires this new endpoint, and this endpoint doesn't exist yet in production.
    
6. API provider takes over the pull request with failing contract tests and implements the new endpoint. Contract tests now pass.
    
7. API provider merges changes and deploys to production.
    
8. API consumer turns on the new mobile app view.
    

There is immense value in this approach:

1. Expression of intent. The desired operation of the API is expressed in code in the form of a contract test. In the above workflow, this is usually in some contract testing-specific DSL or language, such as Groovy.
    
2. Validation of intent. The execution of contract tests validates that the API operates as intended.
    
3. Decoupling consumers and producers. With the creation of stubs, consumers are free to develop against a new endpoint before it's been implemented by the producer.
    

The approach I'll offer in this article is a similar one, focused on the expression and validation of intent. Where I'll deviate from the approaches supported by tools such as [Sprint Cloud Contract](https://cloud.spring.io/spring-cloud-contract/reference/html/getting-started.html#getting-started) and [Pact](https://docs.pact.io/) is that I suggest using an API specification (we'll use [OpenAPI](https://www.openapis.org/)) as the cornerstone of contract testing instead of manually-written tests.

## Motivating Example

For the remainder of the article, I'll use an example API and its API spec to demonstrate how to generate and execute contract tests from that spec.

The example API we'll use is one of dinosaurs - a CRUD API allowing the storage and retrieval of dinos. I'll start with endpoints to add a new dinosaur and fetch all dinosaurs. The example includes an OpenAPI specification with these endpoints.

Then, I'll demonstrate how to generate and execute contract tests with [Dredd](https://dredd.org/en/latest/) before adding a new endpoint to fetch an individual dinosaur. In a subsequent article, I'll show how this can be incorporated into CI with GitHub Actions for continuous validation.

All code as it exists both at our starting point and at the conclusion of this article can be found on GitHub at [christherama/dino-api](https://github.com/christherama/dino-api). There are two ways to get to the starting point:

1. Clone or fork the above repo and checkout the commit for the starting point of this repo:
    
    ```bash
    git clone git@github.com:christherama/dino-api.git
    git checkout v0.0.1
    ```
    
2. Download and extract an archive of the repo at the commit of our starting point at [christherama/dino-api/archive/refs/tags/v0.0.1.zip](https://github.com/christherama/dino-api/archive/refs/tags/v0.0.1.zip)
    

## Prerequisites

* [Docker Desktop](https://docs.docker.com/get-docker/)
    

## Exploring the Dino API

After you've cloned or forked the above repo, or have extracted an archive at the indicated tag, build the Docker image locally (this could take a few minutes):

```bash
docker build -t dino-api:local .
```

Then migrate the database:

```bash
docker run -v $(pwd)/db:/usr/app/db dino-api:local python manage.py migrate
```

And finally, start the app:

```bash
docker run -p 8000:8000 -v $(pwd)/db:/usr/app/db dino-api:local
```

With the app running, you can now visit the API documentation, which is available at [http://localhost:8000/api/docs/](http://localhost:8000/api/docs/).

You can use Postman or cURL to see that we have no dinosaurs in our system yet (output simplified):

```plaintext
curl -v http://localhost:8000/api/dinosaurs/

< HTTP/1.1 200 OK
< Content-Type: application/json
<
[]
```

To add a dino, use the POST endpoint (output simplified):

```plaintext
curl POST http://localhost:8000/api/dinosaurs/ -H "Content-Type: application/json" -d '{"common_name": "T-Rex", "scientific_name": "Tyrannosaurus Rex"}'

> POST /api/dinosaurs/ HTTP/1.1
> Content-Type: application/json
>
< HTTP/1.1 201 Created
< Content-Type: application/json
<
{"id":1,"common_name":"T-Rex","scientific_name":"Tyrannosaurus Rex"}
```

## Generate Contract Tests with Dredd

We now turn to the OpenAPI specification already present in the repo, using it as a basis for generating contract tests to execute against the running API. Here are the general assumptions:

1. Each endpoint we wish to test is documented in the OpenAPI spec
    
2. Each endpoint includes at least one example (refer to this [Swagger guide](https://swagger.io/docs/specification/adding-examples/) for adding examples)
    
3. Any responses containing data previously added will have a previous request that added that data. For example, documentation of a POST request adding a dino should *precede* one that asserts the dino is present in a GET request of that dino.
    

For generating and executing contract tests, we'll use [Dredd](https://dredd.org/en/latest/). Before diving in, note the following:

> The tests generated and executed by Dredd are ephemeral. That is, they will not be stored in any way that requires manual intervention or maintenance.

This is one of the key values of this approach - since *only* the API spec is used for generating contract tests, it is the only artifact requiring maintenance. There is no additional language or DSL (like Groovy or non-OpenAPI YAML as with Spring Cloud Contract). There is only the OpenAPI spec, which already integrates with a [plethora of tools](https://openapi.tools/).

#### Running Dredd Against the Dino API

Though there are options to run Dredd using a node package, we'll stick with the docker image for optimal portability. With the API still running, open a new terminal window or tab in the root of the repo and run the following:

```bash
docker run \
  -v $(pwd)/api/docs:/api \
  apiaryio/dredd \
  dredd /api/openapi.yaml http://host.docker.internal:8000
```

Here I volume-mount the `api/docs` directory of our repo to the `/api` directory of the running `dredd` container. This ensures that our API spec at `api/docs/openapi.yaml` is available to the `dredd` container at `/api/openapi.yaml`, which I pass as the first argument to the `dredd` command. The second argument is the host and port of the API under test, `http://host.docker.internal:8000`. The hostname `host.docker.internal` is specific to Docker Desktop and resolves to an internal IP used by the host machine. We need this since the `dredd` container is running on its own network and `localhost` will loop back to the container itself. The same could be accomplished by using `--network host` and using `localhost` as the hostname.

Assuming you have the API running on the same machine and you've migrated the database, running the above command should produce output that indicates success:

```plaintext
docker run \
  -v $(pwd)/api/docs:/api \
  apiaryio/dredd \
  dredd /api/openapi.yaml http://host.docker.internal:8000

pass: POST (201) /api/dinosaurs/ duration: 196ms
pass: GET (200) /api/dinosaurs/ duration: 42ms
complete: 2 passing, 0 failing, 0 errors, 0 skipped, 2 total
complete: Tests took 244ms
```

Now that we have passing contract tests, let's turn to adding a new endpoint.

## Contract Test-Drive a New Endpoint

Now we want a new endpoint for fetching an individual dino. For this, I'll demonstrate a new workflow:

1. Add the endpoint to the OpenAPI spec
    
2. Generate and run contract tests (they should fail)
    
3. Implement the new endpoint
    
4. Repeat 2. Tests should now pass.
    

#### Add the endpoint to the OpenAPI spec

To fetch a new endpoint, we'll use a `GET` request to `/api/dinosaurs/{id}/`, which we'll now add to `api/docs/openapi.yaml`:

```yaml
...
paths:
  ...
  /api/dinosaurs/{id}/:
    get:
      summary: Get a dino
      description: Retrieve a dinosaur by id
      parameters:
        - in: path
          name: id
          schema:
            type: integer
          required: true
          description: Unique identifier of dino
          example: 1
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                type: object
                properties:
                  id:
                    type: integer
                  common_name:
                    type: string
                  scientific_name:
                    type: string
```

#### Generate and run contract tests

After saving the above changes, generating and running contract tests results in a failure (output simplified):

```plaintext
docker run \
  -v $(pwd)/api/docs:/api \
  apiaryio/dredd \
  dredd /api/openapi.yaml http://host.docker.internal:8000

pass: POST (201) /api/dinosaurs/ duration: 129ms
pass: GET (200) /api/dinosaurs/ duration: 38ms
fail: GET (200) /api/dinosaurs/1/ duration: 87ms
info: Displaying failed tests...
fail: GET (200) /api/dinosaurs/1/ duration: 87ms
fail: headers: At '/content-type' No enum match for: "text/html; charset=utf-8"
body: Can't validate actual media type 'text/plain' against the expected media type 'application/json'.
statusCode: Expected status code '200', but got '404'.

complete: 2 passing, 1 failing, 0 errors, 0 skipped, 3 total
complete: Tests took 260ms
```

Yay for failure! We are officially test-driving the API with our spec. The reference to content-type failures in the output is an implementation detail of Django. With our current Django configuration, a `404` response returns HTML content when it can't find a URL mapping to the one requested. Let's now turn to the implementation of that endpoint.

#### Implement the new endpoint

We'll start with adding a Django view to the bottom of `api/views.py`:

```python
from rest_framework.generics import ListCreateAPIView, RetrieveAPIView

...

class DinosaurView(RetrieveAPIView):
    serializer_class = DinosaurSerializer
    permission_classes = []

    def get_queryset(self):
        return Dinosaur.objects.all()
```

Next we'll map a new URL to this view. In `api/urls.py`, add the following entry to `urlpatterns`:

```python
urlpatterns = [
    ...
    path("dinosaurs/<int:pk>/", views.DinosaurView.as_view(), name="dinosaur"),
]
```

#### Rerun contract tests

Before rerunning tests, let's do the following:

1. Stop the Django container using <kbd>CTRL+C</kbd>
    
2. Rebuild the image
    
3. Delete the existing database. This ensures that a request to `GET /api/dinosaurs/1/` will succeed. In the scenario that the dino with id `1` has been deleted, the next one added will have an `id` greater than `1`, leading to a contract test failure.
    
4. Create a new database and migrate
    
5. Start the container again
    

```plaintext
# Stop the container with CTRL+C

# Rebuild the Django image
docker build -t dino-api:local .

# Delete the existing database
rm db/db.sqlite3

# Create a new database and migrate
docker run -v $(pwd)/db:/usr/app/db dino-api:local python manage.py migrate

# Run the Django container
docker run -p 8000:8000 -v $(pwd)/db:/usr/app/db dino-api:local
```

Now let's rerun our contract tests:

```plaintext
docker run -v $(pwd)/api/docs:/api apiaryio/dredd dredd /api/openapi.yaml http://host.docker.internal:8000

pass: POST (201) /api/dinosaurs/ duration: 243ms
pass: GET (200) /api/dinosaurs/ duration: 46ms
pass: GET (200) /api/dinosaurs/1/ duration: 97ms

complete: 3 passing, 0 failing, 0 errors, 0 skipped, 3 total
complete: Tests took 392ms
```

Pass! Congratulations, you've now driven API development with tests.

## Limitations

With the introduction of a test-driven API development workflow, it's important to understand its limits. These tests are *contract* tests, not functional tests. That is, we're testing only the *shape* of the API, and not the actual *content*. To clarify this distinction, consider the example of sending the following API request:

```plaintext
POST /api/dinosaurs/

{
  "common_name": "t-rex",
  "scientific_name: "tyrannosaurus rex"
}
```

If your API is expected to title-case the value "tyrannosaurus rex" into "Tyrannosaurus Rex", these tests won't catch that. They are, after all, only *contract* tests, in this case ensuring that the response contains the expected property names and that their values are of the expected data types. Asserting that lowercase values are converted to title case values is the subject of unit tests in the repository, not contract tests.

This is to say that contract tests are not meant to be a substitute for other kinds of testing, be that unit, component/integration, or end-to-end.

## Recap and Next Steps

In this article I introduced and demonstrated a workflow for driving the development of an API:

1. Have an OpenAPI spec
    
2. Update the spec with a new endpoint, or a change to an existing endpoint
    
3. Generate and run contract tests with [Dredd](https://dredd.org/en/latest/index.html)
    
4. Implement or fix if needed, according to test results
    

Ensuring an API specification stays in lockstep with implementation is critical for consumers of an API. As such, validating this in a continuous integration pipeline is the next natural step.

In my next article, I'll demonstrate how to add this to CI in GitHub Actions for continuous validation of an API against the expressed intent of the spec.