How to Sanely Test Complex Systems

test coverage

100% is too little

//inverse.js
function inverse(x) {
  return 1 / x;
}


// inverse.test.js
describe("inverse", () => {
  it("should return 1 / x", () => {
    expect(inverse(4)).toBe(0.25);
    expect(inverse(-8)).toBe(-0.125);
  });
});

100% Test Coverage

Uber's service topology

src: https://www.uber.com/blog/microservice-architecture/

Possible Solutions

Concept: Benefit

Test Seeds: stable data

Test Clocks: manageable time

Test Setups: repeatable state setting processes

My Assumptions About You

You have made manual tests in a system
You have experienced the time drain that is doing this by hand every week
You've attempted end-to-end integration tests
You've wondered if there's a better way

Test Seed

Test seeds are unchanging data objects in your system.

Any interactions with them give the same answer every time without fail.

Test Seed

GET /users/42
{ user_id: 42, balance: 10, ... }

GET /users/66
{ user_id: 66, balance: 0, ... }

POST /charge/
{amount: 1, used_id: 42}

Response 201
{id: 111}


POST /charge/
{amount: 1, used_id: 66}

Response 400
{"error": "insufficient funds"}

Stripe test cards
- e.g. 4242424242424242
Twilio magic numbers
- e.g. +1-500-555-0006

Test Seed

Examples in the Wild

Make time an object in your system.

Stop it for an object, a user, or the entire system.

Move it forward by hours or days.

Test Clock

POST /test-clock/
{ name: "1st of the Month", frozenTime: "2023-08-01" }

Response 201
{id: 111}


POST /gift-card/
{ id: 77, ..., expiresAt: "2024-07-31", testClockID: 111}


POST /test-clock/111/advance/
{ advanceTo: "2024-08-01"}


GET /gift-card/77
{ id: 77, ..., status: 'expired'}

Test Clock

Examples in the Wild

Plaid
Stripe

Test Setups

Make APIs to run various steps to put your system into a desired state.

Test Setups

POST /user/mock-bulk/
{ count: 10 }

Response 201
[
  {id: 110, name: "Joe Small", ...},
  {id: 111, name: "Abby Zales", ...},
  {id: 112, name: "Troy Barnes", ...},
  ...
]

Examples in the Wild

Shopify Bogus Gateway

Robert Roskam

@raiderrobert

mastodon | github

Work at