Testing and CI/CD Deployment

Automated Testing (pytest)

Learning Outcome

4

Apply fixtures, parametrize, and TestClient to API tests.

3

Recognize the difference between unit and integration tests.

2

Identify pytest conventions: test_ files, test_ functions, assert.

1

Understand why automated tests protect a prediction API.

5

Connect model and LLM output tests to quality gates.

In competitive exams, a machine checks OMR sheets using an answer key.

It checks every answer, on every sheet

Checking OMR Answer Sheets

In the same way, within seconds, and runs again whenever new sheets arrive.

Why use a machine instead of checking every sheet by hand?

Automatic checking is fast, complete, and never gets tired.

 

Similarly, after every change to our API, we need to check that everything still works:

We write small checks with expected answers, called tests, and a tool runs all of them with one command.This is called Automated Testing with pytest.

 

/health returns 200

Wrong input returns 422

Predictions come in the right format

What is Automated Testing?

Writing code that checks your code: known inputs in, expected results confirmed.

Runs in seconds, as often as needed, with exactly the same checks every time.

Regressions

Environment changes

Model changes

a change quietly breaks something that used to work.

a library upgrade or new base image changes behaviour.

a new model loads fine but returns a different format or worse quality.

How pytest Works

Test files

Test functions

Plain assert

Named test_*.py or *_test.py, usually in a tests folder.

Any function starting with test_.

Failures show actual vs expected side by side.

One command

Pytest runs everything; -v lists tests, -x stops at the first failure, -k predict filters by name.

Code

def test_health_returns_ok():
    response = client.get("/health")
    assert response.status_code == 200

Follow Arrange, Act, Assert: one behaviour per test, with a name that explains a failure.

Fixtures and Parametrisation

Prepares something tests need, such as a client or a valid sample input.

Runs one test with many inputs.

Tests receive a fixture by listing its name as a parameter; share fixtures via conftest.py.

Code

@pytest.mark.parametrize("age", [15, 61, -3])
def test_invalid_age_rejected(client, valid_input, age):
    payload = {**valid_input, "age": age}
    response = client.post("/predict", json=payload)
    assert response.status_code == 422

Testing a FastAPI Service with TestClient (Core Mechanism)

Create client

= TestClient(app).

No Uvicorn, no port, no network.

Send a request: client.get("/health") or client.post("/predict", json=payload).

Assert on the status code and response fields.

 Cover the suite: health and info, the happy path, 422 validation, 404/503 errors, batch limits and per-item errors.

Swap the real model for a small fake (dependency overrides, monkeypatch, or unittest.mock) for fast API tests; test the real model separately.

 

Testing the Model Itself

Smoke

Minimum quality

Invariance

Directional

Loads; one prediction per row; probabilities between

0 and 1.

On a fixed held-out set, F1 ≥ 0.75 (or the agreed metric).

 Changing the student's name doesn't change the prediction.

More support tickets never lower churn probability.

A model can pass every API test and still predict badly. Model tests check behaviour:

 

These become the quality gate for retraining: a model that fails them is never promoted.

Measuring Test Coverage

Pytest-cov reports which lines ran during tests.

Shows the percentage per file and highlights untouched code, such as an error branch.

A guide, not proof code run without meaningful asserts still counts as covered.

Code

pytest --cov=app

Single Prediction vs Batch Prediction (Comparison)

AspectSingle PredictionBatch Prediction
          What it checksOne small piece in isolationSeveral pieces working together
        DependenciesFakes or mocksReal or realistic components
 
 SpeedMilliseconds
 
Seconds or longer
      When it failsPoints to the exact functionShows the system is broken, not always where
         Typical exampleFee validator rejects negativesPOST /predict with the real model
 AnalogyChecking one questionChecking the whole answer sheet

Think About This: Spotting the Difference

    

     

Unit: one validator, isolated.

A test that checks the email validator lowercases input

    

     

A test sending a real request to /predict with the real model from the registry

    

A test where a batch request writes to the real database

Unit tests find where; integration tests prove it all works together.

Integration: API, registry, and model together.

     

Integration: API and database together.

 Real-World Applications

Test inside the image

Docker run churn-api:1.0 pytest checks the exact pinned environment.

LLM features

Assert schema, required fields, length limits, and banned content, never exact text.

Next step

CI runs this suite on every code change automatically

Every bug becomes a test

so it can never return unnoticed.

Summary

5

Model tests check predictions make sense.

4

TestClient tests the API without a server.

3

Fixtures prepare; parametrize repeats inputs.

2

Name test_, check with assert, run pytest.

1

Tests are code that checks code.

Quiz

Which test catches a new model that returns valid JSON but noticeably worse predictions?

A.  /health returns 200

B.  A missing field returns 422

C. Coverage stays above 90%

D. F1 on a fixed held-out set stays at or above 0.75

Quiz-Answer

Which test catches a new model that returns valid JSON but noticeably worse predictions?

A.  /health returns 200

B.  A missing field returns 422

C. Coverage stays above 90%

D. F1 on a fixed held-out set stays at or above 0.75

9.1 Automated Testing (pytest)

By Content ITV

9.1 Automated Testing (pytest)

  • 233