Content ITV PRO
This is Itvedant Content department
Learning Outcome
4
Apply fixtures, parametrize, and TestClient to API tests.
3
Recognize the difference between unit and integration tests.
2
Identify pytest conventions: test_ files, test_ functions, assert.
1
Understand why automated tests protect a prediction API.
5
Connect model and LLM output tests to quality gates.
In competitive exams, a machine checks OMR sheets using an answer key.
It checks every answer, on every sheet
Checking OMR Answer Sheets
In the same way, within seconds, and runs again whenever new sheets arrive.
Why use a machine instead of checking every sheet by hand?
Automatic checking is fast, complete, and never gets tired.
Similarly, after every change to our API, we need to check that everything still works:
We write small checks with expected answers, called tests, and a tool runs all of them with one command.This is called Automated Testing with pytest.
/health returns 200
Wrong input returns 422
Predictions come in the right format
What is Automated Testing?
Writing code that checks your code: known inputs in, expected results confirmed.
Runs in seconds, as often as needed, with exactly the same checks every time.
Regressions
Environment changes
Model changes
a change quietly breaks something that used to work.
a library upgrade or new base image changes behaviour.
a new model loads fine but returns a different format or worse quality.
How pytest Works
Test files
Test functions
Plain assert
Named test_*.py or *_test.py, usually in a tests folder.
Any function starting with test_.
Failures show actual vs expected side by side.
One command
Pytest runs everything; -v lists tests, -x stops at the first failure, -k predict filters by name.
Code
def test_health_returns_ok():
response = client.get("/health")
assert response.status_code == 200Follow Arrange, Act, Assert: one behaviour per test, with a name that explains a failure.
Fixtures and Parametrisation
Prepares something tests need, such as a client or a valid sample input.
Runs one test with many inputs.
Tests receive a fixture by listing its name as a parameter; share fixtures via conftest.py.
Code
@pytest.mark.parametrize("age", [15, 61, -3])
def test_invalid_age_rejected(client, valid_input, age):
payload = {**valid_input, "age": age}
response = client.post("/predict", json=payload)
assert response.status_code == 422Testing a FastAPI Service with TestClient (Core Mechanism)
Create client
= TestClient(app).
No Uvicorn, no port, no network.
Send a request: client.get("/health") or client.post("/predict", json=payload).
Assert on the status code and response fields.
Cover the suite: health and info, the happy path, 422 validation, 404/503 errors, batch limits and per-item errors.
Swap the real model for a small fake (dependency overrides, monkeypatch, or unittest.mock) for fast API tests; test the real model separately.
Testing the Model Itself
Smoke
Minimum quality
Invariance
Directional
Loads; one prediction per row; probabilities between
0 and 1.
On a fixed held-out set, F1 ≥ 0.75 (or the agreed metric).
Changing the student's name doesn't change the prediction.
More support tickets never lower churn probability.
A model can pass every API test and still predict badly. Model tests check behaviour:
These become the quality gate for retraining: a model that fails them is never promoted.
Measuring Test Coverage
Pytest-cov reports which lines ran during tests.
Shows the percentage per file and highlights untouched code, such as an error branch.
A guide, not proof code run without meaningful asserts still counts as covered.
Code
pytest --cov=appSingle Prediction vs Batch Prediction (Comparison)
| Aspect | Single Prediction | Batch Prediction |
|---|---|---|
| What it checks | One small piece in isolation | Several pieces working together |
| Dependencies | Fakes or mocks | Real or realistic components |
| Speed | Milliseconds | Seconds or longer |
| When it fails | Points to the exact function | Shows the system is broken, not always where |
| Typical example | Fee validator rejects negatives | POST /predict with the real model |
| Analogy | Checking one question | Checking the whole answer sheet |
Think About This: Spotting the Difference
Unit: one validator, isolated.
A test that checks the email validator lowercases input
A test sending a real request to /predict with the real model from the registry
A test where a batch request writes to the real database
Unit tests find where; integration tests prove it all works together.
Integration: API, registry, and model together.
Integration: API and database together.
Real-World Applications
Test inside the image
Docker run churn-api:1.0 pytest checks the exact pinned environment.
LLM features
Assert schema, required fields, length limits, and banned content, never exact text.
Next step
CI runs this suite on every code change automatically
Every bug becomes a test
so it can never return unnoticed.
Summary
5
Model tests check predictions make sense.
4
TestClient tests the API without a server.
3
Fixtures prepare; parametrize repeats inputs.
2
Name test_, check with assert, run pytest.
1
Tests are code that checks code.
Quiz
Which test catches a new model that returns valid JSON but noticeably worse predictions?
A. /health returns 200
B. A missing field returns 422
C. Coverage stays above 90%
D. F1 on a fixed held-out set stays at or above 0.75
Quiz-Answer
Which test catches a new model that returns valid JSON but noticeably worse predictions?
A. /health returns 200
B. A missing field returns 422
C. Coverage stays above 90%
D. F1 on a fixed held-out set stays at or above 0.75
By Content ITV