Overview

A unit test passes. The code it covers calls a method that the vendor SDK does not have. In production the first real call raises AttributeError, and the suite never said a word.

The cause is a mock that answers anything. MagicMock creates any attribute you ask for, so a test built on it checks that your code calls something, not that it calls the SDK.

What you’ll do

• Run a test that passes against a method that does not exist.
• Make the mock as strict as the real class with create_autospec, and watch the test fail for the right reason.
• Fix the call, then see what the old test was really checking.

What you’ll need: Python 3.12 or later (tested on 3.12, 3.13 and 3.14) and pytest.

The finished code, broken and fixed side by side: Code on GitHub

Set up the code

Make a folder and a virtual environment for the kata:

mkdir mock-kata && cd mock-kata
python3 -m venv .venv && . .venv/bin/activate && pip install pytest

Add a stand-in for the vendor SDK. It has the real surface your code depends on, and every method would make a network call:

# shopsdk.py
class ShopClient:
    def create_order(self, sku: str, qty: int) -> dict:
        raise RuntimeError("network call: not available in tests")

    def get_order(self, order_id: str) -> dict:
        raise RuntimeError("network call: not available in tests")

The code under test places an order. It calls submit_order, which ShopClient does not have:

# checkout.py
def place_order(client, sku: str, qty: int) -> str:
    order = client.submit_order(sku=sku, qty=qty)   # no such method on ShopClient
    return order["id"]

And the test, written the usual way:

# test_loose.py
from unittest.mock import MagicMock

from checkout import place_order


def test_place_order_returns_the_id():
    client = MagicMock()
    client.submit_order.return_value = {"id": "ord-1"}
    assert place_order(client, "sku-42", 1) == "ord-1"

Run the test that should not pass

python -m pytest -q test_loose.py

1 passed in 0.04s

The test is green, and place_order would raise AttributeError on its first real call. Nothing in the test knows what ShopClient looks like. MagicMock invented submit_order the moment the code asked for it.

Make the mock as strict as the SDK

create_autospec builds a mock from the real class. Attributes the class does not have raise AttributeError, and method calls are checked against the real signatures.

# test_strict.py
from unittest.mock import create_autospec

from checkout import place_order
from shopsdk import ShopClient


def test_place_order_returns_the_id():
    client = create_autospec(ShopClient, instance=True)
    client.create_order.return_value = {"id": "ord-1"}
    assert place_order(client, "sku-42", 1) == "ord-1"
    client.create_order.assert_called_once_with(sku="sku-42", qty=1)

python -m pytest -q test_strict.py

E               AttributeError: Mock object has no attribute 'submit_order'
1 failed in 0.04s

This is the failure production would have shown you, now in the test run.

Fix the call, then rerun both tests

Change submit_order to the method the SDK actually has:

# checkout.py
def place_order(client, sku: str, qty: int) -> str:
    order = client.create_order(sku=sku, qty=qty)
    return order["id"]

python -m pytest -q test_strict.py test_loose.py

The strict test passes. The loose test now fails:

E       AssertionError: assert <MagicMock name='mock.create_order().__getitem__()' id='4475029888'> == 'ord-1'
FAILED test_loose.py::test_place_order_returns_the_id - AssertionError: asser...
1 failed, 1 passed in 0.03s

The loose test passed on the broken code and fails on the fixed code. It was never checking the SDK contract. It was checking that the code called submit_order, which means it was written to match the bug. Delete it; the strict test replaces it.

What to keep

• Build mocks of third-party clients from the real class: create_autospec(Cls, instance=True). Add spec_set=True when you also want attribute assignment checked. Avoid plain MagicMock(spec_set=Cls) for this: it rejects unknown attributes but no longer checks call signatures.
• Assert the call, not only the return value: assert_called_once_with(...) is checked against the real signature.
• When a fix turns an old test red, read the old test before you repair it. It may be a record of the bug.

A mock that answers anything proves only that your code asked.