Introduction & What you’ll build
A green unit suite is easy to mistake for a hermetic one. In our own Python backend, auditing DNS traffic across a single test run turned up 45 non-loopback lookups from 11 tests — plus one on every parallel worker at startup because an imported module read a remote prompt registry at load time. When we hid Application Default Credentials during collection, 6,078 errors appeared at once because an import path initialised a Private CA client on load. Turning off local .env files exposed 67 more failures and a real production bug: an mTLS fallback crashed cert-less requests whenever no client certificate was configured.
Two layers are needed to catch these leaks reliably: (1) a socket guard that blocks non-local DNS and TCP connects and attributes blocked calls back to the owning test even when application code catches OSError, and (2) an environment snapshot per test so process-wide os.environ mutations never bleed from an earlier test into a later one.
In this 20-minute kata you will:
- Run a small Python test suite whose software under test swallows
OSErrorand stays silently green while reaching out over the network. - Switch on a socket guard patching
socket.getaddrinfo,socket.socket.connect, andsocket.socket.connect_ex— including an owner ledger and anOFFLINE_EXPECTEDexception for link-local metadata probes. - Verify why patching
getaddrinfoalone leaves literal-IP connects (such as async event loops) unguarded.
Setup
git clone https://github.com/mvpzone/mvplabs.git
cd mvplabs/kata/hermetic-suite
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
Notice there are no cloud dependencies or live endpoints required: everything runs offline against pytest.
Why silent fallbacks stay green
Look at the client under test in netclient.py and the three tests in test_suite.py:
# netclient.py
def fetch_config(host: str = "config.example.invalid", port: int = 443) -> str:
"""Resolve and connect to a remote config host; return 'default' on failure."""
try:
infos = socket.getaddrinfo(host, port, type=socket.SOCK_STREAM)
family, socktype, proto, _, sockaddr = infos[0]
with socket.socket(family, socktype, proto) as sock:
sock.settimeout(0.2)
sock.connect(sockaddr)
return "remote"
except OSError:
return "default"
# test_suite.py
def test_fetch_config_degrades():
# Software under test swallows OSError and returns 'default',
# so this assertion passes even though a real DNS lookup fired.
assert netclient.fetch_config("config.example.invalid") == "default"
def test_metadata_probe_degrades():
# SDKs commonly probe cloud link-local metadata on startup.
assert netclient.probe_metadata() is False
def test_loopback_resolves():
# Local test servers on loopback remain allowed.
assert netclient.resolve_loopback() == "127.0.0.1"
Run test_suite.py in the default (broken) mode:
pytest -q test_suite.py
... [100%]
3 passed in 0.07s
Every test passes. On a laptop without access to config.example.invalid, socket.getaddrinfo fails slowly against local DNS and fetch_config swallows OSError. In CI — or worse, on a machine holding live credentials — the same function silently hits real infrastructure.
Raising socket.gaierror inside a socket mock looks like a fix, until you realise production code is supposed to catch OSError and degrade gracefully. If you only block the call and never record that the call happened, test_fetch_config_degrades stays green.
Block DNS and literal-IP connects, then keep a ledger
To turn hidden outbound I/O into deterministic test failures, the guard in fixed/hermetic_net.py separates three cases:
- Loopback (
localhost,127.0.0.0/8,::1, Unix sockets) — allowed through untouched so in-process test servers work. - Expected offline probes (
metadata.google.internal,169.254.169.254) — blocked fast so off-cloud SDK startup probes fail without waiting for timeouts, and not recorded as test violations. - Everything else — blocked immediately and recorded in a thread-safe ledger under the current owner:
<collection>while modules are imported,item.nodeidduring a test call, or<after NODEID>after teardown if a lingering background thread reaches out between tests.
Notice we also patch socket.socket.connect and socket.socket.connect_ex, not just socket.getaddrinfo:
# fixed/hermetic_net.py
def is_local(host: object) -> bool:
"""Return True if host is localhost or a loopback IP."""
if not isinstance(host, str) or not host:
return False
clean = host.strip("[]").split("%", 1)[0].lower()
if clean == "localhost":
return True
try:
return ipaddress.ip_address(clean).is_loopback
except ValueError:
return False
# fixed/hermetic_net.py (socket patches)
def guarded_getaddrinfo(host, port, family=0, type=0, proto=0, flags=0):
if host is not None and not is_local(host):
key = str(host).strip("[]").split("%", 1)[0].lower()
if key not in OFFLINE_EXPECTED:
_LEDGER.record(str(host), "getaddrinfo")
raise socket.gaierror(
socket.EAI_NONAME,
f"hermetic test: network lookup to {host!r} blocked",
)
return orig_getaddrinfo(host, port, family, type, proto, flags)
def guarded_connect(self, address):
peer = _blocked_peer(self, address)
if peer is not None:
if peer.lower() not in OFFLINE_EXPECTED:
_LEDGER.record(peer, "connect")
raise OSError(
errno.ENETUNREACH,
f"hermetic test: connect to {peer!r} blocked",
)
return orig_connect(self, address)
Event loops such as asyncio skip getaddrinfo when handed an IP literal like 203.0.113.10. Patching connect and connect_ex closes that gap cleanly.
Extensions written in C that open their own sockets outside Python’s socket module — notably gRPC C-core — bypass Python monkeypatching. In practice they are caught indirectly when the preceding credential layer attempts an HTTP token exchange through Python, or isolated at OS network-namespace level in CI.
Wire the ledger into pytest hooks
In conftest.py, the guard installs at import time so module-level network calls during collection cannot sneak past before fixtures start:
# conftest.py
_net.install()
def pytest_collection_finish(session: pytest.Session) -> None:
leaked = _net.pop_non_test_violations()
if leaked:
summary = ", ".join(f"{owner} -> {host} ({op})" for owner, host, op in leaked)
raise pytest.UsageError(f"hermetic test: network used during collection: {summary}")
def pytest_runtest_setup(item: pytest.Item) -> None:
leaked = _net.pop_non_test_violations()
if leaked:
summary = ", ".join(f"{owner} -> {host} ({op})" for owner, host, op in leaked)
pytest.fail(f"hermetic test: stray network outside active test: {summary}")
_net.begin_owner(item.nodeid)
@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item: pytest.Item, call: pytest.CallInfo):
outcome = yield
report = outcome.get_result()
if report.when == "call":
violations = _net.pop_violations(item.nodeid)
if violations:
hosts = ", ".join(f"{host} ({op})" for host, op in violations)
report.outcome = "failed"
report.longrepr = (
f"hermetic test failed: unexpected outbound network in {item.nodeid}: {hosts}"
)
Re-run test_suite.py with the fixed guard enabled (KATA=fixed), then run the guard unit checks in test_guard.py:
KATA=fixed pytest -q test_suite.py
F.. [100%]
=================================== FAILURES ===================================
__________________________ test_fetch_config_degrades __________________________
hermetic test failed: unexpected outbound network in test_suite.py::test_fetch_config_degrades: config.example.invalid (getaddrinfo)
=========================== short test summary info ============================
FAILED test_suite.py::test_fetch_config_degrades - hermetic test failed: unex...
1 failed, 2 passed in 0.07s
KATA=fixed pytest -q test_guard.py
..... [100%]
5 passed in 0.07s
test_metadata_probe_degrades and test_loopback_resolves both pass; test_fetch_config_degrades fails loudly and points directly to config.example.invalid (getaddrinfo) even though fetch_config caught the OSError.
Pair it with an environment leak detector
Network sockets are only half of hermeticity. The second half is os.environ.
When we ran our suite with .env disabled, an autouse leak detector snapshotted dict(os.environ) before each test and compared keys and values afterwards. It caught our agent loader reading an agent directory’s .env directly into os.environ mid-run and never removing those keys — making subsequent tests pass only if that test ran first.
In our own conftest.py, an autouse fixture checks whether an allow-listed key changed or an unexpected key was left behind, and restoring/asserting at teardown makes order-dependent environment leaks fail at the test that introduced them.
Putting both guards in place cut our backend suite wall-clock time by roughly 40% (77 s → ~45 s) with zero non-local resolutions — and eliminated the slow implicit dependence on developer credentials and .env side effects.