Overview
A policy point is only as complete as the requests it actually sees. Complete mediation means every request is decided, and a decision you cannot show a record for is one you cannot prove was made. The way to prove it is a count: the requests the proxy served, the calls it made to the decision service, and the decisions that service recorded should all be the same number.
We met the gap building a decision service behind a load balancer’s ext_proc callout. The evidence we settled on
was a three-way count per region, requests = callout streams = decision records, and the first version that made it
agree was not the first version we wrote. A design that decides when the body arrives has a blind spot the size of
every request without one: GET, HEAD, DELETE, an empty POST. Envoy calls the service for them. The service
just never gets to the phase where it decides.
What you’ll do
- Run Envoy with an
ext_procdecision service whose policy is one line: deny/admin. - Watch
GET /adminget a 200 whilePOST /admingets a 403. - Count requests, callout streams and decision records, and join them to name every request with no record.
- Fix the service so it decides at the last request phase each stream will see, and records exactly once, even when the client hangs up.
What you’ll need: Docker (or colima with nerdctl), Python 3.12 or later (tested on 3.13), and three terminals.
It builds on Envoy + a stub ext_proc that logs every request,
but stands alone.
Get the code
Clone the lab repo under your home directory (colima shares only ~ with its VM) and install the pinned gRPC runtime
and Envoy’s generated protos:
git clone https://github.com/mvpzone/mvplabs.git
cd mvplabs/codelab/one-record-per-request
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
docker pull envoyproxy/envoy:v1.39.4
The backend answers GET, HEAD, POST and DELETE on any path with a 200. It is not the subject; the decision in
front of it is. Start it in the first terminal and leave it running:
python backend.py
backend listening on :8081
Read the config
envoy.yaml adds three things to the first lab’s config. The body is sent to the processor, buffered, because the
decision wants to see the payload. Every request is written to a JSON access log, keyed by the x-request-id Envoy
generates. And the admin port is open, so Envoy can tell you how many ext_proc streams it opened.
# envoy.yaml (the parts that matter)
admin:
address:
socket_address: { address: 0.0.0.0, port_value: 9901 } # /stats: ext_proc counts its own streams
...
access_log: # count 1: one line per request Envoy served
- name: envoy.access_loggers.stdout
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
log_format:
json_format:
rid: "%REQ(X-REQUEST-ID)%"
method: "%REQ(:METHOD)%"
path: "%REQ(:PATH)%"
status: "%RESPONSE_CODE%"
...
failure_mode_allow: false
processing_mode:
request_header_mode: SEND
response_header_mode: SKIP
request_body_mode: BUFFERED # the decision wants to see the payload
response_body_mode: NONE
The policy and the record live in decide.py, shared by both processors, so the only thing that differs between the
broken and the fixed service is where they are called:
# decide.py
def decide(method: str, path: str) -> str:
"""The whole policy: nobody reaches /admin through this door."""
return "deny" if path.startswith("/admin") else "allow"
def write_record(rid: str, method: str, path: str, verdict: str, phase: str) -> None:
"""Count 3: one line per decision. Append-only; the reconciler reads it."""
... # appends {"rid", "method", "path", "verdict", "phase", "at"} to records.jsonl
Decide on the body
The first service does what feels natural. It reads the request’s identity from the headers, then decides when the body arrives, because that is where the interesting evidence is:
# broken/extproc.py
class DecideOnTheBody(pb_grpc.ExternalProcessorServicer):
"""Decides when the payload arrives: that is where the interesting evidence is."""
def Process(self, requests, context):
rid = method = path = ""
for req in requests:
match req.WhichOneof("request"):
case "request_headers":
h = headers_of(req.request_headers)
rid, method, path = h.get("x-request-id", ""), h[":method"], h[":path"]
yield pb.ProcessingResponse(request_headers=pb.HeadersResponse())
case "request_body":
verdict = decide(method, path)
write_record(rid, method, path, verdict, "request_body")
if verdict == "deny":
yield pb.ProcessingResponse(immediate_response=deny())
return
yield pb.ProcessingResponse(request_body=pb.BodyResponse())
Start it in the second terminal:
python broken/extproc.py
Start Envoy in the third. The --file-flush-interval-msec flag makes Envoy write its access log within 200 ms
instead of the default 10 seconds, so the count is complete when you read it:
docker run -d --name l5-envoy -p 10000:10000 -p 9901:9901 \
--add-host=host.docker.internal:host-gateway \
-v "$PWD/envoy.yaml:/etc/envoy/envoy.yaml:ro" envoyproxy/envoy:v1.39.4 \
-c /etc/envoy/envoy.yaml --file-flush-interval-msec 200
On colima, run it as colima nerdctl -- run ... with the same flags, and use
--add-host=host.docker.internal:192.168.5.2 (host-gateway resolves to the VM, not your Mac).
traffic.sh sends ten requests. Four of them carry no body:
./traffic.sh
GET / -> 200
GET /orders -> 200
HEAD /orders -> 200
POST /orders -> 200
POST /orders (empty) -> 200
DELETE /orders/7 -> 200
GET /admin -> 200
POST /admin -> 403
POST /orders -> 200
GET /admin/users -> 200
The policy denies /admin. POST /admin was denied. GET /admin and GET /admin/users were served. Nothing failed,
nothing logged an error, and a test that only posted to /admin would pass. The processor’s own output shows why:
ext_proc (broken) listening on :50051
record: allow POST /orders (request_body)
record: deny POST /admin (request_body)
record: allow POST /orders (request_body)
Three decisions for ten requests. A request with no body never produces a request_body message, so for seven of the
ten requests the code that decides never ran.
Count three things
You can see the gap above because there are ten requests and you sent them. In production you need a check that finds it without knowing in advance which requests to look for. That check is three counts that must agree:
| Count | Source | Who writes it |
|---|---|---|
| Requests served | Envoy’s access log | the proxy |
ext_proc streams |
http.front.ext_proc.streams_started on the admin port |
the proxy |
| Decision records | records.jsonl |
the decision service |
Two of the three come from the proxy, and only one from the service under test. The service cannot vouch for its own
completeness; the proxy can count what it handed over. reconcile.py reads all three and joins the access log to the
records on x-request-id:
docker logs l5-envoy 2>/dev/null | python reconcile.py
requests served (access log) : 10
ext_proc streams (Envoy stats): 10
decision records : 3
no record: GET / -> 200
no record: GET /orders -> 200
no record: HEAD /orders -> 200
no record: POST /orders -> 200
no record: DELETE /orders/7 -> 200
no record: GET /admin -> 200
no record: GET /admin/users -> 200
NOT RECONCILED
Read the three numbers in order. Requests = streams says Envoy called the service for every request: the filter is
wired and nothing bypassed it at the proxy. Streams ≠ records says the gap is inside the service. The join names every
request that went through without a decision, and the empty POST (the fourth no record line) shows the gap is
about the body, not the method.
Stop the processor (Ctrl-C), and reset the counts before the next run: Envoy’s counters live as long as the container, and the records file is append-only.
docker rm -f l5-envoy
rm records.jsonl
Decide at the end of the request
Every ext_proc stream tells you when the request is complete. The headers message carries end_of_stream: true
means no body will follow, so the headers are the last request phase this stream will ever see. The fixed service
decides there when it has to, waits for the body when one is coming, and records in exactly one place:
# fixed/extproc.py
class OneRecordPerStream(pb_grpc.ExternalProcessorServicer):
"""Decides at the LAST request phase this stream will see, and records exactly once per stream."""
def Process(self, requests, context):
rid = method = path = ""
recorded = False
try:
for req in requests:
phase = req.WhichOneof("request")
if phase == "request_headers":
h = headers_of(req.request_headers)
rid, method, path = h.get("x-request-id", ""), h[":method"], h[":path"]
if not req.request_headers.end_of_stream:
# A body is coming: wait for it, it is part of the evidence.
yield pb.ProcessingResponse(request_headers=pb.HeadersResponse())
continue
# No body will ever come (GET, HEAD, an empty POST). Decide here or never.
elif phase != "request_body":
continue
verdict = decide(method, path)
write_record(rid, method, path, verdict, phase)
recorded = True
if verdict == "deny":
yield pb.ProcessingResponse(immediate_response=deny())
return
if phase == "request_headers":
yield pb.ProcessingResponse(request_headers=pb.HeadersResponse())
else:
yield pb.ProcessingResponse(request_body=pb.BodyResponse())
finally:
if not recorded:
# The stream ended before a decision (the client hung up mid-body, Envoy timed us out).
# Still one record: an attempt with no decision is evidence too.
write_record(rid, method, path, "none", "stream_closed")
Start it, start Envoy again with the same docker run, and send the same traffic:
python fixed/extproc.py
./traffic.sh
GET / -> 200
GET /orders -> 200
HEAD /orders -> 200
POST /orders -> 200
POST /orders (empty) -> 200
DELETE /orders/7 -> 200
GET /admin -> 403
POST /admin -> 403
POST /orders -> 200
GET /admin/users -> 403
Every /admin request is denied now. The processor shows where each decision was made:
record: allow GET / (request_headers)
record: allow GET /orders (request_headers)
record: allow HEAD /orders (request_headers)
record: allow POST /orders (request_body)
record: allow POST /orders (request_headers)
record: allow DELETE /orders/7 (request_headers)
record: deny GET /admin (request_headers)
record: deny POST /admin (request_body)
record: allow POST /orders (request_body)
record: deny GET /admin/users (request_headers)
The empty POST was decided on its headers: Envoy set end_of_stream because its content-length was 0. The
method does not tell you whether a body is coming; end_of_stream does.
docker logs l5-envoy 2>/dev/null | python reconcile.py
requests served (access log) : 10
ext_proc streams (Envoy stats): 10
decision records : 10
RECONCILED
The client that hangs up
One record per decided request is not yet one record per request. A client can open a request with a body and never
finish it. The stream ends with no body phase and no decision, and the fixed service’s finally block is there for
it. hangup.py promises 100 bytes, sends 10, and disconnects:
python hangup.py
docker logs l5-envoy 2>/dev/null | python reconcile.py
sent 10 of 100 bytes, hung up
requests served (access log) : 11
ext_proc streams (Envoy stats): 11
decision records : 11
RECONCILED
Envoy logged the request with status 0 (no response was sent), counted the stream, and the service wrote:
record: none POST /orders (stream_closed)
The record says no decision was made, which is the truth. An attempt that never reached a decision is still an attempt, and leaving it out would make the count agree by making the record set smaller than the traffic it describes.
Clean up:
docker rm -f l5-envoy
Stop the processor and the backend with Ctrl-C.
What to keep
- Decide at the last request phase the stream will see, not at the phase where the best evidence lives. For a request
with no body, that is the headers, and
end_of_streamis how you know. - Record in one place, and make that place reachable on every path out of the stream, including the ones where the client leaves first. A stream that ends undecided gets a record that says so.
- Prove completeness with counts the service does not write. Requests = callout streams = decision records, joined on a request id, finds the gap without knowing which requests to look for, and tells you which side of the callout it is on.
- Test the requests your policy does not mention. The broken service passed every test that sent a body to
/admin.
The policy was one line and correct. It just never ran for the requests that needed it.