Overview

A policy point is only as complete as the requests it actually sees. Complete mediation means every request is decided, and a decision you cannot show a record for is one you cannot prove was made. The way to prove it is a count: the requests the proxy served, the calls it made to the decision service, and the decisions that service recorded should all be the same number.

We met the gap building a decision service behind a load balancer’s ext_proc callout. The evidence we settled on was a three-way count per region, requests = callout streams = decision records, and the first version that made it agree was not the first version we wrote. A design that decides when the body arrives has a blind spot the size of every request without one: GET, HEAD, DELETE, an empty POST. Envoy calls the service for them. The service just never gets to the phase where it decides.

What you’ll do

  • Run Envoy with an ext_proc decision service whose policy is one line: deny /admin.
  • Watch GET /admin get a 200 while POST /admin gets a 403.
  • Count requests, callout streams and decision records, and join them to name every request with no record.
  • Fix the service so it decides at the last request phase each stream will see, and records exactly once, even when the client hangs up.

What you’ll need: Docker (or colima with nerdctl), Python 3.12 or later (tested on 3.13), and three terminals. It builds on Envoy + a stub ext_proc that logs every request, but stands alone.

Get the code

Clone the lab repo under your home directory (colima shares only ~ with its VM) and install the pinned gRPC runtime and Envoy’s generated protos:

git clone https://github.com/mvpzone/mvplabs.git
cd mvplabs/codelab/one-record-per-request
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
docker pull envoyproxy/envoy:v1.39.4

The backend answers GET, HEAD, POST and DELETE on any path with a 200. It is not the subject; the decision in front of it is. Start it in the first terminal and leave it running:

python backend.py
backend listening on :8081

Read the config

envoy.yaml adds three things to the first lab’s config. The body is sent to the processor, buffered, because the decision wants to see the payload. Every request is written to a JSON access log, keyed by the x-request-id Envoy generates. And the admin port is open, so Envoy can tell you how many ext_proc streams it opened.

# envoy.yaml (the parts that matter)
admin:
  address:
    socket_address: { address: 0.0.0.0, port_value: 9901 }   # /stats: ext_proc counts its own streams
...
          access_log:                # count 1: one line per request Envoy served
          - name: envoy.access_loggers.stdout
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
              log_format:
                json_format:
                  rid: "%REQ(X-REQUEST-ID)%"
                  method: "%REQ(:METHOD)%"
                  path: "%REQ(:PATH)%"
                  status: "%RESPONSE_CODE%"
...
              failure_mode_allow: false
              processing_mode:
                request_header_mode: SEND
                response_header_mode: SKIP
                request_body_mode: BUFFERED          # the decision wants to see the payload
                response_body_mode: NONE

The policy and the record live in decide.py, shared by both processors, so the only thing that differs between the broken and the fixed service is where they are called:

# decide.py
def decide(method: str, path: str) -> str:
    """The whole policy: nobody reaches /admin through this door."""
    return "deny" if path.startswith("/admin") else "allow"


def write_record(rid: str, method: str, path: str, verdict: str, phase: str) -> None:
    """Count 3: one line per decision. Append-only; the reconciler reads it."""
    ...  # appends {"rid", "method", "path", "verdict", "phase", "at"} to records.jsonl

Decide on the body

The first service does what feels natural. It reads the request’s identity from the headers, then decides when the body arrives, because that is where the interesting evidence is:

# broken/extproc.py
class DecideOnTheBody(pb_grpc.ExternalProcessorServicer):
    """Decides when the payload arrives: that is where the interesting evidence is."""

    def Process(self, requests, context):
        rid = method = path = ""
        for req in requests:
            match req.WhichOneof("request"):
                case "request_headers":
                    h = headers_of(req.request_headers)
                    rid, method, path = h.get("x-request-id", ""), h[":method"], h[":path"]
                    yield pb.ProcessingResponse(request_headers=pb.HeadersResponse())
                case "request_body":
                    verdict = decide(method, path)
                    write_record(rid, method, path, verdict, "request_body")
                    if verdict == "deny":
                        yield pb.ProcessingResponse(immediate_response=deny())
                        return
                    yield pb.ProcessingResponse(request_body=pb.BodyResponse())

Start it in the second terminal:

python broken/extproc.py

Start Envoy in the third. The --file-flush-interval-msec flag makes Envoy write its access log within 200 ms instead of the default 10 seconds, so the count is complete when you read it:

docker run -d --name l5-envoy -p 10000:10000 -p 9901:9901 \
  --add-host=host.docker.internal:host-gateway \
  -v "$PWD/envoy.yaml:/etc/envoy/envoy.yaml:ro" envoyproxy/envoy:v1.39.4 \
  -c /etc/envoy/envoy.yaml --file-flush-interval-msec 200

On colima, run it as colima nerdctl -- run ... with the same flags, and use --add-host=host.docker.internal:192.168.5.2 (host-gateway resolves to the VM, not your Mac).

traffic.sh sends ten requests. Four of them carry no body:

./traffic.sh
GET    /              -> 200
GET    /orders        -> 200
HEAD   /orders        -> 200
POST   /orders        -> 200
POST   /orders (empty) -> 200
DELETE /orders/7      -> 200
GET    /admin         -> 200
POST   /admin         -> 403
POST   /orders        -> 200
GET    /admin/users   -> 200

The policy denies /admin. POST /admin was denied. GET /admin and GET /admin/users were served. Nothing failed, nothing logged an error, and a test that only posted to /admin would pass. The processor’s own output shows why:

ext_proc (broken) listening on :50051
record: allow POST /orders (request_body)
record: deny  POST /admin (request_body)
record: allow POST /orders (request_body)

Three decisions for ten requests. A request with no body never produces a request_body message, so for seven of the ten requests the code that decides never ran.

Count three things

You can see the gap above because there are ten requests and you sent them. In production you need a check that finds it without knowing in advance which requests to look for. That check is three counts that must agree:

Count Source Who writes it
Requests served Envoy’s access log the proxy
ext_proc streams http.front.ext_proc.streams_started on the admin port the proxy
Decision records records.jsonl the decision service

Two of the three come from the proxy, and only one from the service under test. The service cannot vouch for its own completeness; the proxy can count what it handed over. reconcile.py reads all three and joins the access log to the records on x-request-id:

docker logs l5-envoy 2>/dev/null | python reconcile.py
requests served (access log) : 10
ext_proc streams (Envoy stats): 10
decision records             : 3
  no record: GET    /              -> 200
  no record: GET    /orders        -> 200
  no record: HEAD   /orders        -> 200
  no record: POST   /orders        -> 200
  no record: DELETE /orders/7      -> 200
  no record: GET    /admin         -> 200
  no record: GET    /admin/users   -> 200
NOT RECONCILED

Read the three numbers in order. Requests = streams says Envoy called the service for every request: the filter is wired and nothing bypassed it at the proxy. Streams ≠ records says the gap is inside the service. The join names every request that went through without a decision, and the empty POST (the fourth no record line) shows the gap is about the body, not the method.

Stop the processor (Ctrl-C), and reset the counts before the next run: Envoy’s counters live as long as the container, and the records file is append-only.

docker rm -f l5-envoy
rm records.jsonl

Decide at the end of the request

Every ext_proc stream tells you when the request is complete. The headers message carries end_of_stream: true means no body will follow, so the headers are the last request phase this stream will ever see. The fixed service decides there when it has to, waits for the body when one is coming, and records in exactly one place:

# fixed/extproc.py
class OneRecordPerStream(pb_grpc.ExternalProcessorServicer):
    """Decides at the LAST request phase this stream will see, and records exactly once per stream."""

    def Process(self, requests, context):
        rid = method = path = ""
        recorded = False
        try:
            for req in requests:
                phase = req.WhichOneof("request")
                if phase == "request_headers":
                    h = headers_of(req.request_headers)
                    rid, method, path = h.get("x-request-id", ""), h[":method"], h[":path"]
                    if not req.request_headers.end_of_stream:
                        # A body is coming: wait for it, it is part of the evidence.
                        yield pb.ProcessingResponse(request_headers=pb.HeadersResponse())
                        continue
                    # No body will ever come (GET, HEAD, an empty POST). Decide here or never.
                elif phase != "request_body":
                    continue
                verdict = decide(method, path)
                write_record(rid, method, path, verdict, phase)
                recorded = True
                if verdict == "deny":
                    yield pb.ProcessingResponse(immediate_response=deny())
                    return
                if phase == "request_headers":
                    yield pb.ProcessingResponse(request_headers=pb.HeadersResponse())
                else:
                    yield pb.ProcessingResponse(request_body=pb.BodyResponse())
        finally:
            if not recorded:
                # The stream ended before a decision (the client hung up mid-body, Envoy timed us out).
                # Still one record: an attempt with no decision is evidence too.
                write_record(rid, method, path, "none", "stream_closed")

Start it, start Envoy again with the same docker run, and send the same traffic:

python fixed/extproc.py
./traffic.sh
GET    /              -> 200
GET    /orders        -> 200
HEAD   /orders        -> 200
POST   /orders        -> 200
POST   /orders (empty) -> 200
DELETE /orders/7      -> 200
GET    /admin         -> 403
POST   /admin         -> 403
POST   /orders        -> 200
GET    /admin/users   -> 403

Every /admin request is denied now. The processor shows where each decision was made:

record: allow GET / (request_headers)
record: allow GET /orders (request_headers)
record: allow HEAD /orders (request_headers)
record: allow POST /orders (request_body)
record: allow POST /orders (request_headers)
record: allow DELETE /orders/7 (request_headers)
record: deny  GET /admin (request_headers)
record: deny  POST /admin (request_body)
record: allow POST /orders (request_body)
record: deny  GET /admin/users (request_headers)

The empty POST was decided on its headers: Envoy set end_of_stream because its content-length was 0. The method does not tell you whether a body is coming; end_of_stream does.

docker logs l5-envoy 2>/dev/null | python reconcile.py
requests served (access log) : 10
ext_proc streams (Envoy stats): 10
decision records             : 10
RECONCILED

The client that hangs up

One record per decided request is not yet one record per request. A client can open a request with a body and never finish it. The stream ends with no body phase and no decision, and the fixed service’s finally block is there for it. hangup.py promises 100 bytes, sends 10, and disconnects:

python hangup.py
docker logs l5-envoy 2>/dev/null | python reconcile.py
sent 10 of 100 bytes, hung up
requests served (access log) : 11
ext_proc streams (Envoy stats): 11
decision records             : 11
RECONCILED

Envoy logged the request with status 0 (no response was sent), counted the stream, and the service wrote:

record: none  POST /orders (stream_closed)

The record says no decision was made, which is the truth. An attempt that never reached a decision is still an attempt, and leaving it out would make the count agree by making the record set smaller than the traffic it describes.

Clean up:

docker rm -f l5-envoy

Stop the processor and the backend with Ctrl-C.

What to keep

  • Decide at the last request phase the stream will see, not at the phase where the best evidence lives. For a request with no body, that is the headers, and end_of_stream is how you know.
  • Record in one place, and make that place reachable on every path out of the stream, including the ones where the client leaves first. A stream that ends undecided gets a record that says so.
  • Prove completeness with counts the service does not write. Requests = callout streams = decision records, joined on a request id, finds the gap without knowing which requests to look for, and tells you which side of the callout it is on.
  • Test the requests your policy does not mention. The broken service passed every test that sent a body to /admin.

The policy was one line and correct. It just never ran for the requests that needed it.