Overview

A decision point that sits in the request path has two jobs before it has any policy: see every request, and leave a record of each one. Envoy’s external processing filter, ext_proc, gives you the seat. Envoy opens a gRPC stream to your processor for each HTTP request, sends the request headers, and waits for an answer before it forwards anything.

This lab builds the smallest processor that does both jobs, then gives it one rule.

What you’ll do

• Run Envoy in a container with the ext_proc filter in front of a stand-in backend.
• Write a stub processor in Python that logs one line per request and lets it through.
• Add a rule, run it in observe mode, then in enforce mode, and compare the records.
• Stop the processor and see what Envoy does without it, with and without fail-open.

What you’ll need: Docker (or colima with nerdctl, see the note below), Python 3.12 or later (tested on 3.13 and 3.14), curl, and two terminals.

The “why” behind this lab is in two posts: What May Sit in the Hot Path and A Record for Every Attempt. This is the hands-on part.

Set up

Make a folder, a virtual environment, and install the gRPC runtime and Envoy’s generated protos. xds-protos ships the envoy.service.ext_proc.v3 stubs ready to import, so there is no protoc step. The three versions are pinned together: the generated code refuses to load on an older runtime.

mkdir extproc-lab && cd extproc-lab
python3 -m venv .venv && . .venv/bin/activate
pip install grpcio==1.84.0 xds-protos==1.84.0 protobuf==7.36.2

Pull the Envoy image:

docker pull envoyproxy/envoy:v1.39.4

Write the stub processor

The processor implements one RPC, Process. It is a bidirectional stream: one stream per HTTP request, one message per phase that Envoy is configured to send. Here Envoy sends only the request headers, so each stream carries one message.

# extproc.py
import json
import time
from concurrent import futures

import grpc
from envoy.service.ext_proc.v3 import external_processor_pb2 as pb
from envoy.service.ext_proc.v3 import external_processor_pb2_grpc as pb_grpc


def headers_of(msg) -> dict[str, str]:
    # Envoy sends header values as bytes in raw_value; older builds used value.
    return {h.key: h.value or h.raw_value.decode() for h in msg.headers.headers}


class LogEveryRequest(pb_grpc.ExternalProcessorServicer):
    def Process(self, requests, context):
        # One gRPC stream per HTTP request; one message per phase Envoy sends.
        for req in requests:
            if req.HasField("request_headers"):
                h = headers_of(req.request_headers)
                print(json.dumps({
                    "ts": round(time.time(), 3),
                    "request_id": h.get("x-request-id"),
                    "method": h.get(":method"),
                    "authority": h.get(":authority"),
                    "path": h.get(":path"),
                }), flush=True)
                # "Continue" is an explicit, empty HeadersResponse. A reply with
                # nothing set is malformed, not a no-op.
                yield pb.ProcessingResponse(request_headers=pb.HeadersResponse())


def main() -> None:
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=8))
    pb_grpc.add_ExternalProcessorServicer_to_server(LogEveryRequest(), server)
    server.add_insecure_port("0.0.0.0:50051")
    server.start()
    print("ext_proc listening on :50051", flush=True)
    server.wait_for_termination()


if __name__ == "__main__":
    main()

The repo keeps both versions of the processor, in v1/ and v2/.

Two details carry the lab. The log line comes from the headers phase, which every request has, including GET, HEAD and OPTIONS with no body. And “continue” is a concrete response on the matching field, never an empty ProcessingResponse.

Start it in the first terminal:

python extproc.py

ext_proc listening on :50051

It listens on all interfaces so that the Envoy container can reach it. Stop it when you finish the lab.

Put Envoy in front

The config has two listeners. front on port 10000 is what clients call; it runs ext_proc, then routes to the backend. backend is a stand-in app inside the same Envoy that answers every request with one line. The extproc cluster points at your laptop and speaks HTTP/2, which gRPC needs.

# envoy.yaml
static_resources:
  listeners:
  - name: front                      # what clients call
    address:
      socket_address: { address: 0.0.0.0, port_value: 10000 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: front
          route_config:
            virtual_hosts:
            - name: app
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route: { cluster: backend }
          http_filters:
          - name: envoy.filters.http.ext_proc
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
              grpc_service:
                envoy_grpc: { cluster_name: extproc }
                timeout: 0.5s
              failure_mode_allow: false
              processing_mode:
                request_header_mode: SEND
                response_header_mode: SKIP
                request_body_mode: NONE
                response_body_mode: NONE
                request_trailer_mode: SKIP
                response_trailer_mode: SKIP
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  - name: backend                    # a stand-in app, inside the same Envoy
    address:
      socket_address: { address: 127.0.0.1, port_value: 8081 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: backend
          route_config:
            virtual_hosts:
            - name: backend
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                direct_response:
                  status: 200
                  body: { inline_string: "hello from the backend\n" }
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: backend
    type: STATIC
    load_assignment:
      cluster_name: backend
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: 127.0.0.1, port_value: 8081 }
  - name: extproc                    # the stub, running on your laptop
    type: STRICT_DNS
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config: { http2_protocol_options: {} }
    load_assignment:
      cluster_name: extproc
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: host.docker.internal, port_value: 50051 }

Check the config, then start Envoy in the second terminal:

docker run --rm -v "$PWD/envoy.yaml:/etc/envoy/envoy.yaml:ro" envoyproxy/envoy:v1.39.4 \
  --mode validate -c /etc/envoy/envoy.yaml
docker run -d --name l1-envoy -p 10000:10000 \
  --add-host=host.docker.internal:host-gateway \
  -v "$PWD/envoy.yaml:/etc/envoy/envoy.yaml:ro" envoyproxy/envoy:v1.39.4

configuration '/etc/envoy/envoy.yaml' OK

host-gateway resolves to your machine on Docker Desktop and on Docker Engine for Linux. On colima it resolves to the VM’s gateway (192.168.5.1), not your Mac: use --add-host=host.docker.internal:192.168.5.2 instead. With colima’s containerd runtime there is no docker command: run each one here as colima nerdctl -- , with the same flags (a standalone nerdctl looks for a Lima VM named default and will not find colima’s). colima shares only your home directory with the VM, so keep the lab folder under ~ or the envoy.yaml mount fails with “not a directory”.

Send requests, read the log

Send four requests: a GET, a POST with a body, a HEAD and an OPTIONS.

curl -s localhost:10000/hello
curl -s -o /dev/null -w "%{http_code}\n" -X POST -d '{"a":1}' localhost:10000/orders
curl -s -I localhost:10000/health | head -1
curl -s -o /dev/null -w "%{http_code}\n" -X OPTIONS localhost:10000/orders

hello from the backend
200
HTTP/1.1 200 OK
200

The first terminal now has one line per request (your ids and times will differ):

{"ts": 1792084814.67, "request_id": "ebfedba2-8dd1-4692-8187-f8b75249f8c0", "method": "GET", "authority": "localhost:10000", "path": "/hello"}
{"ts": 1792084814.687, "request_id": "d5a4ccd7-768b-45b3-bfa1-2c1c7a2248ac", "method": "POST", "authority": "localhost:10000", "path": "/orders"}
{"ts": 1792084814.702, "request_id": "3080d6b0-bf13-4bcf-8527-9bd3b10dbdc3", "method": "HEAD", "authority": "localhost:10000", "path": "/health"}
{"ts": 1792084814.716, "request_id": "087e64b7-d824-4ff4-9f25-685264921029", "method": "OPTIONS", "authority": "localhost:10000", "path": "/orders"}

Four requests, four lines. The HEAD and the OPTIONS have no body, so a processor that only looked at bodies would never have seen them. The x-request-id was generated by Envoy before the filter ran, so it is the key that joins this line to anything else Envoy or the backend writes about the same request.

Observe, then enforce

Now give the processor a rule, nothing under /admin, and a mode. The record keeps two fields apart: verdict is what the processor decided, effect is what happened to the request. In observe mode they differ. Replace extproc.py:

# extproc.py
import json
import os
import time
from concurrent import futures

import grpc
from envoy.service.ext_proc.v3 import external_processor_pb2 as pb
from envoy.service.ext_proc.v3 import external_processor_pb2_grpc as pb_grpc
from envoy.type.v3 import http_status_pb2

MODE = os.environ.get("MODE", "observe")      # observe | enforce


def headers_of(msg) -> dict[str, str]:
    return {h.key: h.value or h.raw_value.decode() for h in msg.headers.headers}


def decide(h: dict[str, str]) -> tuple[str, str]:
    """The whole policy: nothing under /admin. Returns (verdict, cause)."""
    if h.get(":path", "").startswith("/admin"):
        return "deny", "path_not_allowed"
    return "allow", "default_allow"


class LogEveryRequest(pb_grpc.ExternalProcessorServicer):
    def Process(self, requests, context):
        for req in requests:
            if req.HasField("request_headers"):
                h = headers_of(req.request_headers)
                verdict, cause = decide(h)
                effect = "deny" if verdict == "deny" and MODE == "enforce" else "allow"
                print(json.dumps({
                    "ts": round(time.time(), 3),
                    "request_id": h.get("x-request-id"),
                    "method": h.get(":method"),
                    "path": h.get(":path"),
                    "verdict": verdict,       # what the decider decided
                    "effect": effect,         # what happened to the request
                    "mode": MODE,
                    "cause": cause,
                }), flush=True)
                if effect == "deny":
                    yield pb.ProcessingResponse(immediate_response=pb.ImmediateResponse(
                        status=http_status_pb2.HttpStatus(code=http_status_pb2.Forbidden),
                        body=b"denied by policy\n",
                    ))
                else:
                    yield pb.ProcessingResponse(request_headers=pb.HeadersResponse())


def main() -> None:
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=8))
    pb_grpc.add_ExternalProcessorServicer_to_server(LogEveryRequest(), server)
    server.add_insecure_port("0.0.0.0:50051")
    server.start()
    print(f"ext_proc listening on :50051 (mode={MODE})", flush=True)
    server.wait_for_termination()


if __name__ == "__main__":
    main()

Stop the old processor (Ctrl-C) and start the new one. It starts in observe mode:

python extproc.py

In the second terminal:

curl -s -w "%{http_code}\n" localhost:10000/orders
curl -s -w "%{http_code}\n" localhost:10000/admin/users

hello from the backend
200
hello from the backend
200

Both requests reached the backend. The records say something different:

ext_proc listening on :50051 (mode=observe)
{"ts": 1792084829.997, "request_id": "ec1ca22a-1771-4f82-a38f-768b3ec39b55", "method": "GET", "path": "/orders", "verdict": "allow", "effect": "allow", "mode": "observe", "cause": "default_allow"}
{"ts": 1792084830.015, "request_id": "8d26df82-efd2-43d6-b2a2-dad67369b86b", "method": "GET", "path": "/admin/users", "verdict": "deny", "effect": "allow", "mode": "observe", "cause": "path_not_allowed"}

verdict: deny, effect: allow. That line is the evidence you want before turning enforcement on: it shows what the rule would have refused, on real traffic, while refusing nothing. A log with one outcome field would have had to choose which of the two to write.

Now restart the processor in enforce mode (Ctrl-C first) and send the same two requests:

MODE=enforce python extproc.py

curl -s -w "%{http_code}\n" localhost:10000/orders
curl -s -i localhost:10000/admin/users

hello from the backend
200
HTTP/1.1 403 Forbidden
content-length: 17
content-type: text/plain
date: Thu, 15 Oct 2026 17:20:31 GMT
server: envoy

denied by policy

ext_proc listening on :50051 (mode=enforce)
{"ts": 1792084832.074, "request_id": "d9a61102-b5da-43d9-a822-972b717e8282", "method": "GET", "path": "/orders", "verdict": "allow", "effect": "allow", "mode": "enforce", "cause": "default_allow"}
{"ts": 1792084832.092, "request_id": "01bcd2c1-47d2-48c5-b8c0-c104c2299355", "method": "GET", "path": "/admin/users", "verdict": "deny", "effect": "deny", "mode": "enforce", "cause": "path_not_allowed"}

The immediate_response made Envoy answer the client itself. The backend never saw the request, and the record of the refusal is the only trace the refusal happened.

Take the decider away

Stop the processor (Ctrl-C) and send a request:

curl -s -i localhost:10000/orders
docker logs l1-envoy 2>&1 | tail -1

HTTP/1.1 500 Internal Server Error
date: Thu, 15 Oct 2026 17:20:37 GMT
server: envoy
content-length: 0

[2026-10-15 17:20:37.828][14][warning][ext_proc] [source/extensions/filters/http/ext_proc/ext_proc.cc:1977] [Tags: "ConnectionId":"24","StreamId":"13435984093411697472"] Received gRPC error on stream: 14, message upstream connect error or disconnect/reset before headers. reset reason: remote connection failure{upstream_reset_before_response_started{remote_connection_failure|delayed_connect_error:_Connection_refused}}

With failure_mode_allow: false, no decider means no traffic. That is fail-closed, and it is why a decider that fails closed has to stay warm. Now set failure_mode_allow: true in envoy.yaml and recreate the container, with the processor still stopped:

docker rm -f l1-envoy
docker run -d --name l1-envoy -p 10000:10000 \
  --add-host=host.docker.internal:host-gateway \
  -v "$PWD/envoy.yaml:/etc/envoy/envoy.yaml:ro" envoyproxy/envoy:v1.39.4
curl -s -w "%{http_code}\n" localhost:10000/admin/users

hello from the backend
200

The request the rule refuses reached the backend, and the processor wrote nothing, because it never saw it. A fail-open pass is invisible to the decider by construction. If you need to count those, the record has to come from the component that let the request through, here Envoy’s own access log, and you need a way to reconcile the two. That reconciliation is a later lab.

Clean up:

docker rm -f l1-envoy

What to keep

• Record at the headers phase. Every request has one; bodyless requests are requests too.
• “Continue” is an explicit HeadersResponse() on the matching field. An empty ProcessingResponse is not a no-op. For bodies the rules are stricter: in one streaming mode an empty body reply is not a continue at all. That is the next lab.
• Keep verdict and effect apart. Observe mode is the case where they differ, and that difference is the evidence for turning enforcement on.
• Decide the failure mode on purpose. Fail-closed stops traffic when the decider is down; fail-open lets it through with no record from the decider.

The decider can only record what it was asked about.