Overview

A decision point that reads request bodies has to hold them without stalling the request. Envoy’s ext_proc filter has a mode for that, FULL_DUPLEX_STREAMED: Envoy streams each body chunk to your processor as it arrives, and the processor streams it back. In that mode the processor holds the body. Whatever it hands back is the body.

Our first processor in this mode was a stub that answered every phase and changed nothing. Behind a managed load balancer, its “changed nothing” reply was taken as “this chunk is now empty”. For three hours every request body through that path arrived empty, and every status code stayed green. This lab reproduces that on a laptop, fixes it, and adds the check that would have caught it.

What you’ll do

  • Run Envoy with request and response bodies in FULL_DUPLEX_STREAMED mode, in front of a small backend.
  • Run the stub that changes nothing, in this mode and in buffered mode.
  • Reproduce the quiet failure: bodies gone, statuses 200.
  • Fix the processor so it hands every chunk back, and prove it with a check that sends a body.

What you’ll need: Docker (or colima with nerdctl), Python 3.12 or later (tested on 3.13), curl, git, and three terminals. If you have not met ext_proc yet, start with Envoy and a Stub ext_proc That Logs Every Request. The “why” is in What May Sit in the Hot Path.

Get the code

The backend, both processors and the check are in the lab repo. Clone it under your home directory (colima shares only ~ with its VM) and install the pinned gRPC runtime and Envoy’s generated protos:

git clone https://github.com/mvpzone/mvplabs.git
cd mvplabs/codelab/ext-proc-full-duplex
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
docker pull envoyproxy/envoy:v1.39.4

The backend, backend.py, has three routes, all POST:

  • /hello answers without reading the request body. It is the control.
  • /rpc is a JSON-RPC endpoint. Like any JSON-RPC server, it reports a bad request inside a 200.
  • /echo returns the body it received.

Start it in the first terminal and leave it running:

python backend.py

backend listening on :8081

Read the config

envoy.yaml is the one from the first lab with two changes: the backend is now on your laptop, and the processing mode sends bodies in both directions.

# envoy.yaml (the ext_proc filter)
              failure_mode_allow: false
              processing_mode:
                request_header_mode: SEND
                response_header_mode: SEND
                request_body_mode: FULL_DUPLEX_STREAMED
                response_body_mode: FULL_DUPLEX_STREAMED
                request_trailer_mode: SEND       # FULL_DUPLEX_STREAMED requires SEND
                response_trailer_mode: SEND

The two body modes you will compare:

Mode What Envoy sends What the processor owes
BUFFERED the whole body, once, after it has all arrived a verdict on it; an empty BodyResponse means “continue unchanged”
FULL_DUPLEX_STREAMED each chunk as it arrives, without waiting for a reply the chunk itself, handed back as a streamed_response

Buffered mode is simple but holds the whole body in Envoy’s memory before the processor sees a byte. Full duplex lets a processor inspect a large or long-lived body as it flows, which is why a policy point that reads bodies wants it.

Run the stub that changes nothing

This processor answers every phase with an empty response of the matching type. In the first lab, an empty HeadersResponse was the correct “continue”. Here it does the same for bodies:

# broken/extproc.py
from concurrent import futures

import grpc
from envoy.service.ext_proc.v3 import external_processor_pb2 as pb
from envoy.service.ext_proc.v3 import external_processor_pb2_grpc as pb_grpc


class ChangeNothing(pb_grpc.ExternalProcessorServicer):
    """A stub: answer every phase, change nothing. Or so it looks."""

    def Process(self, requests, context):
        for req in requests:
            phase = req.WhichOneof("request")
            if phase in ("request_body", "response_body"):
                chunk = getattr(req, phase)
                print(f"{phase}: {len(chunk.body)} bytes, end_of_stream={chunk.end_of_stream}", flush=True)
            else:
                print(phase, flush=True)
            match phase:
                case "request_headers":
                    yield pb.ProcessingResponse(request_headers=pb.HeadersResponse())
                case "response_headers":
                    yield pb.ProcessingResponse(response_headers=pb.HeadersResponse())
                case "request_body":
                    yield pb.ProcessingResponse(request_body=pb.BodyResponse())    # "continue"?
                case "response_body":
                    yield pb.ProcessingResponse(response_body=pb.BodyResponse())   # "continue"?
                case "request_trailers":
                    yield pb.ProcessingResponse(request_trailers=pb.TrailersResponse())
                case "response_trailers":
                    yield pb.ProcessingResponse(response_trailers=pb.TrailersResponse())


def main() -> None:
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=8))
    pb_grpc.add_ExternalProcessorServicer_to_server(ChangeNothing(), server)
    server.add_insecure_port("0.0.0.0:50051")
    server.start()
    print("ext_proc listening on :50051", flush=True)
    server.wait_for_termination()


if __name__ == "__main__":
    main()

Start it in the second terminal, then Envoy and one request in the third:

python broken/extproc.py

docker run -d --name l3-envoy -p 10000:10000 \
  --add-host=host.docker.internal:host-gateway \
  -v "$PWD/envoy.yaml:/etc/envoy/envoy.yaml:ro" envoyproxy/envoy:v1.39.4
curl -s -i localhost:10000/hello -d 'hi'

On colima, run each docker command as colima nerdctl -- with the same flags, and use --add-host=host.docker.internal:192.168.5.2. The first lab explains why.

HTTP/1.1 500 Internal Server Error
date: Fri, 23 Oct 2026 02:14:57 GMT
server: envoy
content-length: 0

The processor saw the request, a 2-byte chunk and an empty final chunk:

ext_proc listening on :50051
request_headers
request_body: 2 bytes, end_of_stream=False
request_body: 0 bytes, end_of_stream=True

Envoy says why it failed the request:

docker logs l3-envoy 2>&1 | grep Spurious

[2026-10-23 02:14:57.933][13][warning][ext_proc] [source/extensions/filters/http/ext_proc/ext_proc.cc:1915] [Tags: "ConnectionId":"0","StreamId":"4539792447903703196"] Spurious response message 3 received on gRPC stream

Message 3 is the request_body field of ProcessingResponse. In this mode, upstream Envoy does not accept an empty BodyResponse as a reply to a streamed chunk. It treats it as a protocol error and, with failure_mode_allow: false, fails the request. Setting failure_mode_allow: true does not change this: the stream is up, so the reply itself is the error.

This is the lucky version. Every request with a body fails, loudly, on the first try. Not every implementation of the protocol is this strict. The one we first ran behind accepted the same reply and forwarded an empty chunk. Two steps on, you reproduce that.

The same stub, buffered

Why would anyone write that stub? Because in the other body modes it is correct. Switch the request body to buffered, the response body and the trailers off:

# envoy.yaml (buffered, for this step only)
              processing_mode:
                request_header_mode: SEND
                response_header_mode: SEND
                request_body_mode: BUFFERED
                response_body_mode: NONE
                request_trailer_mode: SKIP
                response_trailer_mode: SKIP

Recreate Envoy with that file, keep the broken stub running, and send a 300,000-byte body through /echo with the check from the repo:

docker rm -f l3-envoy
docker run -d --name l3-envoy -p 10000:10000 \
  --add-host=host.docker.internal:host-gateway \
  -v "$PWD/envoy.yaml:/etc/envoy/envoy.yaml:ro" envoyproxy/envoy:v1.39.4
./canary.sh

canary OK: 300000 bytes round-tripped (e075fec3384c40e9)

Buffered, the empty BodyResponse is the continue: the stub passes. Same code, same reply, and whether it is right depends on a setting in another file. Put the full-duplex modes back in envoy.yaml (or git checkout envoy.yaml) and recreate the container before the next step.

The quiet failure

Now reproduce what the lenient implementation did: hand every chunk back empty. The fixed processor (next step) has a switch for exactly this. DROP=request returns each request chunk with no bytes in it:

DROP=request python fixed/extproc.py

Call the control, then the JSON-RPC endpoint:

curl -s localhost:10000/hello -d 'hi' -w '[%{http_code}]\n'
curl -s localhost:10000/rpc -d '{"jsonrpc":"2.0","id":1,"method":"ping","params":{"n":1}}' -w '[%{http_code}]\n'

hello from the backend
[200]
{"jsonrpc": "2.0", "id": null, "error": {"code": -32700, "message": "Parse error: Expecting value: line 1 column 1 (char 0)"}}
[200]

The backend’s log shows what it actually got:

backend: POST /hello 200 received=-
backend: POST /rpc 200 received=0

Read that as a dashboard would. Two requests, two 200s, no errors. The control is green because it never reads a body, so it cannot tell a body from no body. The JSON-RPC endpoint is green because JSON-RPC puts its errors inside a 200. The only signal is the text of a parse error, read by whoever calls /rpc. In our case that was the first body-reading client to try the path, an MCP server, which answered Parse error: EOF.

DROP=all empties the response chunks too. Restart the processor with it and call the control again:

DROP=all python fixed/extproc.py
curl -s -i localhost:10000/hello -d 'hi'

HTTP/1.1 200 OK
server: envoy
date: Fri, 23 Oct 2026 02:15:04 GMT
content-type: text/plain
x-envoy-upstream-service-time: 0
transfer-encoding: chunked

A 200 with no body. Even the control’s answer is gone, and its status code still says everything is fine.

Echo every chunk

In full-duplex mode, the processor’s reply to a body chunk is the chunk. Continuing unchanged means handing back the same bytes, with the same end_of_stream, in a streamed_response. Envoy also says which mode each direction uses, once, in the first message’s protocol_config; read it so the same processor stays correct in buffered mode:

# fixed/extproc.py
import os
from concurrent import futures

import grpc
from envoy.extensions.filters.http.ext_proc.v3.processing_mode_pb2 import ProcessingMode
from envoy.service.ext_proc.v3 import external_processor_pb2 as pb
from envoy.service.ext_proc.v3 import external_processor_pb2_grpc as pb_grpc

# DROP=request hands every request chunk back EMPTY; DROP=all does it in both directions. The quiet-failure step uses it.
DROP = os.environ.get("DROP", "")


def body_response(chunk, streamed: bool, phase: str) -> pb.BodyResponse:
    """In FULL_DUPLEX_STREAMED mode a body chunk is yours until you hand it back."""
    if not streamed:
        return pb.BodyResponse()   # buffered / streamed modes: the empty response is the continue
    drop = DROP == "all" or (DROP == "request" and phase == "request_body")
    body = b"" if drop else chunk.body
    return pb.BodyResponse(response=pb.CommonResponse(body_mutation=pb.BodyMutation(
        streamed_response=pb.StreamedBodyResponse(body=body, end_of_stream=chunk.end_of_stream))))


class EchoEveryChunk(pb_grpc.ExternalProcessorServicer):
    def Process(self, requests, context):
        request_streamed = response_streamed = True
        for i, req in enumerate(requests):
            if i == 0 and req.HasField("protocol_config"):
                # Envoy says which body mode each direction uses, once, in the first message.
                cfg = req.protocol_config
                request_streamed = cfg.request_body_mode == ProcessingMode.FULL_DUPLEX_STREAMED
                response_streamed = cfg.response_body_mode == ProcessingMode.FULL_DUPLEX_STREAMED
            phase = req.WhichOneof("request")
            if phase in ("request_body", "response_body"):
                chunk = getattr(req, phase)
                print(f"{phase}: {len(chunk.body)} bytes, end_of_stream={chunk.end_of_stream}", flush=True)
            else:
                print(phase, flush=True)
            match phase:
                case "request_headers":
                    yield pb.ProcessingResponse(request_headers=pb.HeadersResponse())
                case "response_headers":
                    yield pb.ProcessingResponse(response_headers=pb.HeadersResponse())
                case "request_body":
                    yield pb.ProcessingResponse(request_body=body_response(req.request_body, request_streamed, phase))
                case "response_body":
                    yield pb.ProcessingResponse(response_body=body_response(req.response_body, response_streamed, phase))
                case "request_trailers":
                    yield pb.ProcessingResponse(request_trailers=pb.TrailersResponse())
                case "response_trailers":
                    yield pb.ProcessingResponse(response_trailers=pb.TrailersResponse())


def main() -> None:
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=8))
    pb_grpc.add_ExternalProcessorServicer_to_server(EchoEveryChunk(), server)
    server.add_insecure_port("0.0.0.0:50051")
    server.start()
    print("ext_proc listening on :50051", flush=True)
    server.wait_for_termination()


if __name__ == "__main__":
    main()

Restart it with no DROP and call /rpc again:

python fixed/extproc.py
curl -s localhost:10000/rpc -d '{"jsonrpc":"2.0","id":1,"method":"ping","params":{"n":1}}' -w '[%{http_code}]\n'

{"jsonrpc": "2.0", "id": 1, "result": {"echo": {"n": 1}}}
[200]

backend: POST /rpc 200 received=57

The backend received no content-length header for /rpc. Envoy cannot know the final length in this mode, because the processor may change the body, so it forwards the body with transfer-encoding: chunked. Real servers handle that. Python’s http.server does not, which is why backend.py carries its own read_body. A backend that trusts content-length alone reads zero bytes here, and you are back to the previous step by a different road.

A control that reads the body

The control in our incident was a service that never reads a body, so it could not show the one thing that broke. canary.sh exercises that thing: it sends 300,000 random bytes through /echo and compares the hash of what comes back.

# canary.sh
#!/bin/sh
# A control that exercises the body: send random bytes through the proxy, compare what comes back.
set -eu
URL="${1:-http://localhost:10000/echo}"
head -c 300000 /dev/urandom > /tmp/canary.in
curl -s "$URL" --data-binary @/tmp/canary.in -o /tmp/canary.out
sent=$(shasum -a 256 < /tmp/canary.in | cut -c1-16)
got=$(shasum -a 256 < /tmp/canary.out | cut -c1-16)
if [ "$sent" = "$got" ]; then
  echo "canary OK: 300000 bytes round-tripped ($sent)"
else
  echo "canary FAILED: sent $sent, got $got ($(wc -c < /tmp/canary.out | tr -d ' ') bytes back)"
  exit 1
fi

Against the fixed processor:

./canary.sh

canary OK: 300000 bytes round-tripped (75e6ab8579f1a9c5)

The processor’s log shows why a large body is worth sending. The 300,000 bytes crossed it as 5 request chunks and 44 response chunks, so the check covers many chunks and the final empty one, not just a single small body.

Now restart the processor with DROP=request and run it again:

canary FAILED: sent b886f8b85754f88c, got e3b0c44298fc1c14 (0 bytes back)

It exits 1, so a deploy pipeline or a prober can act on it. e3b0c442… is the SHA-256 of zero bytes; you will see it again.

Clean up:

docker rm -f l3-envoy

Stop the processor and the backend with Ctrl-C.

What to keep

  • In FULL_DUPLEX_STREAMED mode the processor holds the body. “Continue” is handing each chunk back, bytes and end_of_stream, as a streamed_response. The empty BodyResponse that continues in buffered mode is a protocol error or a deletion here.
  • Read protocol_config from the first message instead of assuming a mode. The same reply is right in one mode and destructive in the other.
  • Implementations differ on the wrong reply: upstream Envoy fails the request, and one we ran forwarded an empty chunk. Write to the protocol, not to the error you happen to get.
  • Pick controls that exercise what you changed. A service that never reads a body cannot vouch for bodies; a round-trip hash can.

The stub did nothing, and in that mode nothing meant delete.