Problem: Netty direct-memory OOM: OutOfDirectMemoryError / Direct buffer memory (ByteBuf leak + unset MaxDirectMemorySize) —— Trove problem library

What you will run into

A Spring Cloud Gateway (reactor-netty) style gateway suddenly returns occasional 500s; the logs show:

io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 16777216 byte(s) of direct memory (used: 1006632967, max: 1013645312)

The frequency climbs, the service becomes unavailable, health checks fail. Heap (-Xmx) looks fine and GC logs look normal - but direct memory is nearly exhausted (used 960MB / max 967MB).

Why this happens

Netty's ByteBuf defaults to PooledByteBufAllocator, i.e. off-heap direct memory. Direct memory is not managed by GC - allocation/release is your responsibility; leaked ByteBufs are never reclaimed and grow until the cap. Three compounding factors:

Low traffic hides it; concurrency or a few large simultaneous requests blow it up.

Netty's ByteBuf defaults to PooledByteBufAllocator: memory is allocated off-heap (direct memory) and not managed by GC — allocation and release rely entirely on reference counting (refCnt):

retain increments, release decrements, and a single missed release leaks forever until MaxDirectMemorySize is hit.

Three compounding factors each have their own mechanism: with no explicit cap the default equals -Xmx (a K8s container with 1500Mi leaves little direct space after a 1000MB heap); a Filter caching request bodies with no size limit (reads a 50MB upload + exception paths never release → leak);

a reactive join interrupted by disconnect/error orphans intermediate buffers. Under low traffic, pooled buffer reuse hides the leak; it blows up once concurrency rises.

Deterministic takeaway: direct OOM is a "leak + cap" two-variable problem — first fix the leak (retain/release pairing, finally-release on exception paths), then tune the cap (-XX:MaxDirectMemorySize matched to container memory); jcmd <pid> VM.native_memory shows the direct allocation trend;

fixed diagnostic: OOM with heap not full + Netty in the stack = direct-memory problem, not an -Xmx problem.

Step by step

1. Confirm direct memory from the signature — Required

OutOfDirectMemoryError (Netty wrapper) or OutOfMemoryError: Direct buffer memory (JDK native), carrying used/max numbers - check whether max is roughly -Xmx (the default when the flag is unset).

2. Inspect JVM flags and container limits — Diagnose

# Is MaxDirectMemorySize set? (0 = defaults to -Xmx)
java -XX:+PrintFlagsFinal -version 2>&1 | grep MaxDirectMemorySize
# Compare with the container limit: heap + direct + metaspace + stacks must all fit
kubectl get deploy <name> -o yaml | grep -A2 resources

3. Find who eats the direct memory — Required

• Gateway: Netty I/O itself uses direct memory (normal); focus on custom Filters that cache request/response bodies into byte[], DataBufferUtils.join, and retain without guaranteed release;
• Enable leak detection during a low-traffic window: -Dio.netty.leakDetectionLevel=paranoid - the logs will print the leak origin;
• jstack/heap dumps do not help with direct memory; rely on code review + leak detection.

4. Fix — Required

• Add a size cap to request-body caching (e.g. 1MB), skip the cache above it and log a warn;
• Add doFinally / onErrorResume to the reactive chain to guarantee release;
• Set -XX:MaxDirectMemorySize explicitly, sized for gateway traffic (heap and direct memory separated, neither squeezes the other);
• Recompute the K8s container limit (heap + direct + metaspace + stacks + OS overhead).

5. Verify — Required

Re-run the trigger path under load (large requests + concurrency): the direct-memory watermark stays flat; leak detection reports no new leaks.

How to confirm it works

The same trigger path no longer raises OutOfDirectMemoryError; direct-buffer watermarks (e.g. jcmd <pid> VM.native_memory) stay stable; health checks pass.

Other environments

Symptom

Gateway returns occasional 500s that escalate: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate N byte(s) of direct memory (used: X, max: Y), service becomes unavailable, health checks fail; heap and GC logs stay normal.

Cause

Direct (off-heap) memory is not managed by GC: -XX:MaxDirectMemorySize defaults to -Xmx when unset; custom Filters cache request/response bodies without a size cap and rely on downstream consumption for release, so exceptions/timeouts leak ByteBufs until the cap is hit.

Solution

1. Set -XX:MaxDirectMemorySize explicitly (separate heap and direct memory) and raise the container limit accordingly;
2. Cap request/response body caching, skip above the limit;
3. Add doFinally / onErrorResume release guarantees to the reactive chain;
4. Enable -Dio.netty.leakDetectionLevel=paranoid to locate leaks;
5. Verify the direct-memory watermark under load.