Problem: Java OOM signature cheat-sheet: heap space / Metaspace / StackOverflowError / native thread / GC Overhead —— Trove problem library

What you will run into

A Java service suddenly dies / returns 502 / the API answers 500, and the logs show various OOM errors. Different signatures point to completely different root causes - match the signature first:

Why this happens

JVM memory has four independent areas - heap / metaspace / thread stacks / native threads - each can be exhausted on its own:

JVM "memory" is not one thing — it is four independent exhaustion channels, each with different OOM text and different fixes: the Java heap is GC-managed and only exhausts when objects accumulate;

Metaspace stores class metadata and exhausts when classes are generated dynamically (e.g.

full XSD→classes); thread stacks are fixed per thread (~1MB class) and exhaust via deep recursion (StackOverflowError);

native threads are bounded by the OS thread limit (ulimit -u / threads-max) and exhaust when an unbounded pool only grows (threads blocked on log writes).

GC thrashing (reclaiming <2% while using >98% of time) is the fingerprint of "many unreachable-but-uncollected objects".

Deterministic takeaway: the region name in the OOM text (Java heap space / Metaspace / unable to create native thread / GC overhead) decides the direction — heap OOM: object accumulation and leaks;

Metaspace OOM: class-loading volume; native-thread OOM: thread count and blocking;

do not blanket-apply "-Xmx"; fixed diagnostic: read the exception class name before acting.

Step by step

1. Record the signature (decide the class first) — Required

Java heap space / Metaspace / StackOverflowError / unable to create new native thread / GC Overhead limit exceeded / Direct buffer memory - different signature, different weapon.

2. Reproduce and collect evidence — Required

• Enable -XX:+HeapDumpOnOutOfMemoryError to get a .hprof on OOM;
• jmap -dump:format=b,file=app.hprof <pid> while the service is still alive;
• Open the dump in MAT: Dominator Tree big objects + Incoming references (case 8: ThreadLocal leak = Incoming references point to ThreadLocalMap);
• jstack <pid> for thread stacks (check total threads when native threads are exhausted).

3. Fix by root cause — Required

4. Verify — Required

Re-run the original trigger path under load: no OOM, stable GC log, thread count back to normal.

How to confirm it works

The same trigger path no longer OOMs, the GC log shows no consecutive zero-reclaim Full GCs, the thread count is stable, and the old big objects / leak references are gone from a fresh dump.

Other environments

Symptom

Java service OOMs: Java heap space (big file / big query / ZIP bomb / -Xmn imbalance), Metaspace (full schema load), StackOverflowError (recursion without exit), unable to create new native thread (thread exhaustion), GC Overhead limit exceeded (GC churn).

Cause

Each JVM memory area exhausts independently: heap filled by big objects/collections, Metaspace by class metadata, thread stacks by deep recursion, native threads over the OS limit, and leaked objects make GC churn with zero reclaim.

Solution

1. Enable -XX:+HeapDumpOnOutOfMemoryError + GC log first;
2. Match the error signature to its class (see the cheat-sheet);
3. Stream big files / external links, paginate + cache big queries, block ZIP bombs (size/nesting/path checks), chunked schema loading, parameter validation + recursion to loop, bounded thread pool + non-blocking logging, sane -Xmn ratio, remove ThreadLocal in finally;
4. Prove with MAT/jstack, then verify under load.