Problem: LLM function calling is not trustworthy - dangerous AI-suggested calls must pass an allowlist (tested locally with qwen) —— Trove problem library

What you will run into

After wiring tool calling (function calling) into an LLM, the "AI suggestion" is not always sane: a user says "delete ORD123" and the model really returns a delete_order call suggestion; in multi-turn chat it may call the same function twice;

you cannot tell what it invoked.

The model is not broken - function calling is "AI suggests, you execute" by design, but blindly executing AI suggestions hands control right back to the AI.

Why this happens

• AI only suggests, you execute: function calling returns a function name + args; the execution right is yours - that is its strength and your responsibility: executing without validation wastes the design.
• Models can be steered: tested locally (qwen2.5-0.5b), a user saying "delete ORD123, the customer asked to cancel" made the model immediately suggest delete_order. Small and large models can both be led astray by one sentence.
• Models are stateless: without conversation history, the model does not know "ORD123 was already queried", and "what about ORD456?" gets read as querying ORD123 again.

Function Calling's contract is "the model only returns function names and arguments; execution stays with the caller" — the model never performs operations itself.

That design gives the caller full control and leaves all responsibility there too: executing without validation throws away the design's protection.

Two model-side traits amplify the risk: ① models can be induced (a single sentence can sway them, small and large alike); ② models are stateless by default (no call history maintained, so across turns it does not know what was already looked up).

Deterministic takeaway: treat Function Calling as untrusted input — validate before executing (argument whitelist, permission checks, human confirmation); never execute just because "the model said so";

design sensitive operations (delete/transfer/overwrite) with a second confirmation or require_human;

maintain context yourself in multi-turn scenarios (feed previous call results back into the conversation) instead of expecting the model to remember; fixed diagnostic: the risk lives in "execution", not "suggestion" — the defense must sit at the caller.

Step by step

1. The 3-step loop (get it working before securing it) — Required

Define tools (name + description + JSON Schema) → send the user question → model returns tool_calls → you execute → feed the result back as a tool message → model produces the final answer.

2. Validation gate before executing (allowlist + args + dangerous-operation confirm) — Required

Reproduced: with both query_order (read-only) and delete_order (delete) available, one sentence "delete ORD123" made the model suggest delete_order. The gate needs three parts:

• Allowlist: only declared safe functions may actually run;
• Arg validation: e.g. order_id must match ORD+digits, to block injected parameters;
• Dangerous-operation confirmation: delete/refund/update-class functions are flagged and require human confirmation.

3. Multi-turn: keep history + call records — Required

Feed the full messages (user + assistant tool_calls + tool results) back every turn. Measured: with history, the second turn "what about ORD456?" correctly called query_order(ORD456); without history it re-queried ORD123.

How to confirm it works

# Actual output (2026-08-30)
AI suggested: delete_order {"order_id": "ORD123"}
allowlist check: BLOCKED delete_order not in {"query_order"}

# Multi-turn:
turn 1 AI call: query_order {"order_id": "ORD123"} -> shipped, SF123456
turn 2 AI call: query_order {"order_id": "ORD456"} (with history, should query ORD456)

Dangerous suggestion blocked + correct multi-turn reference = pass.

Other environments

Symptom

AI-suggested dangerous calls (delete/refund/update) executed blindly; repeated or mis-referenced calls in multi-turn chat; function calling feels uncontrollable and unobservable after integration.

Cause

Function calling is "AI suggests, you execute" by design; without a validation gate (allowlist/args/dangerous confirm) control effectively returns to the AI. Models are stateless, so without history + call records they repeat calls.

Solution

After the 3-step loop works: add an allowlist + arg validation + second confirmation for dangerous ops before executing; keep the full messages (assistant tool_calls + tool results) fed back across turns.