litellm

What you will run into

You have several LLM providers (or several clients needing one model entry point) and you want one OpenAI-compatible endpoint in front of them all — LiteLLM is that proxy gateway: one base_url, unified keys and cost tracking. The two things people trip on: the dependency set is heavy (install into an isolated venv; PyPI official is slow on mainland networks — the Tsinghua mirror took 80s, verified), and "it is a server": you start it with --config ... --port ... and it stays in the foreground (stop with pkill). When to use it:

• One OpenAI-compatible endpoint for multiple models/providers;
• Unified key management and cost tracking across providers;
• Mock smoke-testing the proxy surface with zero accounts.

When not to use it: you only call a single provider directly — the proxy layer is extra moving parts.

Path: Quick fix (30s, one command) -> Step by step (skimmable) -> How to confirm it works (success criteria). In a hurry? Jump to Quick fix.

Why this happens

pip installs litellm[proxy] (heavy deps, install into an isolated venv). The proxy reads a YAML config (--config, e.g. model_list with litellm_params), listens on a port (default 4000, changeable with --port), and exposes an OpenAI-compatible API (health/liveliness, v1/models, chat completions). Real use needs real model names + API keys in the config; with openai/mock + a dummy key you can verify the whole proxy surface without any account (verified 2026-08-29). Key concepts:

• model_list: providers/models + keys in the YAML config;
• proxy: foreground/managed process (pkill to stop);
• mock: openai/mock + dummy key for zero-account smoke.

Where it writes: no system/user-level config; the proxy config belongs to you (the --config path).

Step by step

Step 1: Install into an isolated venv (why venv: litellm deps are heavy, an isolated env uninstalls cleanest)

Why: litellm[proxy] has many dependencies; installing into a dedicated venv (~/.litellm-venv) avoids polluting system Python, and uninstall = delete the directory. Mainland PyPI official is slow; use the Tsinghua mirror (verified: 80s install on 2026-08-29).

python3 -m venv ~/.litellm-venv
~/.litellm-venv/bin/pip install -i https://pypi.tuna.tsinghua.edu.cn/simple litellm[proxy]
~/.litellm-venv/bin/litellm --version

Expected output: LiteLLM: Current Version = 1.98.0 After install, confirm:

• --version prints 1.98.0;
• use ~/.litellm-venv/bin/litellm for subsequent commands (or activate the venv).

Stuck? How to recover:

• pip timeout → make sure the command has the Tsinghua mirror prefix;
• venv creation failed → confirm Python >= 3.8 and the venv module is available.

Step 2: Write a minimal proxy config (mock smoke, zero accounts)

Why: real use requires model API keys in the config; this environment has zero accounts, so use openai/mock + a dummy key to smoke — it fully verifies the proxy surface (startup, health check, model list); for real forwarding, swap in real keys.

cat > /tmp/ltm-config.yaml <<'EOF'
model_list:
- model_name: mock-gpt
litellm_params:
model: openai/mock
api_key: dummy
litellm_settings:
drop_params: true
EOF

Expected output: config file ready (verified 2026-08-29) After install, confirm:

• /tmp/ltm-config.yaml content is correct.

Stuck? How to recover:

• config format error → check YAML indentation and field names (model_list/litellm_settings).

Step 3: Start the proxy + smoke

Why: litellm --config ... --port 8099 starts (default 4000; 8099 avoids port collisions here). Then health check and model list verify the proxy surface.

~/.litellm-venv/bin/litellm --config /tmp/ltm-config.yaml --port 8099

Expected output: Uvicorn running on http://0.0.0.0:8099 (Application startup complete; verified 2026-08-29)

curl -s http://127.0.0.1:8099/health/liveliness
curl -s http://127.0.0.1:8099/v1/models

Expected output: "I'm alive!"; {"data":[{"id":"mock-gpt",...}]} (verified 2026-08-29) After install, confirm:

• startup log shows Uvicorn running;
• both curl calls return the expected content.

Stuck? How to recover:

• port taken → change --port (default 4000, check with lsof -i :4000);
• startup failed → check the config (step 2) and the logs.

How to confirm it works

Step 1: Confirm version and command surface

Why: ~/.litellm-venv/bin/litellm --version printing 1.98.0 confirms install and PATH; use ~/.litellm-venv/bin/litellm for all subsequent commands.

~/.litellm-venv/bin/litellm --version

Expected output: LiteLLM: Current Version = 1.98.0 Stuck? How to recover:

• command not found → check the venv was created and the path is right;
• version mismatch → reinstall and confirm the mirror prefix;
• Missing dependency websockets / prompted to install litellm[proxy] → the base litellm package (no extras) was installed; litellm --version tries to start the proxy and fails on the missing dep — install litellm[proxy] (reproduced by isolated verify 2026-08-31).

Step 2: Start the proxy and run health checks (smoke)

Why: the proxy starting, health returning I'm alive! and the model list returning mock-gpt confirm the full proxy surface; real use only requires swapping mock for real models and keys in the config.

~/.litellm-venv/bin/litellm --config /tmp/ltm-config.yaml --port 8099

Expected output: Uvicorn running on http://0.0.0.0:8099 (Application startup complete)

curl -s http://127.0.0.1:8099/health/liveliness
curl -s http://127.0.0.1:8099/v1/models

Expected output: "I'm alive!"; {"data":[{"id":"mock-gpt",...}]} Stuck? How to recover:

• port taken → change --port (default 4000, check with lsof -i :4000);
• startup failed → check the config (step 2) and the logs.

Other environments

• Default port: the proxy listens on 4000 by default; this guide uses 8099 to avoid collisions with common services — adjust as needed.
• Windows / macOS: pip install commands are the same (mirror still recommended); no system-level config writes, proxy config comes entirely from the --config path.
• Isolated uninstall: in venv scenarios, deleting ~/.litellm-venv completes cleanup with no system residue.
• Background process: the litellm proxy is a foreground/managed process; stop with pkill when not needed (e.g. pkill -f litellm).

Daily usage

Startup and entry

• Start: ~/.litellm-venv/bin/litellm --config <path> [--port <port>];
• Clients uniformly use http://127.0.0.1:<port> (OpenAI-compatible).

Common operations

Operation 1: Write a minimal proxy config (mock smoke)

cat > /tmp/ltm-config.yaml <<'EOF'
model_list:
- model_name: mock-gpt
litellm_params:
model: openai/mock
api_key: dummy
litellm_settings:
drop_params: true
EOF

Expected output: config file ready (verified 2026-08-29) Operation 2: Start the proxy service

~/.litellm-venv/bin/litellm --config /tmp/ltm-config.yaml --port 8099

Expected output: Uvicorn running on http://0.0.0.0:8099 (Application startup complete; verified 2026-08-29) Operation 3: Health check and model list

curl -s http://127.0.0.1:8099/health/liveliness
curl -s http://127.0.0.1:8099/v1/models

Expected output: "I'm alive!"; {"data":[{"id":"mock-gpt",...}]} (verified 2026-08-29)

Data and directories

• venv: ~/.litellm-venv/;
• Proxy config: the --config path (user-managed, e.g. /tmp/ltm-config.yaml).

Uninstall & cleanup

note: deleting the isolated venv completes uninstall; stop the proxy process first (kill)

pkill -f 'litellm.*--config' 2>/dev/null || true
rm -rf ~/.litellm-venv

Expected output: no residue (command surface gone after venv deletion) Manual checklist (for full cleanup):

• proxy config (the --config path, e.g. /tmp/ltm-config.yaml)
• listening port process (default 4000, check with lsof -i :4000)

Dependency warnings

• Version: [1.98.0, 1.98.0] locked (verified 2026-08-29).
• Environment: Python >= 3.8; isolated venv recommended (heavy deps).
• Network: PyPI official slow, must use the Tsinghua mirror (verified 80s on 2026-08-29); fetching the remote model cost map (raw.githubusercontent.com) at startup times out and falls back to the local backup with a WARNING — non-blocking (isolated verify 2026-08-31).
• Port: default 4000 (--port changeable).
• Write surface: no system/user-level config; the proxy config belongs to you (--config path).
• Background process: the litellm proxy is a foreground/managed process (stop with pkill).
• Co-install: with logfire / claude-code-router judged possible-compatible (different layers, inferred no write-surface overlap).

FAQ

• Q: Will it change my config? A: no system/user-level config writes; proxy config comes from the --config path, user-managed.
• Q: Redundant with claude-code-router / cc-switch? A: different layers — litellm is a server-side gateway (unified entry for multiple services), the other two are client-side config/routing; judged possible-compatible, co-install not tested.
• Q: How do I connect real models? A: put provider model names + api_key in model_list (e.g. openai/gpt-4o, anthropic/claude-*); one base_url serves all clients (OpenAI-compatible).
• Q: Can I try it without an API key? A: yes — openai/mock + dummy key verifies the whole proxy surface (verified 2026-08-29); real forwarding needs real keys.
• Q: Can't pip install domestically? A: use the Tsinghua mirror prefix from the install section (verified 2026-08-29).
• Q: Where to report problems? A: install/uninstall issues on this marketplace go through the entry feedback channel; project issues go to the author's repository.