Combo Draft: Local LLM Finetune / Inference Environment (ollama + open-webui + LLaMA-Factory)

Scenario

Local LLM environment = model download and runtime base (ollama, one command runs models + OpenAI-compatible API) + web UI (open-webui, chat with/manage models in the browser) + finetune tool (LLaMA-Factory, local finetune/training). Each tool owns one stage: base → UI → training. Users usually get stuck on "models won't download (China network)", "GPU driver / memory enough?", "port conflicts" — this combo provides download sources, order and pitfalls. Plus a local knowledge base (RAG): feed documents to open-webui and let it answer from your own material (full chain measured 2026-08-30).

Combo members

ollama: model download and runtime base (core)

Download: official https://ollama.com/download (install.sh / Windows installer / macOS dmg); Linux binary = GitHub Release asset ollama-linux-amd64.tar.zst (China uses the gh-proxy prefix)
Version: >= 0.33.2 (2026-08-29 latest v0.33.2; versions < 0.30 have poor API/model compatibility, recommend going straight to latest)
Install:

# Linux (official script; when China direct GitHub is slow use the manual path below)
curl -fsSL https://gh-proxy.com/https://github.com/ollama/ollama/releases/latest/download/install.sh | sh
# or manual: download the tar.zst and extract (needs tar --zstd or the zstd tool)
curl -sL -o /tmp/ollama.tar.zst https://gh-proxy.com/https://github.com/ollama/ollama/releases/latest/download/ollama-linux-amd64.tar.zst
mkdir -p /tmp/ollama-bin && tar --zstd -xf /tmp/ollama.tar.zst -C /tmp/ollama-bin
/tmp/ollama-bin/bin/ollama --version   # expected output: ollama version 0.33.2

note: model downloads default to the official ollama registry (may time out in China); alternative = download the GGUF from ModelScope and import with ollama create (measured 2026-08-29: qwen2.5-0.5b 491MB downloaded in 58.5s + create import success + CPU smoke passed, evidence local repro log 2026-08-29).
note: Windows uses the official installer (https://ollama.com/download/windows), inside WSL2 same path as Linux; macOS uses Ollama-darwin.zip.

open-webui: web UI (companion)

Download: official Docker image ghcr.io/open-webui/open-webui (GitHub Release v0.11.1, 2026-08-29); China mirror ghcr.nju.edu.cn measured working (2026-08-30, image 7.16GB)
Version: >= 0.11.1 (Docker tag recommended :main to follow latest; pip form needs Python 3.11+)
Install:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data --name open-webui --restart always \
-e ENABLE_SIGNUP=true \
-e RAG_EMBEDDING_ENGINE=ollama \
-e RAG_EMBEDDING_MODEL=nomic-embed-text \
ghcr.nju.edu.cn/open-webui/open-webui:main
# China mirror: ghcr.nju.edu.cn directly (measured 2026-08-30, image 7.16GB); official ghcr.io works with the same prefix swap

note: in-container port 8080, host mapping example 3000; open http://localhost:3000 in the browser. First start pulls the image ~7.16GB (measured, China network is the biggest bottleneck). The three env vars are required for RAG: ENABLE_SIGNUP=true (first signup becomes admin), RAG_EMBEDDING_ENGINE=ollama + RAG_EMBEDDING_MODEL=nomic-embed-text (local embeddings, no external model download).
note: pip install open-webui also works (officially supported, not measured; heavy deps, recommend the Docker form).

nomic-embed-text: knowledge-base embedding (RAG light piece)

Download: ollama pull nomic-embed-text (official registry, 274MB, measured 2026-08-30 ~45s at 6.3MB/s); if the registry is unreachable, get a GGUF from ModelScope and import (same path as ollama models)
Version: latest (works with ollama 0.32.x, measured 2026-08-30)
Install: see Step 4 in the tutorial

LLaMA-Factory: local finetune tool (optional heavy piece)

Download: official PyPI (package llamafactory; source GitHub hiyouga/LlamaFactory, latest tag v0.9.5, 2026-08-29); China uses the Tsinghua mirror
Version: >= 0.9.5 (Python >= 3.10; needs PyTorch 2.x, CUDA version requires installing the matching torch wheels first)
Install:

pip install llamafactory --index-url https://pypi.tuna.tsinghua.edu.cn/simple
llamafactory-cli version   # expected output: LLaMA Factory version (measured 0.9.5, 2026-08-30)

note: finetune training needs an NVIDIA GPU + CUDA environment (no GPU = CPU-inference-level trials only, training unrealistic); install the CUDA PyTorch first (pip install torch --index-url https://download.pytorch.org/whl/cu12x, China can use Aliyun/Tsinghua torch mirrors, inferred not measured), then LLaMA-Factory.

Install order

ollama first (base) — model download/run/API all depend on it; after install ollama --version smoke, ollama pull <model> verifies the model source (China network is most likely to stick at this step).
Then open-webui (UI) — Docker form is independent of the ollama process, connects to ollama via port 11434; start ollama first then the UI so the UI can auto-discover models.
Finally LLaMA-Factory (finetune) — optional heavy piece; only install when you're sure about finetuning, so the torch big dependency doesn't slow down the first two steps.
(Optional) Inference-only users can skip LLaMA-Factory; llama.cpp is a lightweight inference alternative to ollama (runs GGUF directly, see dependency warnings).
(Optional but recommended) Knowledge base (RAG): install the embedding model and configure open-webui — see Step 4.

Dependency warnings

Conflicts: this trio is a general-computer combo (outside the plugin-domain 21 entries), no write-surface overlap between them (ollama writes ~/.ollama/ + model dir; open-webui writes a Docker volume; LLaMA-Factory writes the project dir/venv, inferred compatible, not measured). Conflicts with plugin-domain entries aren't in the matrix (beyond the recorded scope); ports 11434/8080 don't clash with claude-mem etc., but when all share the GPU, stagger usage (inferred).
Versions: ollama >= 0.33.2; open-webui >= 0.11.1 (image :main); LLaMA-Factory >= 0.9.5 (Python >= 3.10, PyTorch 2.x); nomic-embed-text latest (274MB).
Environment:
GPU driver: NVIDIA driver (ollama ships its own CUDA runtime, but the driver must be recent; no NVIDIA card = CPU-only inference, slow); AMD cards use ROCm assets (ollama-linux-amd64-rocm.tar.zst, inferred); Apple Silicon uses MPS (native macOS support).
Memory: recommend >= 16GB; a quantized model (Q4) takes roughly half its parameter count in memory (7B model ≈ 4-5GB memory + KV cache); the embedding model is tiny (well under 0.3GB, negligible).
Disk: model files 4-70GB (7B Q4 ≈ 4.5GB, 70B Q4 ≈ 40GB+), confirm with df -h before installing; the model dir defaults to ~/.ollama/models (changeable with OLLAMA_MODELS).
Ports: ollama defaults to 11434 (OLLAMA_HOST can change); open-webui host port example 3000 (in-container 8080); change the mapped port when it conflicts with other services.
System differences: Linux (install.sh/tar.zst), Windows (official installer, WSL2 recommended), macOS (dmg, Apple Silicon native).
China network: ①ollama binary download via gh-proxy (direct is slow); ②model pulls go through the ollama registry — small models measured reachable 2026-08-30 (274MB/45s), large models may still time out — alternative ModelScope download GGUF + ollama create import (measured passing 2026-08-29: qwen2.5-0.5b 491MB downloaded in 58.5s); ③open-webui image via ghcr.nju.edu.cn (measured, 7.16GB); ④embedding model 274MB via the official registry (measured) or ModelScope GGUF; ⑤pip always via the Tsinghua mirror. The most likely sticking points = large model pull, ghcr image pull, torch big dependency.

Tutorial

Step 1: install ollama (base) — the official script or the manual tar.zst path (see ollama member notes); verify with ollama --version and curl http://localhost:11434/api/version. Step 2: install open-webui (UI) — docker run command above (with the three RAG env vars); verify docker ps shows Up and http://localhost:3000 opens the signup/login page. First signup becomes the admin. Step 3: install LLaMA-Factory (optional) — pip install llamafactory --index-url https://pypi.tuna.tsinghua.edu.cn/simple; verify llamafactory-cli version. Step 4: knowledge base (RAG, optional but recommended) — install the embedding model inside ollama:

ollama pull nomic-embed-text   # 274MB, official registry (measured ~45s)
curl -s localhost:11434/api/embed -d '{"model":"nomic-embed-text","input":["test"]}'   # expected: 768-dim vector

Then in the browser (http://localhost:3000, admin login):

Admin panel (bottom-left user menu) -> Settings -> Retrieval: confirm Embedding engine = Ollama, model = nomic-embed-text (set by the env vars on first install; for upgraded/reused volumes check manually, see FAQ).
Main screen -> left sidebar "Knowledge" -> create a knowledge base -> upload documents (md/txt/pdf etc.; md measured working).
New chat -> "#" button left of the input -> select the knowledge base -> ask. Answers come with citations.

note: automatic retrieval depends on the model's tool-calling ability: 0.5B-class small models measured NOT triggering retrieval (hallucinated answers); use >= 7B models with function calling, or paste the key content directly into the prompt.
note: measured — the embedding model name must NOT carry an ollama/ prefix (open-webui passes it verbatim to ollama which returns 404).

Acceptance check (one shot)

Run this block after installing; all green = combo ready; any failure -> back to the matching step in the install walkthrough.

ollama --version
curl -s http://localhost:11434/api/version
docker ps | grep open-webui
llamafactory-cli version
curl -s localhost:11434/api/embed -d '{"model":"nomic-embed-text","input":["ok"]}' | head -c 120

预期输出: ollama 0.33.2; api/version returns a version; open-webui Up; llamafactory-cli prints a version; embed returns a 768-dim vector (member-level measured 2026-08-29/30; full fine-tune pipeline not measured)

Note: if open-webui cannot reach ollama, confirm the ollama service is running (step 1 curl check); if the port changed, sync it in the UI settings.

Verification evidence

Every combo is measured in an isolated environment (smoke + write-surface check); unmeasured items are honestly marked in the text, and measurement dates are listed in the verification section below.

Verified 2026-08-30 · TTL expires 2027-02-26 (180 days)

Measured (2026-08-29, WSL Ubuntu x86_64, Docker 29.1.3): ollama v0.33.2 binary downloaded via gh-proxy (tar.zst 1422262024 bytes) → apt installed zstd 1.5.7 → tar --zstd extracted → all smoke tests passed: ollama --version → client version is 0.33.2; ollama serve starts normally (OLLAMA_MODELS effective, port 11434, blobs: 0); ollama list exit 0 (empty table); curl /api/version → {"version":"0.33.2"}.
Measured observations (evidence of China bottlenecks): ollama serve logs show cloud model cache pull failures (ollama.com:443 connection failed/timeout) — the cloud-model display cache depends on the official registry, fails on China networks; local model pulls also depend on ollama.com, alternative = ModelScope GGUF download + ollama create import (inferred, not measured).
Measured (2026-08-30, RAG full chain, evidence local repro log 2026-08-30): nomic-embed-text pulled from the official registry (274MB/45s) -> /api/embed returns 768-dim normalized vectors -> open-webui switched to ollama embeddings (bare model name required; ollama/ prefix measured 404) -> knowledge created/uploaded/split/embedded (1/1 items) -> query/collection retrieval hit the correct chunk (metadata echoes ollama + nomic-embed-text).
Measured limitation (same log): open-webui's RAG runs through model tool calls; qwen2.5-0.5b did not trigger automatic retrieval (hallucinated); the retrieval layer itself is verified, use >= 7B models or pick the knowledge base manually.
Environment fix (same log): a leftover legacy dockerd (29.1.3) in WSL hijacked port 3000 from Docker Desktop's engine; stopped and disabled the legacy daemon. Does not affect this combo's install path.
Not measured (stated honestly): model pull (4GB+ in size, big downloads limited in this environment); LLaMA-Factory install and finetune (needs GPU/CUDA, no NVIDIA card in this environment); ModelScope import for embedding models (registry worked this round).

Combo Draft: Local LLM Finetune / Inference Environment (ollama + open-webui + LLaMA-Factory) ✓ Verified