ollama

What you will run into

You install ollama, run ollama pull qwen2.5, and twenty minutes later it is still spinning — the official registry times out on direct connections from mainland China. Or you cannot even install it: the official install.sh fetches a script from ollama.com, which is painfully slow on mainland networks. Ollama itself is good: one command starts a local model, it ships an OpenAI-compatible API, and setting base_url lets scripts, agents and IDE plugins all talk to the local model. The real hurdle is "getting it installed" on mainland networks. This guide is the fully verified path: binary via gh-proxy from the GitHub Release (1.42GB tar.zst, verified), models via ModelScope GGUF download + import (491MB / 58s, verified), everything copy-pasteable. When to use it:

• Local inference: data never leaves the machine, offline-capable, no per-token billing;
• Multiple models in parallel with hot switching;
• Building a local LLM environment (open-webui as UI, LLaMA-Factory for fine-tuning) — see combo combo-local-llm-finetune.

When not to use it: when you need cloud-scale model capability (very large models / high concurrency / zero operations), a local runtime is not a replacement.

Path: Quick fix (30s, one command) -> Step by step (skimmable) -> How to confirm it works (success criteria). In a hurry? Jump to Quick fix.

Why this happens

Ollama is a single binary plus a model directory. ollama serve starts the service (default port 11434), ollama pull <model> downloads models, ollama run <model> runs them interactively, and the API is OpenAI-compatible (/api/version etc.). An NVIDIA GPU enables CUDA automatically; without a GPU it falls back to CPU inference (usable but slow). Two key mechanisms:

• OLLAMA_MODELS: model directory env var (default ~/.ollama/models, can point to a large partition — single model files are 4~70GB);
• GGUF import: download a GGUF file from ModelScope or similar, then import it with ollama create, bypassing the official registry — this is the right way to pull models on mainland networks.

Where it writes: ~/.ollama/ (models and config; OLLAMA_MODELS can relocate); binary under /usr/local (verified path on this machine).

Step by step

Step 1: Install the binary

Why: the official install.sh fetches its script from ollama.com, which can be very slow on direct mainland connections (inferred). The verified path is gh-proxy download of the GitHub Release ollama-linux-amd64.tar.zst (1.42GB) extracted to /usr/local. Needs tar --zstd (zstd 1.5.7, apt install) or the zstd tool. The download is large — watch stability when switching between campus network and proxy.

curl -fsSL -o /tmp/ollama.tar.zst https://gh-proxy.com/https://github.com/ollama/ollama/releases/latest/download/ollama-linux-amd64.tar.zst
tar --zstd -xf /tmp/ollama.tar.zst -C /usr/local
ollama --version

Expected output: ollama version 0.33.2 (verified 2026-08-29) After install, confirm:

• ollama --version prints ollama version 0.33.2;
• ollama list runs (empty model table exits 0).

Stuck? How to recover:

• tar: unrecognized option --zstd → run apt-get install -y zstd first (or use a system-provided zstd tool);
• download interrupted → resume with curl -C -; before downloading 1.42GB, check disk (df -h /) and network stability;
• Windows users → use the official installer (WSL2 recommended, same path as Linux); macOS uses the official dmg (Apple Silicon native).

Step 2: Start the service + import the first model

Why: ollama serve starts the service (default port 11434). Model pulls go through the official registry, which times out on mainland networks (verified) — the fallback is ModelScope GGUF download + ollama create import (verified 2026-08-29: qwen2.5-0.5b-instruct-q4_k_m 491MB download in 58.5s + create success + CPU smoke passed, evidence local repro log 2026-08-29).

OLLAMA_MODELS=~/.ollama/models ollama serve

Expected output: server config log normal (ssh key generation, blobs: 0; default port 11434; verified 2026-08-29)

curl -s http://127.0.0.1:11434/api/version

Expected output: {"version":"0.33.2"} (verified 2026-08-29) After install, confirm:

• serve log is normal with no errors;
• /api/version returns the version number;
• ollama list shows the imported model (once one exists).

Stuck? How to recover:

• serve won't start / port taken → if 11434 is busy, change OLLAMA_HOST (e.g. OLLAMA_HOST=127.0.0.1:11435);
• ollama pull times out → the official registry times out on mainland networks (verified; serve log shows cloud model cache pull failure on ollama.com:443); use the ModelScope GGUF + create import path;
• model directory out of space → point OLLAMA_MODELS at a large partition (models are 4~70GB; check df -h first).

How to confirm it works

Three smoke checks; only all-green counts as installed:

ollama --version

Expected output: ollama version 0.33.2

curl -s http://127.0.0.1:11434/api/version

Expected output: {"version":"0.33.2"}

ollama list

Expected output: NAME ID SIZE MODIFIED (rows appear once a model is imported)

• Service is up: curl /api/version returns a version number, meaning serve is listening on 11434;
• Model is ready: ollama list shows model rows;
• GGUF-imported models: ollama run qwen2.5:0.5b "你好" produces text (CPU smoke, verified 2026-08-29).

Other environments

• Windows: use the official Windows installer; inside WSL2, follow this guide's Linux path (the 2026-08-29 environment is WSL2 Ubuntu — this path is what was verified).
• macOS: official dmg (Apple Silicon native MPS; Intel uses CPU).
• No NVIDIA GPU: falls back to CPU inference, usable but slow (0.5B small-model smoke is fine; large models want a GPU).
• AMD GPU: ROCm assets (inferred, not tested).
• Port taken: OLLAMA_HOST can change it (e.g. OLLAMA_HOST=127.0.0.1:11435).
• Old versions: < 0.30 has poor API/model compatibility; use the latest (this guide verified 0.33.2).

Common scenarios (by user goal)

Scenarios are organized by "what you want to do", not by command surface; every command is verified with a checkable expected output.

Scenario 1: Get the service running

OLLAMA_MODELS=~/.ollama/models ollama serve

Expected output: server config log normal (ssh key generation, blobs: 0; default port 11434; verified 2026-08-29) Two quick checks after it is up:

curl -s http://127.0.0.1:11434/api/version

Expected output: {"version":"0.33.2"} (verified 2026-08-29)

ollama list

Expected output: NAME ID SIZE MODIFIED (empty table exits 0; verified 2026-08-29)

Scenario 2: Pull models on mainland networks (bypass the upstream registry)

ollama pull uses the upstream registry, which times out on mainland direct connections. The verified path = download the GGUF from ModelScope + import with ollama create (verified 2026-08-29: 491MB download in 58.5s + create success + CPU smoke passed; evidence local repro log 2026-08-29):

curl -sL -o /tmp/qwen2.5-0.5b-instruct-q4_k_m.gguf https://modelscope.cn/models/Qwen/Qwen2.5-0.5B-Instruct-GGUF/resolve/master/qwen2.5-0.5b-instruct-q4_k_m.gguf
printf 'FROM /tmp/qwen2.5-0.5b-instruct-q4_k_m.gguf\n' > Modelfile-qwen
ollama create qwen2.5:0.5b -f Modelfile-qwen
ollama run qwen2.5:0.5b "你好"

Expected output: download completes (491MB/58.5s) then create shows copying file -> success -> run replies (verified 2026-08-29) For larger models: search ModelScope for the same-name GGUF (e.g. Qwen2.5-7B-Instruct-GGUF) and replace the URL and model name.

Scenario 3: Connect the open-webui UI (including RAG knowledge base)

Ollama only exposes an API; for a web chat UI + document knowledge base, use combo combo-local-llm-finetune (verified end-to-end 2026-08-30: docker run with 3 env vars + embedding nomic-embed-text 768-dim + knowledge base build and retrieval hit):

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway \
-v open-webui:/app/backend/data --name open-webui --restart always \
-e ENABLE_SIGNUP=true \
-e RAG_EMBEDDING_ENGINE=ollama \
-e RAG_EMBEDDING_MODEL=nomic-embed-text \
ghcr.nju.edu.cn/open-webui/open-webui:main

Expected output: docker run prints a container ID; docker ps shows open-webui Up (verified 2026-08-30; image 7.16GB, ghcr mirror prefix) Verified RAG pitfalls:

• The embedding model name must NOT carry the ollama/ prefix (open-webui passes it through and ollama returns 404); use the bare name nomic-embed-text;
• With an existing data volume, persisted DB config takes priority over env vars - change the embedding engine via admin panel -> Settings -> Retrieval, or POST /api/v1/retrieval/embedding/update;
• For the chat layer use a >=7B function-calling model (0.5B models do not trigger automatic tool retrieval; verified hallucination).

Data and directories

• Models and config: default ~/.ollama/ (snap installs: /var/snap/ollama/common/models); OLLAMA_MODELS can point to a large partition.
• Relocate the model directory (verified 2026-08-30): stop serve -> mv the whole model directory to a new location -> restart with OLLAMA_MODELS=<new path> ollama serve. Verified: after relocating, list/run both work and models are read from the new path; changing the env var for system services (snap/systemd) not individually verified (evidence local repro log 2026-08-30).
• Free disk space (verified 2026-08-30): ollama rm <model> deletes the model and its blobs. Isolated test: after pulling a 274MB model blobs = 274302450 bytes -> after rm back to 0, list converges to empty (evidence local repro log 2026-08-30).
• Single model files are 4~70GB; watch disk usage over time; ollama list to view, ollama rm <model> to free space.

Uninstall & cleanup

note: Remove the binary directory and model directory; stop the serve process first

pkill -f 'ollama serve' 2>/dev/null || true
rm -rf /usr/local/lib/ollama /usr/local/bin/ollama ~/.ollama

Expected output: command surface gone (ollama does not exist) Manual checklist (for full cleanup):

• Model directory (default ~/.ollama/models, or the OLLAMA_MODELS path; models are 4~70GB, confirm before deleting)
• Listening process (default 11434, check with lsof -i :11434)

Dependency warnings

• Version: [0.33.2, 0.33.2] (latest verified 2026-08-29; versions < 0.30 have poor API/model compatibility, use the latest).
• Environment: Linux x86_64 (this path verified); no NVIDIA GPU → CPU inference (usable but slow); AMD GPU → ROCm assets (inferred); Apple Silicon → MPS.
• Disk: models are 4~70GB; run df -h / before installing; OLLAMA_MODELS can point to a large partition.
• Network: binary download via gh-proxy (verified); model pulls from the official registry time out on mainland networks (verified) → ModelScope GGUF import fallback.
• Port: default 11434 (OLLAMA_HOST can change).
• Write surface: ~/.ollama/ (models and config).
• Co-install: with open-webui / LLaMA-Factory judged possible-compatible (write surfaces do not overlap: ollama writes ~/.ollama/; open-webui writes a Docker volume; LLaMA-Factory writes the project directory); combo: site docs.

FAQ

• Q: Models won't pull (mainland network)? A: ollama pull goes through the official registry, verified timeout on mainland networks (serve log shows cloud model cache pull failure on ollama.com:443); fallback = ModelScope GGUF download + ollama create import (verified 2026-08-29: qwen2.5-0.5b-instruct-q4_k_m 491MB download in 58.5s + ollama create success + CPU smoke output normal, evidence local repro log 2026-08-29).
• Q: Do I need a GPU? A: NVIDIA driver + CUDA enables automatically; no GPU falls back to CPU (slow); AMD GPU uses ROCm assets (inferred); Apple Silicon uses MPS.
• Q: Port conflict? A: default 11434, OLLAMA_HOST can change it; if another service occupies 11434, change the env var.
• Q: Where are models stored? A: default ~/.ollama/models (OLLAMA_MODELS can point to a large partition; models are 4~70GB, check df -h first).
• Q: Use with open-webui / LLaMA-Factory? A: judged possible-compatible — write surfaces do not overlap (ollama writes ~/.ollama/ + model dir; open-webui writes a Docker volume; LLaMA-Factory writes the project dir); combo: site docs.
• Q: Where to report problems? A: install/uninstall issues on this marketplace go through the entry feedback channel; project issues go to the author's repository.