Lessons

Running a Local AI Coding Assistant on Linux with Ollama

John Coffey August 17, 2026 8 min read

Every editor seems to have an AI assistant bolted on now, and most of them want to ship your code to somebody else’s cloud to do it. If that’s a non-starter for you — client contracts that forbid it, an air-gapped box, or just not wanting to hand another subscription fee to a company every month — running the model locally with Ollama gets you most of the workflow without any of that.

This lesson goes end to end: installing Ollama, understanding what you’re actually choosing between when you pick a model, wiring it into a terminal assistant (Aider) and an editor (Continue), tuning it for your hardware, and what to do when it doesn’t work. Once the model is downloaded, none of this touches the network.

What You’ll Need

A Linux box with at least 16GB of system RAM if you’re running CPU-only, or a GPU with 8GB+ of VRAM if you want responses that don’t make you question your choices. Nvidia is the path of least resistance; Ollama also supports AMD via ROCm on recent kernels, and Apple Silicon if you’re running this on a Mac instead. None of this requires a GPU — it just requires one to be pleasant.

If you’ve got an Nvidia card, confirm the driver and CUDA are actually visible before you install anything:

nvidia-smi

You want to see your GPU listed with a driver version and available memory. If that command isn’t found, sort out the driver first — Ollama will silently fall back to CPU inference, and you’ll spend an hour thinking Ollama is slow when it’s actually just not using the card.

Installing Ollama

The official install script handles the binary, sets up a systemd service, and configures it to listen on localhost only by default:

curl -fsSL https://ollama.com/install.sh | sh

Confirm it’s running and check the version:

systemctl status ollama
ollama --version

That “listens on localhost only by default” detail matters — Ollama binds to 127.0.0.1:11434 unless you tell it otherwise. If you want to reach it from another machine on your LAN (say, a beefy desktop serving a laptop), you’ll need to override the bind address, and you should understand that this exposes the API with no authentication whatsoever:

sudo systemctl edit ollama

Add this in the override file it opens:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Then restart it:

sudo systemctl restart ollama

If you do this, put it behind a firewall rule scoped to your LAN subnet. Don’t do this on a box with a public IP.

Choosing and Pulling a Model

Ollama’s library has models fine-tuned specifically for code rather than general chat, and that distinction actually matters for this use case. qwen2.5-coder and deepseek-coder-v2 are the two worth starting with — both are trained heavily on code and punch well above their parameter count on coding benchmarks compared to general-purpose models of the same size.

The number after the colon is parameter count, and the tag after that (when present) is the quantization level — how much the model’s weights have been compressed to save memory. A few reference points:

# ~5GB, fits on 8GB VRAM cards, usable on CPU with patience
ollama pull qwen2.5-coder:7b

# ~9GB, wants 12-16GB VRAM, noticeably sharper on multi-file context
ollama pull qwen2.5-coder:14b

# ~19GB, wants 24GB VRAM, this is where local starts genuinely competing with hosted models
ollama pull qwen2.5-coder:32b

Every model in Ollama’s library defaults to a 4-bit quantization (Q4_K_M) unless you specify otherwise — that’s the sweet spot between quality and size for most people, and it’s why a 7B model is ~5GB instead of the ~28GB it’d be at full 16-bit precision. If you have the VRAM to spare, a higher-precision tag like qwen2.5-coder:7b-instruct-q8_0 will be noticeably better than the default 4-bit version at the same parameter count, at roughly double the size.

Sanity-check whichever one you pulled before wiring anything else up:

ollama run qwen2.5-coder:7b "Write a bash function that retries a command up to 3 times with backoff"

If that comes back with something reasonable, the model’s good to go. To see actual performance numbers — tokens per second, load time — add --verbose:

ollama run --verbose qwen2.5-coder:7b "explain what SIGPIPE does"

Anything under roughly 15-20 tokens/second starts to feel sluggish for interactive use. If you’re seeing single digits, you’re almost certainly running on CPU — go back and check nvidia-smi.

Tuning the Context Window

Ollama defaults new models to a fairly small context window (2048 or 4096 tokens depending on the model), which is fine for one-off questions and miserable for coding assistants that need to see several files at once. You can raise this per-session with the API, but the durable way to do it is a custom Modelfile.

Create a file named Modelfile:

FROM qwen2.5-coder:14b
PARAMETER num_ctx 16384

Then build it as a new named model:

ollama create qwen-coder-16k -f Modelfile

Use qwen-coder-16k anywhere you’d reference the base model from here on. Keep in mind VRAM usage scales with context length, not just parameter count — doubling num_ctx can push a model that fit comfortably into one that doesn’t. Watch nvidia-smi while you experiment.

Terminal Workflow: Aider

Aider is a command-line pair-programming tool that edits files directly in your repo and commits as it goes. It talks to Ollama over its OpenAI-compatible local API:

pip install aider-chat

export OLLAMA_API_BASE=http://127.0.0.1:11434
aider --model ollama/qwen2.5-coder:14b

Typing that export line every session gets old fast. Drop your defaults into ~/.aider.conf.yml in your home directory instead:

model: ollama/qwen2.5-coder:14b
edit-format: diff
auto-commits: true

and put the environment variable in your shell profile instead of exporting it by hand each time. From there, running aider in a repo behaves like any other session — /add files into context, describe the change in plain English, and it proposes a diff and commits it. The only difference from using a cloud model is that nothing left your machine.

Editor Workflow: Continue

If you’d rather stay in VS Code or a JetBrains IDE, install the Continue extension and point it at Ollama instead of a cloud provider. Continue actually separates the model used for chat/edits from the model used for inline autocomplete — a smaller, faster model for autocomplete keeps suggestions feeling instant, while a bigger one handles chat where you’re willing to wait a couple seconds. In Continue’s config.json:

{
  "models": [
    {
      "title": "Qwen2.5 Coder 14B (chat)",
      "provider": "ollama",
      "model": "qwen2.5-coder:14b"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen2.5 Coder 1.5B (autocomplete)",
    "provider": "ollama",
    "model": "qwen2.5-coder:1.5b-base"
  }
}

Pull the smaller autocomplete model the same way as any other:

ollama pull qwen2.5-coder:1.5b-base

Restart the editor after saving the config, and both inline chat and autocomplete will run entirely against your local Ollama instance.

Managing Models

Models add up fast — a 14B and a 32B variant of the same family can easily eat 30GB between them. A few commands you’ll actually use:

# See what's installed and how much space it's using
ollama list

# Remove something you're not using
ollama rm qwen2.5-coder:32b

# Update a model to the latest version in the library
ollama pull qwen2.5-coder:14b

Models live under /usr/share/ollama/.ollama/models by default on Linux. If you’re tight on root partition space, that’s the directory to symlink or bind-mount onto a bigger disk before you pull a bunch of 20GB+ models.

Troubleshooting

“CUDA out of memory” or the model loads but responses are painfully slow: the model (plus context) doesn’t fit in VRAM and Ollama is offloading layers to system RAM. Drop to a smaller model, a smaller quantization, or lower num_ctx.

Ollama is definitely running but Aider or Continue can’t reach it: almost always the OLLAMA_API_BASE / provider URL not matching. Confirm the API actually responds:

curl http://127.0.0.1:11434/api/tags

You should get back JSON listing your installed models. If that hangs or refuses the connection, check systemctl status ollama before touching any client config.

Responses are fast but noticeably worse than you expected: double-check you actually pulled a coder-tagged model and not a general chat model, and that num_ctx is high enough that your files aren’t getting silently truncated out of context.

Where This Actually Falls Short

Be honest with yourself about what a 7B or 14B local model is and isn’t good at. Autocomplete, boilerplate, single-function refactors, explaining unfamiliar code — all solid. Reasoning across a dozen files, catching a subtle architectural issue, or holding a long multi-turn conversation about a gnarly bug — a local model this size will start to show its limits well before a frontier cloud model would. A 32B model narrows that gap considerably if you’ve got the VRAM for it, but it’s still a gap.

The tradeoff is worth it for privacy-sensitive work, for staying productive offline, or just for not handing another subscription fee to a company every month. For everything else, at least now you’ve got both options set up and can reach for whichever one the task actually needs.

If you’re interested in the serving side of this rather than the editor-integration side — running a model behind an OpenAI-compatible proxy for multiple users or devices — see our lesson on deploying lightweight language models on embedded Linux with LiteLLM.