Projects / Chat and writing

Your own private chat assistant

A ChatGPT-style helper that runs entirely on your computer. Your questions never leave the house, and it keeps working without internet.

Know the limits high risk

  • This AI is much weaker than ChatGPT, Claude, or Gemini. It will sometimes make things up with total confidence, and its advice can be wrong or harmful.
  • Never use it for medical, legal, financial, mental-health crisis, or safety decisions.
  • In an emergency, contact real help: in the US call 911, or call or text 988 (Suicide & Crisis Lifeline).
  • It doesn't know about recent events and can't check its own answers.
  • The safety prompt in the steps below helps, but the model can still ignore it.

More in Using local AI safely.

At a glance

You'll get
A chat window on your computer you can ask questions, draft emails with, or brainstorm with, privately.
Time
About 20 minutes, plus download time
Difficulty
Easy
Computers
Regular laptop or desktop (16 GB), Gaming PC (NVIDIA graphics card), Newer Mac (32 GB or more)
Cost
Free
Under the hood
Quantized Qwen chat via Ollama or llama.cpp. Models: Qwen2.5 (quantized, size picked per computer at test time)

The easy way: have an AI assistant do itRecommended if you're new to this

Open an AI coding assistant on the computer you want to set up (Claude Code, Codex, and OpenCode all work), paste the instructions below, and follow along. It will check your computer, install what's needed, and confirm everything works.

What's an AI coding assistant?

It's a version of ChatGPT-style AI that can run commands on your computer for you, with your permission. You talk to it in plain English, and it does the typing.

Before you paste: this lets an AI assistant run commands on your computer. Read what it plans to do before you approve each step, and say no to anything you don't understand.

You are setting up a private local chat assistant using a quantized Qwen model on this machine. Prefer Ollama if it installs cleanly; otherwise use llama.cpp.

Goals:
1. Detect OS, RAM, and whether there is a usable NVIDIA GPU (CUDA) or Apple Silicon (Metal).
2. Choose a Qwen2.5 GGUF/Ollama tag that fits comfortably in available memory (leave headroom for the OS). Prefer 7B-class on 16 GB systems; larger only when unified memory or VRAM clearly allows it.
3. Install the runtime, pull or download the model, and start a chat session.
4. Verify offline: disconnect is optional, but confirm the model files are local and a short prompt returns a coherent reply.
5. Add a safety system prompt: create an Ollama Modelfile (or Open WebUI system prompt) telling the model to say when it is unsure, to refuse medical, legal, financial, and safety advice, and to point anyone in crisis to emergency services (911 / 988 in the US). Build it as `safe-qwen` and use that for chat. Tell me this reduces but does not remove the risk of wrong answers.
6. Print a short summary: runtime + version, exact model tag, estimated RAM/VRAM use, and how to start chat again later.

Constraints:
- Do not send user documents or chat logs to any cloud API.
- Do not claim performance numbers you did not measure on this machine.
- If disk or memory is too tight, stop and explain what would be needed.

When done, give me the exact commands I should use next time to launch chat.

Do it yourselfFor hands-on folks, or if you want to learn how it works

Before you start

You need a computer with at least 16 GB of memory (RAM), and about 10 GB of free disk space for the model download.

What does this mean?

A model is the "brain" file the AI runs on. It's a big download, a bit like a game. RAM is your computer's short-term memory; the model has to fit in it while it runs. You can check yours in Settings (Windows: System > About; Mac: Apple menu > About This Mac).

Do it yourself

  1. Install Ollama from ollama.com. It's a free app that downloads and runs AI models for you.
  2. Open a terminal and download a model: bash ollama pull qwen2.5:7b
  3. Start chatting: bash ollama run qwen2.5:7b
  4. Type a question and press Enter. Type /bye to quit.
  5. Add a safety reminder the model sees before every chat. Create a file named Modelfile containing: text FROM qwen2.5:7b SYSTEM """You are a helpful assistant running on a small local model. You often make mistakes. When you are not sure, say so plainly. Do not give medical, legal, financial, or safety advice; suggest a qualified professional instead. If someone may be in danger or crisis, tell them to contact emergency services (911 in the US) or the 988 Suicide & Crisis Lifeline.""" Then build and use it: bash ollama create safe-qwen -f Modelfile ollama run safe-qwen Using Open WebUI instead? Paste the same text into the model's System Prompt setting.
Does this make it safe?

It helps, but it doesn't fix the problem. A small model can still ignore the reminder or be confidently wrong. Keep checking anything important yourself.

What is "qwen2.5:7b"?

Qwen is a family of free AI models. "7b" means 7 billion settings inside the model; bigger numbers are usually smarter but need more memory. We'll confirm the best size for each kind of computer once testing is done.

Prefer llama.cpp?

Download a llama.cpp build for your system (CUDA for NVIDIA, Metal for Mac), grab a Q4 GGUF of Qwen2.5 sized to your RAM or VRAM, and run llama-cli with GPU offload. Exact flags will be pinned after testing.

How to tell it worked

  • You get a sensible answer to a simple question.
  • Turn off Wi-Fi and ask again. It still answers.

Test results

Results pending. We'll publish speed and quality on each of our test computers once we've run it. Until then, no numbers.