How do you run your own private ChatGPT on your own server?
Open-source AI models are free to download and run — so why pay a monthly cloud AI subscription for every seat? This article explains what a private LLM server is, what it can realistically run on normal hardware, how you install one in under an hour, and when hosted AI is the smarter choice.
What does "private AI" actually mean?
When you type a prompt into a public AI service, your text — often containing client names, financial figures or internal processes — is processed on someone else's servers. A private LLM server is different: the AI model runs inside your own server room (or your own virtual machine), and conversations never leave your network.
This is possible because the best-known models are open-weight releases: their trained parameters are published under permissive licenses (MIT, Apache), and anyone can download them — free. DeepSeek, Alibaba (Qwen), Meta (Llama) and Google (Gemma) publish models ranging from 0.5B to hundreds of billions of parameters. The smaller ones run on hardware you probably already own.
What does a private LLM server look like in practice?
A typical setup is surprisingly small:
- The runtime — Ollama (or llama.cpp) loads the model and generates text, optimized for CPU/RAM. Quantized versions (GGUF, Q4) shrink a model to about half a gigabyte per billion parameters.
- The gateway — a small API proxy (e.g. LiteLLM) exposes the same standard API as cloud services, so one key gives access to several models: local open models for privacy, plus cloud models as a fallback for maximum quality.
- The interface — a simple web chat UI behind HTTPS login, so employees use it like a normal company application. A model manager panel lets an admin download a new model with one click — no AI engineer needed.
In our own deployment, the complete stack runs on a modest virtual server with 4 GB of RAM: nginx for HTTPS, Ollama for inference, a gateway for API access, and a chat interface for employees. It serves several small models (1.5B–3B) in parallel, and every request is monitored for response time, tokens and server load.
What can you realistically run?
Model size is a physics problem, not a configuration problem:
| Server | Runs | Typical speed |
|---|---|---|
| 8 GB RAM | 1–3B models (Qwen, Llama small) | fast |
| 16–32 GB RAM | 7–14B models — the practical sweet spot | good |
| 64–128 GB RAM | 30B "Mixture-of-Experts" models, R1-70B | usable |
| 192 GB+ or GPU | 284B-class frontier models | slow on CPU |
A smart trick: Mixture-of-Experts models (e.g. 30B total, 3B active per token) feel like large models but stay fast on normal servers. And when a request truly needs the best model — or one your hardware can't hold — the gateway transparently forwards it to a cloud provider (DeepSeek's API is one of the cheapest; Google's Gemini has a free tier). The employee never notices the difference.
How do you actually install it?
If you can install a printer driver, you can install a private AI server:
- Install the runtime (one installer, no license).
- Download a model (one command, or one click in the model manager).
- Put the gateway in front, with an API key.
- Expose a HTTPS web page for employees — or connect your existing business software through the API.
The whole first installation takes under an hour. Adding a new model later is a single command: ollama pull qwen3:4b — or a click in the web interface, which even supports pulling models directly from HuggingFace (GGUF format).
When does it not make sense?
Honesty matters here. A private server is the wrong answer if you need frontier-class models all day (they need 100+ GB of memory — a data-centre workload), or if nobody can restart a service and read a log file. In those cases, hosted AI — managed for you, possibly on a dedicated server in a controlled environment — is the smarter buy. The architecture is identical in both scenarios, so you can start privately and move to hosted later without changing a thing.
Checklist — is your company ready for a private AI server?
- We know which data must never leave our network (client names, financials, HR).
- We have a server with at least 8 GB free RAM and 100 GB free disk.
- We know which employees should use it (10–30 users is comfortable on one box).
- We have a model strategy: small private models for daily work, a cloud fallback for heavy lifting.
- We've defined who manages models and who answers user questions.
FAQ
- Is open-source AI really free? The models are free to download and run. You pay for hardware, electricity and maintenance — not per token.
- Is my data safe? On a private server, prompts and answers stay inside your network. Cloud fallbacks should be reserved for non-sensitive requests.
- Do I need a GPU? No. CPU-only servers run small and mid-size models well; a GPU adds 5–20× speed for larger models.
- Is it hard to keep up to date? No — updating a model is one command; nothing else changes.
- Can my existing business software use it? Yes. The gateway exposes a standard API, so your software (including FileMaker) can send requests to your own AI.
- Why is this better than a ChatGPT subscription? For the price of a few seats, you get unlimited use for your whole team — and the data never leaves the building.