[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"$fH0LuFdnH7VEs4XyvHLWknYhJv3Cga2qpSxos0PYblmU":3},{"item":4},{"id":5,"idKnowledge":6,"idDomain":7,"idCluster":7,"kindOverride":8,"slug":9,"title":10,"description":11,"bodyMarkdown":12,"bodyHtml":13,"author":14,"date":15,"createdAt":16,"topics":17,"image":31,"hasDownload":32,"fileName":7,"youtubeId":33},"467","07C3827C-143A-7446-BDBA-E21024876D31","","article","how-do-you-run-your-own-private-chatgpt-on-your-own-server","How do you run your own private ChatGPT on your own server?","Open-source AI models are free to download and run — so why pay a monthly cloud AI subscription for every seat? This article explains what a private LLM server is, what it can realistically run on normal hardware, how you install one in under an hour, and when hosted AI is the smarter choice.","## What does \"private AI\" actually mean?\n\nWhen you type a prompt into a public AI service, your text — often containing client names, financial figures or internal processes — is processed on someone else's servers. A **private LLM server** is different: the AI model runs inside your own server room (or your own virtual machine), and conversations never leave your network.\n\nThis is possible because the best-known models are **open-weight releases**: their trained parameters are published under permissive licenses (MIT, Apache), and anyone can download them — free. DeepSeek, Alibaba (Qwen), Meta (Llama) and Google (Gemma) publish models ranging from 0.5B to hundreds of billions of parameters. The smaller ones run on hardware you probably already own.\n\n## What does a private LLM server look like in practice?\n\nA typical setup is surprisingly small:\n\n- **The runtime** — Ollama (or llama.cpp) loads the model and generates text, optimized for CPU\u002FRAM. Quantized versions (GGUF, Q4) shrink a model to about half a gigabyte per billion parameters.\n- **The gateway** — a small API proxy (e.g. LiteLLM) exposes the same standard API as cloud services, so one key gives access to several models: local open models for privacy, plus cloud models as a fallback for maximum quality.\n- **The interface** — a simple web chat UI behind HTTPS login, so employees use it like a normal company application. A model manager panel lets an admin download a new model with one click — no AI engineer needed.\n\nIn our own deployment, the complete stack runs on a modest virtual server with **4 GB of RAM**: nginx for HTTPS, Ollama for inference, a gateway for API access, and a chat interface for employees. It serves several small models (1.5B–3B) in parallel, and every request is monitored for response time, tokens and server load.\n\n## What can you realistically run?\n\nModel size is a physics problem, not a configuration problem:\n\n| Server | Runs | Typical speed |\n|---|---|---|\n| 8 GB RAM | 1–3B models (Qwen, Llama small) | fast |\n| 16–32 GB RAM | 7–14B models — the practical sweet spot | good |\n| 64–128 GB RAM | 30B \"Mixture-of-Experts\" models, R1-70B | usable |\n| 192 GB+ or GPU | 284B-class frontier models | slow on CPU |\n\nA smart trick: **Mixture-of-Experts** models (e.g. 30B total, 3B active per token) feel like large models but stay fast on normal servers. And when a request truly needs the best model — or one your hardware can't hold — the gateway transparently forwards it to a cloud provider (DeepSeek's API is one of the cheapest; Google's Gemini has a free tier). The employee never notices the difference.\n\n## How do you actually install it?\n\nIf you can install a printer driver, you can install a private AI server:\n\n1. Install the runtime (one installer, no license).\n2. Download a model (one command, or one click in the model manager).\n3. Put the gateway in front, with an API key.\n4. Expose a HTTPS web page for employees — or connect your existing business software through the API.\n\nThe whole first installation takes under an hour. Adding a new model later is a single command: `ollama pull qwen3:4b` — or a click in the web interface, which even supports pulling models directly from HuggingFace (GGUF format).\n\n## When does it *not* make sense?\n\nHonesty matters here. A private server is the wrong answer if you need frontier-class models all day (they need 100+ GB of memory — a data-centre workload), or if nobody can restart a service and read a log file. In those cases, **hosted AI** — managed for you, possibly on a dedicated server in a controlled environment — is the smarter buy. The architecture is identical in both scenarios, so you can start privately and move to hosted later without changing a thing.\n\n## Checklist — is your company ready for a private AI server?\n\n- We know which data must never leave our network (client names, financials, HR).\n- We have a server with at least 8 GB free RAM and 100 GB free disk.\n- We know which employees should use it (10–30 users is comfortable on one box).\n- We have a model strategy: small private models for daily work, a cloud fallback for heavy lifting.\n- We've defined who manages models and who answers user questions.\n\n## FAQ\n\n- **Is open-source AI really free?** The models are free to download and run. You pay for hardware, electricity and maintenance — not per token.\n- **Is my data safe?** On a private server, prompts and answers stay inside your network. Cloud fallbacks should be reserved for non-sensitive requests.\n- **Do I need a GPU?** No. CPU-only servers run small and mid-size models well; a GPU adds 5–20× speed for larger models.\n- **Is it hard to keep up to date?** No — updating a model is one command; nothing else changes.\n- **Can my existing business software use it?** Yes. The gateway exposes a standard API, so your software (including FileMaker) can send requests to your own AI.\n- **Why is this better than a ChatGPT subscription?** For the price of a few seats, you get unlimited use for your whole team — and the data never leaves the building.","\u003Ch2>What does "private AI" actually mean?\u003C\u002Fh2>\n\u003Cp>When you type a prompt into a public AI service, your text — often containing client names, financial figures or internal processes — is processed on someone else's servers. A \u003Cstrong>private LLM server\u003C\u002Fstrong> is different: the AI model runs inside your own server room (or your own virtual machine), and conversations never leave your network.\u003C\u002Fp>\n\u003Cp>This is possible because the best-known models are \u003Cstrong>open-weight releases\u003C\u002Fstrong>: their trained parameters are published under permissive licenses (MIT, Apache), and anyone can download them — free. DeepSeek, Alibaba (Qwen), Meta (Llama) and Google (Gemma) publish models ranging from 0.5B to hundreds of billions of parameters. The smaller ones run on hardware you probably already own.\u003C\u002Fp>\n\u003Ch2>What does a private LLM server look like in practice?\u003C\u002Fh2>\n\u003Cp>A typical setup is surprisingly small:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>The runtime\u003C\u002Fstrong> — Ollama (or llama.cpp) loads the model and generates text, optimized for CPU\u002FRAM. Quantized versions (GGUF, Q4) shrink a model to about half a gigabyte per billion parameters.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>The gateway\u003C\u002Fstrong> — a small API proxy (e.g. LiteLLM) exposes the same standard API as cloud services, so one key gives access to several models: local open models for privacy, plus cloud models as a fallback for maximum quality.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>The interface\u003C\u002Fstrong> — a simple web chat UI behind HTTPS login, so employees use it like a normal company application. A model manager panel lets an admin download a new model with one click — no AI engineer needed.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>In our own deployment, the complete stack runs on a modest virtual server with \u003Cstrong>4 GB of RAM\u003C\u002Fstrong>: nginx for HTTPS, Ollama for inference, a gateway for API access, and a chat interface for employees. It serves several small models (1.5B–3B) in parallel, and every request is monitored for response time, tokens and server load.\u003C\u002Fp>\n\u003Ch2>What can you realistically run?\u003C\u002Fh2>\n\u003Cp>Model size is a physics problem, not a configuration problem:\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Server\u003C\u002Fth>\n\u003Cth>Runs\u003C\u002Fth>\n\u003Cth>Typical speed\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>8 GB RAM\u003C\u002Ftd>\n\u003Ctd>1–3B models (Qwen, Llama small)\u003C\u002Ftd>\n\u003Ctd>fast\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>16–32 GB RAM\u003C\u002Ftd>\n\u003Ctd>7–14B models — the practical sweet spot\u003C\u002Ftd>\n\u003Ctd>good\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>64–128 GB RAM\u003C\u002Ftd>\n\u003Ctd>30B "Mixture-of-Experts" models, R1-70B\u003C\u002Ftd>\n\u003Ctd>usable\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>192 GB+ or GPU\u003C\u002Ftd>\n\u003Ctd>284B-class frontier models\u003C\u002Ftd>\n\u003Ctd>slow on CPU\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\u003C\u002Ftable>\n\u003Cp>A smart trick: \u003Cstrong>Mixture-of-Experts\u003C\u002Fstrong> models (e.g. 30B total, 3B active per token) feel like large models but stay fast on normal servers. And when a request truly needs the best model — or one your hardware can't hold — the gateway transparently forwards it to a cloud provider (DeepSeek's API is one of the cheapest; Google's Gemini has a free tier). The employee never notices the difference.\u003C\u002Fp>\n\u003Ch2>How do you actually install it?\u003C\u002Fh2>\n\u003Cp>If you can install a printer driver, you can install a private AI server:\u003C\u002Fp>\n\u003Col>\n\u003Cli>Install the runtime (one installer, no license).\u003C\u002Fli>\n\u003Cli>Download a model (one command, or one click in the model manager).\u003C\u002Fli>\n\u003Cli>Put the gateway in front, with an API key.\u003C\u002Fli>\n\u003Cli>Expose a HTTPS web page for employees — or connect your existing business software through the API.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>The whole first installation takes under an hour. Adding a new model later is a single command: \u003Ccode>ollama pull qwen3:4b\u003C\u002Fcode> — or a click in the web interface, which even supports pulling models directly from HuggingFace (GGUF format).\u003C\u002Fp>\n\u003Ch2>When does it \u003Cem>not\u003C\u002Fem> make sense?\u003C\u002Fh2>\n\u003Cp>Honesty matters here. A private server is the wrong answer if you need frontier-class models all day (they need 100+ GB of memory — a data-centre workload), or if nobody can restart a service and read a log file. In those cases, \u003Cstrong>hosted AI\u003C\u002Fstrong> — managed for you, possibly on a dedicated server in a controlled environment — is the smarter buy. The architecture is identical in both scenarios, so you can start privately and move to hosted later without changing a thing.\u003C\u002Fp>\n\u003Ch2>Checklist — is your company ready for a private AI server?\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>We know which data must never leave our network (client names, financials, HR).\u003C\u002Fli>\n\u003Cli>We have a server with at least 8 GB free RAM and 100 GB free disk.\u003C\u002Fli>\n\u003Cli>We know which employees should use it (10–30 users is comfortable on one box).\u003C\u002Fli>\n\u003Cli>We have a model strategy: small private models for daily work, a cloud fallback for heavy lifting.\u003C\u002Fli>\n\u003Cli>We've defined who manages models and who answers user questions.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2>FAQ\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>\u003Cstrong>Is open-source AI really free?\u003C\u002Fstrong> The models are free to download and run. You pay for hardware, electricity and maintenance — not per token.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Is my data safe?\u003C\u002Fstrong> On a private server, prompts and answers stay inside your network. Cloud fallbacks should be reserved for non-sensitive requests.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Do I need a GPU?\u003C\u002Fstrong> No. CPU-only servers run small and mid-size models well; a GPU adds 5–20× speed for larger models.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Is it hard to keep up to date?\u003C\u002Fstrong> No — updating a model is one command; nothing else changes.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Can my existing business software use it?\u003C\u002Fstrong> Yes. The gateway exposes a standard API, so your software (including FileMaker) can send requests to your own AI.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Why is this better than a ChatGPT subscription?\u003C\u002Fstrong> For the price of a few seats, you get unlimited use for your whole team — and the data never leaves the building.\u003C\u002Fli>\n\u003C\u002Ful>\n","Shubham","2026-08-14",1786696551000,[18,19,20,21,22,23,24,25,26,27,28,29,30],"FileMaker","AI","private LLM server","self-hosted LLM","Ollama","open-weight models","GGUF quantization","local AI inference","MoE models","AI gateway","on-prem AI","AI-ready business","business data privacy","\u002Fapi\u002Fknowledge\u002Fimage\u002F467\u002F?v=f47a682161ec",false,null]