Local LLM: Running AI Without the Cloud — Hardware, Models, Tools and Limits

What Is a Local LLM?

A local LLM is a large language model that doesn’t run in a provider’s cloud but on hardware you control: a laptop, a workstation, a server in your own data centre, or one hosted with a European provider. This is possible thanks to open-weight models — Meta (Llama), Alibaba (Qwen), Google (Gemma), Mistral and others make their models available for download. For what an LLM is in the first place, see our article on what LLM means.

The difference from ChatGPT and similar tools is fundamental: your inputs never leave the building. There’s no data processing agreement (DPA), because there’s no processor; no token costs, because you own the compute; no dependence on price changes or models being discontinued. In exchange, you take on the hardware, setup, operation and updates yourself.

What Are the Advantages of a Local LLM?

• ▸Privacy and confidentiality: patient records, client files, engineering drawings, source code — it all stays on your own network. For professionals bound by confidentiality and regulated industries, this is often the only acceptable option.
• ▸Cost structure: a one-off hardware purchase instead of ongoing token costs. At high volumes — classifying tens of thousands of documents, say — it pays for itself within months.
• ▸Independence: no provider can shut the model down, raise prices or change the terms of use. A model you’ve tested once will behave the same way tomorrow as it does today.
• ▸Offline capability: runs without an internet connection — relevant for field work, air-gapped networks or production environments.
• ▸Customisability: open models can be fine-tuned and wired up to your own tools without worrying about provider limits.

What Hardware Do You Need for a Local LLM?

The decisive figure is graphics memory (VRAM) — or, on Apple Silicon Macs and AMD Ryzen AI systems, the shared system memory. The model has to fit into it, or it slows to a crawl. Rule of thumb: an 8-billion-parameter model needs around 5–6 GB in compressed form (4-bit quantisation), a 30B model around 18–20 GB, and a 70B model around 40–45 GB.

Do you absolutely need a dedicated GPU? No — small models run on any modern CPU, just more slowly. For team use, though, a GPU is essential: nobody puts up with response times of several seconds per sentence day to day. A good middle ground for SMEs is a server with a 24 GB card — for €3,000 to €5,000 you get hardware that can serve a 30B model to an entire team.

Which Local LLM Models Are Available?

Which model is best depends on the task and the hardware. As all-rounders for German-language office work, Qwen3 (8B for single workstations, 30B-A3B for teams) and Gemma 3 27B have proven themselves in 2026; for coding tasks, Qwen’s Coder variants; and for maximum quality where the hardware allows it, Llama 70B. All the models listed are free to download — “free LLM models” are the norm, not the exception. Just check each licence for commercial-use conditions.

Ollama, LM Studio, vLLM and Open WebUI Compared

• ▸Ollama: a command-line tool and server, ready to go in a minute with “ollama run qwen3:8b”. Provides an API that plugs into n8n, your own applications or RAG systems. The default choice for 90% of business setups — runs on Windows, macOS and Linux.
• ▸LM Studio: a graphical desktop app with a model catalogue, chat interface and local API server. Ideal for beginners and single workstations.
• ▸llama.cpp: the engine underneath both, usable directly for maximum control and unusual hardware.
• ▸vLLM: a high-performance server for many concurrent users — the choice for production deployment on professional GPUs.
• ▸Open WebUI: a ChatGPT-style interface for teams with user management, document upload and RAG — together with Ollama, the de facto standard package for privacy-strict setups in German-speaking countries.

How to Set Up a Local LLM

• ▸1. Check your hardware: work out your VRAM or system memory and derive the maximum model size from it (see table).
• ▸2. Install Ollama: download the installer from ollama.com, then run “ollama run qwen3:8b” in the terminal — the model loads and the chat starts.
• ▸3. Add a front end: start Open WebUI via Docker and connect it to Ollama; set up user accounts.
• ▸4. Connect your documents: upload documents in Open WebUI, or build your own RAG system with pgvector and n8n so the model answers from company knowledge.
• ▸5. Secure and measure: restrict access to the internal network or VPN, enable logging, and compare answer quality against a cloud model using 30 real questions.

Can Local LLMs Keep Up With Cloud Models?

For plenty of everyday tasks, yes: summarising, rephrasing, classifying, pulling information from documents, answering questions about supplied context — here, a good 30B model from 2026 matches the top cloud models from a year or two ago, and in practice that’s usually enough. For complex reasoning, long multi-step tasks, agentic workflows and demanding code, today’s leading models from Anthropic, OpenAI and Google remain noticeably ahead. The gap is closing, but it hasn’t disappeared.

So the choice is rarely either/or. A proven pattern: a local model for anything that touches sensitive data or arrives in high volume, and a cloud model with a data processing agreement (DPA) for the demanding one-off cases. An abstraction layer (in n8n, for example) keeps the model swappable.

Are Local LLMs GDPR-Compliant?

A local model is the most privacy-friendly way to deploy AI: there’s no transfer to third parties, no third-country transfer, and no use for training. GDPR still applies, though — you’re still processing personal data and need a legal basis, purpose limitation, a deletion concept and documentation. What falls away is the most complicated part: the data processing agreement and the third-country question. For medical practices, law firms and tax advisers bound by professional confidentiality, that’s often the decisive advantage.

Limits and Effort: An Honest Look

• ▸Running it is work: updates, model changes, monitoring, GPU drivers. Somebody has to own it.
• ▸Scaling: ten concurrent users on a consumer GPU get slow. Bigger teams need vLLM and professional hardware.
• ▸Power costs: a GPU under load draws 300–450 watts. Running continuously, that’s a line item that belongs in the calculation.
• ▸No support: with open models there’s no provider who’s liable or on hand to help — community and service providers fill that gap instead.
• ▸Quality gap: if you need top-tier reasoning quality, there’s no getting around cloud models.

Our Verdict

Local LLMs are no longer a hobbyist solution in 2026. With a 24 GB GPU, Ollama and Open WebUI, an SME can get an AI assistant for a few thousand euros that understands company documents, keeps data in-house, and incurs no ongoing token costs. The cloud remains the tool of choice for the hardest tasks — the smart approach is to combine both.