How to Run an LLM Locally on Your Own Computer

You’d like to use AI on client contracts, patient notes or unreleased financials, but your company won’t let that data leave the building. Or you’re tired of rate limits and per-token bills while you experiment. Either way, the question is the same: can you just run the model yourself?

You can. Free tools now make it possible to download an open-weight language model and run it on a normal laptop or desktop, with no internet connection needed once the model is downloaded. This guide covers the hardware you need, a step-by-step setup and an honest look at when local AI beats the cloud, and when it doesn’t.

What running an LLM locally means

A large language model is a file of learned numbers (weights) plus software that uses them to generate text. When you use a cloud chatbot, both live on the provider’s servers. Running locally means the weights sit on your drive and your own processor or graphics card does the work.

You can only run models whose weights are published, often called open-weight models. Well-known families include Meta’s Llama, Google’s Gemma, Mistral’s models, Alibaba’s Qwen and Microsoft’s Phi. The frontier models behind the biggest commercial chat apps generally are not downloadable. Our comparison of open-source vs closed AI models covers the trade-offs, and small language models and on-device AI explains why smaller models have become so capable.

Check each model’s license before business use. Open-weight licenses vary, and some include acceptable use rules or conditions for very large companies.

When local beats cloud, and when it doesn’t

Local is a strong fit for:

• Sensitive documents you’re not allowed to send to third parties
• Offline work, such as travel, field sites or air-gapped environments
• Heavy experimentation where API costs would add up
• Learning how models, prompts and parameters behave
• Simple, repetitive tasks like summarizing, classifying or reformatting text

Cloud is usually better for complex reasoning, long documents, coding help on large projects and anything many people use at once. Many teams use both: local for private drafts and cloud for heavier work.

Key takeaway: Local AI trades some quality and convenience for privacy and control. Start with a small model on the computer you already have, and only buy hardware once you know what tasks you need it for.

Hardware: what you actually need

The limiting factor is memory. The whole model (or most of it) has to fit in your GPU’s video memory (VRAM), or in system RAM on machines with unified memory, such as Apple silicon Macs. If it doesn’t fit, it either won’t load or will run very slowly.

A quick way to estimate memory

Models are usually quantized, which means their numbers are stored at lower precision to save memory. The llama.cpp project, which powers many local tools, supports quantization from 1.5-bit up to 8-bit. 4-bit is a common balance between quality and size.

A rough rule: memory for the weights is about the parameter count multiplied by the bytes per parameter. At 4-bit, that’s roughly half a byte per parameter, plus extra room for the conversation (the context window) and the software itself. That gives these approximate starting points:

These are estimates. Actual needs depend on the quantization type, the context length you set and the tool you use.

Vendor requirements to check

• LM Studio recommends 16 GB or more of RAM on Mac and Windows, requires Apple silicon on macOS (Intel Macs aren’t supported) and recommends at least 4 GB of dedicated VRAM for GPU use on Windows.
• Ollama supports NVIDIA GPUs with compute capability 5.0 or higher, AMD GPUs through ROCm and Vulkan, and Apple GPUs through Metal. It can also run on the CPU alone, just more slowly.

If you’re shopping for a new machine, our guide to what an AI PC is explains what NPUs and GPUs actually do for local AI.

Step-by-step: run your first local model

Two beginner-friendly options are Ollama (command line plus a local API) and LM Studio (a desktop app with a chat window). Both are free to download. Here’s the Ollama path.

1. Install Ollama. Download the installer for macOS, Windows or Linux from ollama.com, or use the install command shown in its documentation.
2. Open a terminal (Terminal on Mac, PowerShell on Windows).
3. Run a small model. Type ollama run followed by a model name from the Ollama model library, choosing a small model (a few billion parameters) for your first test. The first run downloads the model, which can be several gigabytes.
4. Chat. Type a question at the prompt. Responses appear as they’re generated. Type /bye to exit.
5. Check where it’s running. The ollama ps command shows whether the model is on your GPU, CPU or both. If it’s CPU-only and slow, try a smaller model.
6. Try a bigger model once things work, and compare quality and speed on a real task.

Prefer a graphical app? In LM Studio, open the model search, download a model the app marks as compatible with your hardware, then load it and start chatting. It shows the file size and quantization of each download so you can pick one that fits your memory.

Connect it to your own tools

Local runners can act as a mini API server on your own computer. Ollama listens on port 11434 at localhost by default and offers a subset of the OpenAI API at a /v1/ address, so many apps and scripts built for the OpenAI API can point at your local model by changing the base URL. llama.cpp’s server offers an OpenAI-compatible API too.

That lets you:

• Use a local model inside a code editor extension or chat interface
• Build private document search with retrieval-augmented generation
• Prototype features locally before switching to a cloud API (see how to use an LLM API)

Privacy and security tips for local AI

Local doesn’t automatically mean secure. Keep these points in mind:

• Keep the server private. Ollama binds to 127.0.0.1 by default, which means only your machine can reach it. Changing the OLLAMA_HOST setting to expose it on your network lets other devices connect, so only do that behind a firewall with access controls.
• Download models from trusted sources, such as the official model library or the model maker’s verified page.
• Watch for cloud features. Some local tools also offer optional cloud-hosted models. Ollama’s FAQ says it doesn’t see your prompts when you run locally; make sure you know which mode you’re using.
• Protect the device. Prompts, outputs and documents still sit on your drive, so use disk encryption and normal access controls.
• Follow company policy. Even local tools may need approval. Our guide on whether it’s safe to put company data into AI lists questions to ask.

Troubleshooting common problems

• Very slow responses: the model is likely running on the CPU or spilling out of VRAM. Use a smaller model or a lower-bit quantization.
• Model won’t load: not enough memory. Close other apps or pick a smaller download.
• It forgets earlier parts of a long document: the context window is too small. Ollama’s FAQ lists a default context length of 4,096 tokens, which you can raise with a setting, at the cost of more memory. Our guide to tokens and context windows explains why.
• Answers are weaker than a cloud chatbot: expected for small models. Be more specific in prompts, give examples, or move up a model size.

Conclusion

Running an LLM locally is now a 15-minute job: install a runner, download a small open-weight model and start chatting. It’s the simplest way to use AI on private data, work offline and experiment without a meter running. Start with the computer you have, test on real tasks, and decide from results whether you need bigger hardware or a cloud model for the heavy lifting.

Sources

• Ollama on GitHub — installation per operating system and local API on port 11434
• Ollama docs: FAQ — local privacy, default 127.0.0.1 binding, OLLAMA_HOST, default context length and ollama ps
• Ollama docs: Hardware support (GPU) — NVIDIA, AMD and Apple Metal GPU requirements
• Ollama docs: OpenAI compatibility — subset of the OpenAI API at localhost:11434/v1
• LM Studio docs: System requirements — RAM, VRAM and OS requirements
• llama.cpp on GitHub — supported hardware backends, 1.5-bit to 8-bit quantization and OpenAI-compatible server