Introduction


Running large language models locally has become practical thanks to quantization techniques, efficient inference engines, and a thriving open-source ecosystem. Whether for privacy, cost savings, or offline availability, local LLMs offer a compelling alternative to cloud APIs for many workloads. This guide covers the two most popular local LLM platforms — Ollama and LM Studio.


Ollama


[Ollama](https://ollama.ai) is the most popular tool for running LLMs locally, known for its simplicity and command-line focus.


Installation



# macOS

brew install ollama



# Linux

curl -fsSL https://ollama.ai/install.sh | sh



# Windows

# Download from https://ollama.ai/download

Getting Started


Ollama makes running a model a single command:



# Pull and run a model

ollama run llama3.2:3b



# List available models

ollama list



# Pull a specific model without running

ollama pull mistral:7b

Popular Models for Ollama


| Model | Size | RAM Required | Best For |

|-------|------|-------------|----------|

| Llama 3.2 3B | 2.0 GB | 4 GB | Fast responses, simple tasks |

| Llama 3.1 8B | 4.7 GB | 8 GB | General purpose Q&A |

| Mistral 7B | 4.1 GB | 8 GB | Code, reasoning, instruction following |

| Qwen2.5 7B | 4.8 GB | 8 GB | Strong multilingual, coding |

| Mixtral 8x7B | 26 GB | 48 GB | High quality, close to GPT-3.5 |

| DeepSeek-R1 7B | 4.5 GB | 8 GB | Strong reasoning, step-by-step |


Using Ollama Programmatically


Ollama provides a REST API at `http://localhost:11434`:



import requests



response = requests.post("http://localhost:11434/api/generate", json={

    "model": "llama3.2:3b",

    "prompt": "Explain quantum computing in three sentences.",

    "stream": False

})



print(response.json()["response"])

Or use the official Python library:



import ollama



response = ollama.chat(model="llama3.2:3b", messages=[

    {"role": "user", "content": "What is the capital of France?"}

])

print(response["message"]["content"])

Custom Modelfiles


Create custom models with system prompts and parameters:



FROM llama3.1:8b



# Set system prompt

SYSTEM "You are a helpful coding assistant. Provide concise code examples."



# Configure parameters

PARAMETER temperature 0.3

PARAMETER top_p 0.9

Build and run:



ollama create my-coding-assistant -f Modelfile

ollama run my-coding-assistant

LM Studio


[LM Studio](https://lmstudio.ai) is a GUI-focused alternative that excels for users who prefer visual interfaces and easy model browsing.


Key Features


  • **Built-in model browser**: Search and download models from Hugging Face
  • **GUI chat interface**: Familiar ChatGPT-like experience
  • **Local API server**: OpenAI-compatible API endpoint
  • **Model configuration**: Easy sliders for context length, GPU offloading, and temperature
  • **Multi-model support**: Load multiple models and switch between them

    • Setup


  • Download from [lmstudio.ai](https://lmstudio.ai)

    • 2. Open the app and browse the model catalog

    3. Download a model (start with Llama 3.2 3B or Mistral 7B)

    4. Load the model and start chatting


    API Server


    LM Studio can serve models via an OpenAI-compatible API:


    
http://localhost:1234/v1/chat/completions

    This means any tool that works with OpenAI's API can use your local model by changing the base URL:


    
from openai import OpenAI



client = OpenAI(

    base_url="http://localhost:1234/v1",

    api_key="not-needed"

)



response = client.chat.completions.create(

    model="local-model",

    messages=[{"role": "user", "content": "Hello!"}]

)

    Performance Optimization


    Quantization Levels


    Quantization reduces model size at the cost of some accuracy:


  • **Q4_K_M**: Best balance of quality and size (4-bit, recommended)
  • **Q5_K_M**: Higher quality, larger (5-bit, use if you have RAM headroom)
  • **Q8_0**: Near-full quality, 2x RAM requirement
  • **Q2_K**: Minimal RAM, noticeable quality loss

    • Rule of thumb: each quantization step roughly doubles model size but improves quality marginally. Q4_K_M is the sweet spot.


    GPU Acceleration


    Both Ollama and LM Studio support GPU acceleration via CUDA (NVIDIA), Metal (Apple Silicon), or Vulkan (AMD):


    
# Ollama uses Metal automatically on Apple Silicon

# For NVIDIA, install CUDA and Ollama detects it



# Check which device is being used

ollama run llama3.2:3b --verbose

    Context Window


    Larger context windows consume more memory. A 128K context with Q4_K_M requires approximately:


  • 7B model: ~8 GB total
  • 13B model: ~14 GB total
  • 70B model: ~48 GB total

    • Use Cases for Local LLMs


  • **Privacy-sensitive data**: Medical records, legal documents, personal information
  • **Offline environments**: Air-gapped systems, travel, remote locations
  • **Cost-sensitive workloads**: High-volume batch processing
  • **Experimentation**: Rapid testing of different models without API costs
  • **Latency-critical applications**: No network calls for inference

    • Conclusion


    Running LLMs locally is easier than ever with Ollama and LM Studio. Ollama offers command-line simplicity and a rich set of pre-built models. LM Studio provides a polished GUI and OpenAI-compatible API. Start with a 7B model at Q4_K_M quantization on a machine with 8-16 GB of RAM, and scale up as your needs grow. Local LLMs won't replace cloud APIs for every use case, but they are an essential tool in the AI practitioner's toolkit.