Tutorial: Running Local LLM Councils with Ollama

Why Local Councils?

Local LLM councils offer:

• Complete privacy: No data leaves your machine
• Zero API costs: After hardware investment
• No rate limits: Unlimited queries
• Offline capability: Works without internet

Prerequisites

• 16GB+ RAM (32GB recommended)
• GPU with 8GB+ VRAM (optional but faster)
• Docker or native install capability

Step 1: Install Ollama

macOS/Linux

curl -fsSL https://ollama.com/install.sh | sh

Windows

Download from ollama.com and run the installer.

Docker

docker pull ollama/ollama
docker run -d -v ollama:/root/.ollama -p 11434:11434 ollama/ollama

Step 2: Pull Models

For a balanced local council:

# Reasoning model
ollama pull llama3.2:3b

# Coding model
ollama pull deepseek-coder:6.7b

# Fast model
ollama pull mistral:7b

# Optional: Larger model
ollama pull llama3.1:70b

Step 3: Verify Installation

ollama list
ollama run llama3.2:3b "Hello, how are you?"

Step 4: Configure SPRAPP

1. Open SPRAPP settings
2. Navigate to Local Models
3. Enable Ollama integration
4. Add your local models:
   • llama3.2:3b
   • deepseek-coder:6.7b
   • mistral:7b

Step 5: Create Local Council

{
  "name": "Local Council",
  "models": [
    "ollama:llama3.2:3b",
    "ollama:deepseek-coder:6.7b",
    "ollama:mistral:7b"
  ],
  "mode": "consensus",
  "local_only": true
}

Performance Tips

Hardware Optimization

• Use GPU acceleration when available
• Allocate sufficient RAM
• Consider quantized models (q4, q5)

Model Selection

• 7B models: Good balance of speed/quality
• 3B models: Fast, lower quality
• 70B models: High quality, slow

Quantization

# Pull quantized version
ollama pull llama3.2:3b-q4

Hybrid Approach

Combine local and cloud:

• Local: Privacy-sensitive queries
• Cloud: Complex reasoning needs
• Hybrid: Local fan-out, cloud synthesis

Limitations

• Local models trail SOTA by 6-12 months
• Hardware requirements significant
• Setup more complex than cloud
• No real-time information

When to Use Local

• Regulated industries (healthcare, finance)
• Trade secret protection
• Offline requirements
• High-volume, cost-sensitive applications

Your private LLM council is ready!