Fine-Tuning for Councils: Customizing Models in Multi-Model AI Systems

To Fine-Tune or Not?

Fine-tuning can improve council performance but adds complexity. Here's how to decide.

When to Fine-Tune

Clear Signals

• Prompt engineering plateaued
• Domain-specific terminology
• Consistent error patterns
• Large evaluation gap

Good Candidates

• Medical/clinical AI
• Legal document analysis
• Technical domain expertise
• Company-specific knowledge

Not Worth It

• General-purpose use
• Rare edge cases
• Rapidly changing domains
• Small performance gaps

Fine-Tuning Approaches

Supervised Fine-Tuning (SFT)

Training data: Input-output pairs
Process: Update model weights
Best for: Style, format, basic knowledge

Reinforcement Learning from Human Feedback (RLHF)

Training data: Human preferences
Process: Reward model + PPO
Best for: Alignment, helpfulness

Direct Preference Optimization (DPO)

Training data: Preference pairs
Process: Direct optimization
Best for: Simpler alignment

Retrieval-Augmented Fine-Tuning (RAFT)

Training data: Documents + queries
Process: Domain injection
Best for: Knowledge-intensive tasks

Council-Specific Considerations

Fine-Tune Which Models?

Option 1: All Models

• Maximum customization
• Highest cost
• Maintenance burden

Option 2: Synthesis Model Only

• Consistent output style
• Moderate effort
• Good ROI

Option 3: Specialist Models Only

• Domain-specific improvements
• Targeted investment
• Best balance

Training Data Requirements

Approach	Examples Needed	Quality Requirement
SFT	1,000-10,000	High
RLHF	10,000+ preferences	Very high
DPO	1,000-5,000 pairs	High
RAFT	Documents + 100 queries	Medium

Fine-Tuning Workflow

1. Data Collection

Sources:
- Production logs
- Expert annotations
- Synthetic generation
- Public datasets

2. Data Preparation

Tasks:
- Clean and validate
- Format conversion
- Train/val/test split
- Quality filtering

3. Training

Options:
- Self-hosted (HuggingFace, Axolotl)
- Cloud (OpenAI, Together, Fireworks)
- Managed (Anthropic, some limited)

4. Evaluation

Compare:
- Base vs. fine-tuned
- On held-out test set
- On real-world queries
- A/B test in production

5. Deployment

Integrate into council:
- Replace base model
- Compare with alternatives
- Monitor performance

Alternatives to Fine-Tuning

Prompt Engineering

Start here:

• Cheapest approach
• Fastest iteration
• May be sufficient

RAG (Retrieval-Augmented Generation)

Better for:
- Large knowledge bases
- Frequently updated info
- Source attribution needed

Few-Shot Learning

Better for:
- Limited examples
- Quick adaptation
- Testing concepts

Model Selection

Sometimes the answer:
- Different model better suited
- Prompt model with examples
- Council approach handles diversity

Cost-Benefit Analysis

Fine-Tuning Costs

Data collection: $5,000-$50,000
Training compute: $500-$5,000
Infrastructure: $1,000-$10,000
Maintenance: $500-$2,000/month

Expected Benefits

Quality improvement: 5-20%
Latency: Same or worse
Cost: Same or higher (inference)
Flexibility: Reduced

Break-Even Analysis

Worth it if:
- Quality gain > 10%
- High-volume use case
- Long-term commitment
- Domain stability

SPRAPP Approach

We recommend:

1. Exhaust prompt engineering first
2. Try RAG for knowledge needs
3. Fine-tune only when clear ROI
4. Start with synthesis model
5. Measure rigorously

The council of LLMs often achieves customization through model selection and prompt engineering before fine-tuning is needed.