Testing LLM Councils: Ensuring Quality in Multi-Model AI Systems

Testing AI is Hard

Testing non-deterministic AI systems is challenging. Testing multi-model councils adds complexity. Here's how to do it right.

Testing Levels

1. Unit Testing

Test individual components:

Model Connector Tests

describe('ClaudeConnector', () => {
  it('should handle successful response', async () => {
    const result = await connector.query('test');
    expect(result).toHaveProperty('content');
  });
  
  it('should handle rate limits', async () => {
    mockRateLimit();
    await expect(connector.query('test')).rejects.toThrow('rate limit');
  });
});

Consensus Algorithm Tests

describe('ConsensusCalculator', () => {
  it('should detect unanimous agreement', () => {
    const votes = ['A', 'A', 'A', 'A', 'A'];
    expect(calcConsensus(votes)).toEqual({ winner: 'A', confidence: 1.0 });
  });
  
  it('should handle ties', () => {
    const votes = ['A', 'A', 'B', 'B'];
    expect(calcConsensus(votes)).toEqual({ winner: null, confidence: 0.5 });
  });
});

2. Integration Testing

Test council workflows:

Fan-Out Integration

it('should query all models in parallel', async () => {
  const start = Date.now();
  const results = await council.fanOut('test query', ['claude', 'gpt', 'gemini']);
  const duration = Date.now() - start;
  
  expect(results).toHaveLength(3);
  expect(duration).toBeLessThan(10000); // Parallel, not sequential
});

Debate Integration

it('should complete debate rounds', async () => {
  const result = await council.debate('complex question', { rounds: 2 });
  
  expect(result.rounds).toBe(2);
  expect(result.finalAnswer).toBeDefined();
  expect(result.consensus).toBeGreaterThan(0.5);
});

3. Quality Testing

Test actual output quality:

Benchmark Suites

const benchmarks = [
  { query: '2+2', expectedAnswer: '4', category: 'math' },
  { query: 'Capital of France', expectedAnswer: 'Paris', category: 'facts' },
  { query: 'Explain photosynthesis', keywords: ['sunlight', 'plants', 'energy'], category: 'science' }
];

benchmarks.forEach(b => {
  it(`should answer: ${b.query}`, async () => {
    const result = await council.query(b.query);
    expect(containsAnswer(result, b.expectedAnswer)).toBeTruthy();
  });
});

4. Regression Testing

Prevent quality degradation:

Golden Master Testing

const goldenResponses = loadGoldenMasters();

it('should maintain quality for known queries', async () => {
  for (const [query, expected] of Object.entries(goldenResponses)) {
    const result = await council.query(query);
    expect(similarity(result, expected)).toBeGreaterThan(0.8);
  }
});

Test Data Management

Curated Test Sets

• Factual questions with known answers
• Reasoning problems with verifiable solutions
• Edge cases known to challenge models

Adversarial Examples

• Questions designed to induce hallucinations
• Ambiguous queries
• Contradictory prompts

Real Query Samples

• Sample from production queries
• Anonymize sensitive data
• Regular refresh

Testing Metrics

Metric	Target	Measurement
Factual accuracy	>95%	Known-answer tests
Consensus rate	>70%	Agreement distribution
Latency P95	<8s	Performance tests
Error rate	<1%	Reliability tests

Continuous Testing

CI/CD Integration

• Run quality tests on every commit
• Block deployment on regression
• Track quality trends over time

A/B Testing

Compare configurations:

• Control: Current council config
• Treatment: Modified config
• Measure: Quality, speed, cost

Canary Testing

Gradual rollout:

• 5% traffic to new config
• Monitor metrics
• Increase or rollback

SPRAPP Testing

Features for quality assurance:

• Built-in benchmark suites
• Regression test framework
• A/B testing infrastructure
• Quality dashboards

The council of LLMs requires systematic testing for production reliability.