CouncilAsAJudge

​

Overview

​

Installation

pip install -U swarms

​

Evaluation Dimensions

1. Accuracy: Factual correctness, source credibility, logical consistency
2. Helpfulness: Practical value, solution feasibility, problem-solving efficacy
3. Harmlessness: Safety assessment, ethical considerations, bias detection
4. Coherence: Structural integrity, logical flow, organization
5. Conciseness: Communication efficiency, precision, information density
6. Instruction Adherence: Compliance with requirements, constraint adherence

​

Attributes

​

Methods

​

run()

def run(self, task: str)

• task (str): Task containing the response to evaluate

1. Clears the conversation and records the task as a User message, so each run starts fresh
2. Runs the six judges in parallel. Each judge’s analysis is recorded under its own name as it finishes
3. Sends the aggregator the task and every judge’s analysis as separate chat turns, each labelled with the judge’s name
4. Records the aggregator’s report and returns the formatted conversation

​

Usage Examples

​

Basic Evaluation

from swarms import CouncilAsAJudge

# Create council
council = CouncilAsAJudge(
    name="Response-Evaluator",
    output_type="final"
)

# Evaluate a response
task_response = """
Question: Explain how neural networks work.

Response: Neural networks are computational models inspired by the human brain. 
They consist of layers of interconnected nodes (neurons) that process information. 
Each connection has a weight that adjusts during training. The network learns by 
adjusting these weights to minimize error between predictions and actual outputs.
This process is called backpropagation.
"""

evaluation = council.run(task_response)
print(evaluation)

​

Custom Model Configuration

council = CouncilAsAJudge(
    model_name="gpt-5.4-mini",  # judges
    aggregation_model_name="claude-sonnet-4-6",  # final report
    output_type="dict",
)

evaluation = council.run(task_response)

​

Full Conversation History

# Get complete evaluation breakdown
council = CouncilAsAJudge(
    output_type="dict-all-except-first"
)

full_evaluation = council.run(task_response)

# Access individual dimension evaluations
for message in full_evaluation:
    print(f"\n{message['role']}:")
    print(message['content'][:200])

​

Specific Judge Model

council = CouncilAsAJudge(
    judge_agent_model_name="claude-sonnet-4-6",
    aggregation_model_name="gpt-5.4",
)

evaluation = council.run(task_response)

​

Evaluation Report Structure

​

1. Executive Summary

• Key strengths and weaknesses
• Critical issues requiring immediate attention
• Overall assessment

​

2. Detailed Analysis

• Cross-dimensional patterns
• Specific examples and their implications
• Technical impact assessment

​

3. Recommendations

• Prioritized improvement areas
• Specific technical suggestions
• Implementation considerations

​

Dimension-Specific Evaluations

1. Specific Observations: References exact parts of the response
2. Impact Analysis: Explains how issues affect quality
3. Concrete Examples: Demonstrates strengths and weaknesses
4. Improvement Suggestions: Actionable recommendations

​

Parallel Execution

• Workers: Automatically configured based on CPU count (75% of cores)
• Concurrency: All 6 dimensions evaluated in parallel
• Error Handling: If any dimension fails, run raises EvaluationError and no report is produced
• Performance: Significantly faster than sequential evaluation

​

Example Output

evaluation = council.run(task_response)

# Output structure (when output_type="dict"). Judge entries appear in the order the judges finished.
[
    {"role": "User", "content": "Task response..."},
    {"role": "accuracy_judge", "content": "Accuracy analysis..."},
    {"role": "helpfulness_judge", "content": "Helpfulness analysis..."},
    {"role": "harmlessness_judge", "content": "Safety analysis..."},
    {"role": "coherence_judge", "content": "Coherence analysis..."},
    {"role": "conciseness_judge", "content": "Conciseness analysis..."},
    {"role": "instruction_adherence_judge", "content": "Adherence analysis..."},
    {"role": "aggregator_agent", "content": "Final comprehensive report..."}
]

​

Features

• Multi-Dimensional Evaluation: Comprehensive assessment across 6 key dimensions
• Parallel Processing: All evaluations run concurrently for maximum speed
• Expert Judge Agents: Each dimension evaluated by specialized agent
• Intelligent Aggregation: Senior agent synthesizes all findings
• Technical Analysis: Detailed, actionable feedback for improvement
• Flexible Models: Support for any LLM model via LiteLLM
• Caching: LRU cache for frequently used prompts
• Error Handling: Failures surface as EvaluationError that names the failing dimension
• Multiple Output Formats: Choose from various output types

​

Use Cases

1. Response Quality Assessment: Evaluate LLM outputs before deployment
2. Model Comparison: Compare different models’ responses
3. Training Data Evaluation: Assess quality of training examples
4. Content Review: Evaluate generated content for publication
5. Automated QA: Build quality assurance pipelines
6. A/B Testing: Compare different prompt variations

​

Best Practices

1. Output Type Selection:
   • Use "final" for quick summary reports
   • Use "dict" to analyze individual dimension evaluations
   • Use "json" for integration with other systems
2. Model Selection:
   • Set model_name or judge_agent_model_name to choose the judges’ model, and aggregation_model_name for the final report
   • Use stronger models (gpt-5.4, claude-sonnet-4-6) for critical evaluations
   • Use faster models (gpt-5.4-mini) for development and testing
3. Performance:
   • Council auto-configures workers based on CPU count
   • Consider cache_size for repeated similar evaluations
   • Monitor costs when using multiple premium models
4. Integration:
   • Parse the aggregated report for actionable insights
   • Use dimension-specific feedback for targeted improvements
   • Store evaluations for tracking quality over time