LLMCouncil

Overview

The LLMCouncil creates a council of specialized LLM agents that collaborate through independent responses, peer review, and synthesis. Inspired by Andrej Karpathy’s llm-council, it demonstrates how different models evaluate and rank each other’s work.

Installation

pip install -U swarms

Workflow

Dispatch: Task sent to all council members in parallel
Respond: Each member independently answers the task
Evaluate: Every member reviews and ranks all responses, which are relabelled Response A, Response B, and so on in a shuffled order. Evaluations run concurrently
Synthesize: The Chairman receives every response under its member’s name, every evaluation, and the mapping from anonymous IDs to members, and writes the final answer

Each run starts from an empty conversation, so reusing a council for several tasks never mixes their transcripts.

Attributes

str

default:"generate_id('llm-council')"

Unique identifier for the council

str

default:"LLM Council"

Name of the council

str

default:"A collaborative council of LLM agents..."

Description of the council’s purpose

Optional[List[Agent]]

default:"None"

List of Agent instances representing council members. If None, creates the default council of four members on OpenAI, Gemini, Anthropic, and xAI models.

str

default:"gpt-5.1"

Model name for the Chairman agent that synthesizes responses. The Chairman runs with temperature=0.7.

bool

default:"True"

Whether to log one line per stage through loguru: initialization, collecting responses, ranking, synthesis, and completion.

HistoryOutputType

default:"dict-all-except-first"

Format for the output ("list", "dict", "string", "final", "json", "yaml", and so on). The default returns every message after the task as a list of message dicts. Use "final" for only the Chairman’s answer.

Methods

run()

Execute the full LLM Council workflow.

def run(self, task: str)

Parameters:

task (str): The user’s task to process. Required.

Returns: The conversation formatted according to output_type, containing the member responses, evaluations, and the Chairman’s synthesis Raises: ValueError if task is not a string, is empty, or contains only whitespace

The query alias was removed. Call run(task=...) or pass the task positionally.

batched_run()

Run the LLM Council workflow for a batch of tasks, one after another.

def batched_run(self, tasks: List[str]) -> list

Parameters:

tasks (List[str]): List of tasks to process

Returns: List of formatted outputs

Usage Examples

Basic Usage with Default Council

from swarms import LLMCouncil

# Create council with default members
council = LLMCouncil(
    verbose=True,
    output_type="final"
)

# Process a query
result = council.run(
    task="What are the key considerations for building a production AI system?"
)

print(result)

Custom Council Members

from swarms import Agent, LLMCouncil

# Define custom council members
council_members = [
    Agent(
        agent_name="Security-Expert",
        agent_description="Security and privacy specialist",
        system_prompt="You are a security expert. Focus on security implications.",
        model_name="openai/gpt-5.4",
        max_loops=1,
    ),
    Agent(
        agent_name="Performance-Expert",
        agent_description="Performance and scalability specialist",
        system_prompt="You are a performance expert. Focus on scalability.",
        model_name="claude-sonnet-4-6",
        max_loops=1,
    ),
    Agent(
        agent_name="UX-Expert",
        agent_description="User experience specialist",
        system_prompt="You are a UX expert. Focus on user experience.",
        model_name="gemini/gemini-2.5-flash",
        max_loops=1,
    ),
]

council = LLMCouncil(
    council_members=council_members,
    chairman_model="claude-sonnet-4-6",
    verbose=True
)

result = council.run(task="Design a mobile app for health tracking")

Batch Processing

tasks = [
    "Explain quantum computing in simple terms",
    "What are best practices for API design?",
    "How to optimize database queries?"
]

results = council.batched_run(tasks)

for i, result in enumerate(results):
    print(f"\nTask {i+1} Result:")
    print(result)

Different Output Formats

# Get only final synthesized answer
council_final = LLMCouncil(output_type="final")
final_answer = council_final.run(task="Explain machine learning")

# Get the full conversation, including the task, as a list of message dicts
council_dict = LLMCouncil(output_type="dict")
full_conversation = council_dict.run(task="Explain machine learning")

# Get as JSON
council_json = LLMCouncil(output_type="json")
json_output = council_json.run(task="Explain machine learning")

Non-Verbose Mode

# Run without logging progress
council = LLMCouncil(verbose=False)
result = council.run(task="Some query")

With verbose=True, the council logs one line per stage:

[LLM Council] Initialized with 4 members: GPT-5.1-Councilor, Gemini-3-Pro-Councilor, Claude-Sonnet-4.5-Councilor, Grok-4-Councilor
[LLM Council] Collecting responses from 4 members
[LLM Council] Members ranking the anonymized responses
[LLM Council] Chairman synthesizing the final answer
[LLM Council] Session complete

Default Council Members

When no custom members are provided, the default council includes:

Agent name	Model	Temperature	Persona
`GPT-5.1-Councilor`	`gpt-5.1`	0.7	Analytical and comprehensive
`Gemini-3-Pro-Councilor`	`gemini-2.5-flash`	0.7	Concise and well-structured
`Claude-Sonnet-4.5-Councilor`	`anthropic/claude-sonnet-4-5`	0.0	Thoughtful and balanced
`Grok-4-Councilor`	`xai/grok-4-1-fast-reasoning`	0.8	Creative and innovative

The agent names do not always match the model: the Gemini councilor runs on gemini-2.5-flash. The default council calls four providers, so you need credentials for OpenAI, Anthropic, xAI, and Google. LiteLLM routes the unprefixed gemini-2.5-flash to Vertex AI. If you only have API keys, pass your own council_members, for example with gemini/gemini-2.5-flash.

Prompts

All council prompts live in swarms.prompts.llm_council_prompts. Use them to give custom members the same personas:

Function	Used for
`get_gpt_councilor_prompt()`	GPT councilor system prompt
`get_gemini_councilor_prompt()`	Gemini councilor system prompt
`get_claude_councilor_prompt()`	Claude councilor system prompt
`get_grok_councilor_prompt()`	Grok councilor system prompt
`get_chairman_prompt()`	Chairman system prompt
`get_evaluation_prompt(query, responses, evaluator_name)`	Peer-ranking task sent to each member
`get_synthesis_prompt(query, original_responses, evaluations, id_to_member)`	Synthesis task sent to the Chairman

from swarms import Agent, LLMCouncil
from swarms.prompts.llm_council_prompts import get_gpt_councilor_prompt

analyst = Agent(
    agent_name="Analyst-Councilor",
    system_prompt=get_gpt_councilor_prompt(),
    model_name="gpt-5.4",
    max_loops=1,
)

Evaluation Process

Each council member evaluates all responses (anonymized) and provides:

Rankings: Ordered list from best to worst response
Reasoning: Explanation for each ranking
Observations: Additional insights about strengths/weaknesses

Example evaluation format:

RANKINGS:
1. Response B: Clear structure and comprehensive coverage
2. Response A: Good depth but could be more organized
3. Response D: Creative but lacks some technical details
4. Response C: Too verbose, missing key points

ADDITIONAL OBSERVATIONS:
Common strength: All responses addressed the core question
Area for improvement: More concrete examples needed

Synthesis Process

The Chairman agent:

Reviews all original responses
Considers all evaluations and rankings
Identifies strongest elements from each response
Creates cohesive final answer incorporating best aspects
Acknowledges which perspectives influenced the synthesis

Output Structure

The full conversation (for example with output_type="dict") looks like this. The default "dict-all-except-first" omits the first User entry.

[
    {"role": "User", "content": "Original query"},
    {"role": "GPT-5.1-Councilor", "content": "Response..."},
    {"role": "Gemini-3-Pro-Councilor", "content": "Response..."},
    {"role": "Claude-Sonnet-4.5-Councilor", "content": "Response..."},
    {"role": "Grok-4-Councilor", "content": "Response..."},
    {"role": "GPT-5.1-Councilor-Evaluation", "content": "Evaluation..."},
    {"role": "Gemini-3-Pro-Councilor-Evaluation", "content": "Evaluation..."},
    {"role": "Claude-Sonnet-4.5-Councilor-Evaluation", "content": "Evaluation..."},
    {"role": "Grok-4-Councilor-Evaluation", "content": "Evaluation..."},
    {"role": "Chairman", "content": "Final synthesized answer"}
]

Features

Parallel Execution: All council members respond simultaneously
Anonymous Evaluation: Responses are anonymized during peer review
Multi-Model Diversity: Leverages different LLM strengths
Peer Review: Each member evaluates all responses objectively
Intelligent Synthesis: Chairman creates cohesive final answer
Transparent Process: Full conversation history available
Flexible Output: Multiple output format options
Batch Processing: Handle multiple queries efficiently

Best Practices

Council Composition: Include agents with complementary strengths
Clear Queries: Provide well-defined questions for best results
Output Type: Use "final" for end-user answers, and the default "dict-all-except-first" or "dict" for analysis
Custom Members: Tailor council to your domain/use case
Verbose Mode: Enable for understanding the decision process

Core Classes

Multi-Agent Harnesses

Tools & Utilities

Deployment

Overview

Installation

Workflow

Attributes

Methods

run()

batched_run()

Usage Examples

Basic Usage with Default Council

Custom Council Members

Batch Processing

Different Output Formats

Non-Verbose Mode

Default Council Members

Prompts

Evaluation Process

Synthesis Process

Output Structure

Features

Best Practices

​Overview

​Installation

​Workflow

​Attributes

​Methods

​run()

​batched_run()

​Usage Examples

​Basic Usage with Default Council

​Custom Council Members

​Batch Processing

​Different Output Formats

​Non-Verbose Mode

​Default Council Members

​Prompts

​Evaluation Process

​Synthesis Process

​Output Structure

​Features

​Best Practices

Overview

Installation

Workflow

Attributes

Methods

run()

batched_run()

Usage Examples

Basic Usage with Default Council

Custom Council Members

Batch Processing

Different Output Formats

Non-Verbose Mode

Default Council Members

Prompts

Evaluation Process

Synthesis Process

Output Structure

Features

Best Practices