Agent Judge

The AgentJudge evaluates and critiques outputs from other AI agents, providing structured feedback on quality, accuracy, and areas for improvement. It supports single-shot evaluations and iterative refinement through multiple evaluation loops with context building. Based on the research paper: Agent-as-a-Judge: Evaluate Agents with Agents

Capability	Description
Quality Assessment	Evaluates correctness, clarity, and completeness of agent outputs
Structured Feedback	Provides detailed critiques with strengths, weaknesses, and suggestions
Multimodal Support	Can evaluate text outputs alongside images
Context Building	Maintains evaluation context across multiple iterations
Custom Evaluation Criteria	Supports weighted evaluation criteria for domain-specific assessments
Batch Processing	Evaluates a list of outputs one after another

Architecture

Parameters

Parameter	Type	Default	Description
`id`	`str`	`generate_id("agent-judge")`	Unique identifier for the judge instance (`<prefix>-<32 hex chars>`)
`agent_name`	`str`	`"Agent Judge"`	Name of the agent judge
`description`	`str`	`"You're an expert AI agent judge..."`	Description of the agent’s role, passed to the internal agent as `agent_description`
`system_prompt`	`str`	`None`	Custom system instructions (uses the built-in evaluation prompt if None)
`model_name`	`str`	`"openai/o1"`	LLM model for evaluation
`max_loops`	`int`	`1`	Number of evaluation iterations in `run()`
`verbose`	`bool`	`False`	Stored on the instance
`evaluation_criteria`	`Optional[Dict[str, float]]`	`None`	Criteria and weights, appended to the system prompt and to every evaluation request
`return_score`	`bool`	`False`	Make `run()` return a 0 or 1 keyword reward instead of the conversation
`args`, `*kwargs`			Extra `Agent` arguments applied to the internal agent

The constructor raises ValueError when max_loops is 0 or None, or when model_name is None.

Methods

step()

Processes a single task and returns the agent’s evaluation. task is the output to evaluate. It is wrapped in an evaluation prompt that asks for strengths, weaknesses, suggestions and factual errors. messages are earlier chat turns sent before it.

result = judge.step(task: str = None, img: Optional[str] = None, messages: Optional[List[Dict[str, str]]] = None) -> str

run()

Executes evaluation in multiple iterations with context building. Each iteration sends the judge’s earlier verdicts back to it as chat turns, so later iterations refine earlier ones.

result = judge.run(task: str, img: Optional[str] = None) -> Union[str, int]

Returns str (the judge’s conversation as text) if return_score=False, or int if return_score=True. With return_score=True, the int is 1 when the conversation text contains “correct”, “good”, “excellent” or “perfect” (case-insensitive), and 0 otherwise. The conversation includes the task you passed in, so a task that contains one of those words always scores 1.

run_batched()

Calls run() on each task in order.

results = judge.run_batched(tasks: List[str]) -> List[Union[str, int]]

step() and run() raise AgentJudgeExecutionError when evaluation fails.

judge.conversation is not cleared between calls. Each run() result, including each run_batched() result, contains every earlier task and verdict from the same judge, and later evaluations see them as context. Create a new AgentJudge for each independent evaluation.

Examples

Basic Evaluation

from swarms import AgentJudge

judge = AgentJudge(
    agent_name="quality-judge",
    model_name="claude-sonnet-4-6",
    max_loops=2
)

agent_output = "The capital of France is Paris. The city is known for its famous Eiffel Tower."

evaluations = judge.run(task=agent_output)

Custom Evaluation Criteria

from swarms import AgentJudge

judge = AgentJudge(
    agent_name="technical-judge",
    model_name="claude-sonnet-4-6",
    max_loops=1,
    evaluation_criteria={
        "accuracy": 0.4,
        "completeness": 0.3,
        "clarity": 0.2,
        "logic": 0.1,
    },
)

technical_output = "To solve x^2 + 5x + 6 = 0, we use the quadratic formula..."
evaluations = judge.run(task=technical_output)

Scoring Mode

from swarms import AgentJudge

judge = AgentJudge(
    agent_name="scoring-judge",
    model_name="claude-sonnet-4-6",
    max_loops=2,
    return_score=True
)

score = judge.run(task="The derivative of x^3 is 3x^2.")

Batch Processing

from swarms import AgentJudge

judge = AgentJudge()

tasks = [
    "The capital of France is Paris.",
    "2 + 2 = 4",
    "The Earth is flat."
]

evaluations = judge.run_batched(tasks=tasks)

Reference

@misc{zhuge2024agentasajudgeevaluateagentsagents,
    title={Agent-as-a-Judge: Evaluate Agents with Agents},
    author={Mingchen Zhuge and Changsheng Zhao and Dylan Ashley and Wenyi Wang and Dmitrii Khizbullin and Yunyang Xiong and Zechun Liu and Ernie Chang and Raghuraman Krishnamoorthi and Yuandong Tian and Yangyang Shi and Vikas Chandra and Jürgen Schmidhuber},
    year={2024},
    eprint={2410.10934},
    archivePrefix={arXiv},
    primaryClass={cs.AI},
}

Getting Started

Core Concepts

Agent Development

Reasoning Agents

Multi-Agent Architectures

Deployment & Operations

Integrations

CLI Reference

Changelog

Architecture

Parameters

Methods

step()

run()

run_batched()

Examples

Basic Evaluation

Custom Evaluation Criteria

Scoring Mode

Batch Processing

Reference

​Architecture

​Parameters

​Methods

​step()

​run()

​run_batched()

​Examples

​Basic Evaluation

​Custom Evaluation Criteria

​Scoring Mode

​Batch Processing

​Reference

Architecture

Parameters

Methods

step()

run()

run_batched()

Examples

Basic Evaluation

Custom Evaluation Criteria

Scoring Mode

Batch Processing

Reference