PlannerGeneratorEvaluator

Overview

The PlannerGeneratorEvaluator is a domain-agnostic three-agent orchestration harness inspired by the GAN-style architecture described in Anthropic’s harness design research. It coordinates long-running autonomous tasks from a short natural-language prompt, using an iterative generate-evaluate feedback loop to converge on high-quality output across any domain. All three agents communicate through a single shared file on disk. The harness follows this workflow:

Planning: Planner expands a short prompt into an ambitious plan with steps and evaluation criteria
Contract Negotiation: Generator proposes what “done” looks like for each step; Evaluator reviews
Execution: Generator produces concrete output and self-assesses before handoff
Evaluation: Evaluator scores output per-criterion with hard thresholds — any criterion below its threshold fails the step
Feedback Loop: On failure, Generator receives scores + trajectory signal (refine or pivot) and retries
All state on disk: The shared state file is the single append-only record of the run. Each run() call rewrites it from scratch

Installation

pip install -U swarms

Key Features

Feature	Description
GAN-Style Separation	Distinct Generator and Evaluator agents prevent self-evaluation bias
Step Contracts	Generator and Evaluator agree on success criteria before execution
Hard Threshold Enforcement	Any single criterion below its threshold fails the step
Score Trajectory	Tracks score trends across retries — signals Generator to refine or pivot
Self-Assessment	Generator self-evaluates before Evaluator handoff
Shared State File	Single append-only `.md` file for all inter-agent communication
Domain-Agnostic	Planner defines evaluation criteria tailored to the task domain
Custom Agents	Pass pre-configured agents with tools, MCP, or any Agent settings
Configurable Thresholds	Default thresholds plus Planner-defined per-criterion thresholds

Attributes

str

default:"None"

Unique identifier for this harness instance. Auto-generated via generate_id("planner-generator-evaluator") if not provided.

str

default:"PlannerGeneratorEvaluator"

Human-readable name for this harness.

str

Description of the harness purpose.

str

default:"gpt-5.4"

Model identifier for all three agents

str

default:"None"

Override model for the Planner

str

default:"None"

Override model for the Generator

str

default:"None"

Override model for the Evaluator

int

default:"10"

Upper bound on plan steps to execute

int

default:"3"

Retries allowed after a failed evaluation before the harness advances to the next step. Each step gets up to max_retries_per_step + 1 attempts; 0 means one attempt

str

default:"os.getcwd()"

Directory where output is produced. Created if it does not exist

str

default:"None"

Path for the shared state file. Defaults to {working_directory}/pge_shared_state_{id}.md

Dict[str, float]

default:"{}"

Fallback score thresholds by criterion name, on the same 1-10 scale the Evaluator scores with. Criterion names are matched in lowercase. See Pass and fail rules

OutputType

default:"dict"

Format for the returned conversation history, such as "dict", "str", "list", "final", "json" or "yaml"

bool

default:"False"

Enable verbose logging

str

default:"PLANNER_SYSTEM_PROMPT"

System prompt for the Planner agent. Defaults to the built-in planner prompt.

str

default:"GENERATOR_SYSTEM_PROMPT"

System prompt for the Generator agent. Defaults to the built-in generator prompt.

str

default:"EVALUATOR_SYSTEM_PROMPT"

System prompt for the Evaluator agent. Defaults to the built-in evaluator prompt.

Agent

default:"None"

Pre-configured Agent for planning. When set, planner_model_name and planner_system_prompt are not applied to it

Agent

default:"None"

Pre-configured Agent for generation (e.g., with file/code tools). When set, generator_model_name and generator_system_prompt are not applied to it

Agent

default:"None"

Pre-configured Agent for evaluation (e.g., with Playwright MCP). When set, evaluator_model_name and evaluator_system_prompt are not applied to it

Raises:

Exception	Condition
`ValueError`	If `max_steps < 1`, `max_retries_per_step < 0`, or `model_name` is empty

Methods

run()

Execute the full PGE harness pipeline from a short prompt to completed output.

def run(self, task: str, *args, **kwargs) -> Any

Parameters:

task (str): A short natural-language description of the desired task

Returns: Formatted conversation history according to output_type Raises: ValueError if task is empty or not a string. Any error raised by an agent is logged and re-raised. Each call clears the conversation and rewrites the shared state file, so reusing one harness for several tasks does not carry the previous task forward. After run() completes, access harness.last_result for structured metadata:

Field	Type	Description
`output_path`	`str`	Path to the shared state file
`plan`	`str`	The generated plan text
`step_logs`	`List[Dict]`	Per-step metadata (contract, scores, retries)
`total_duration`	`float`	Wall-clock time in seconds
`total_steps_completed`	`int`	Number of steps that passed evaluation
`total_retries`	`int`	Total retry attempts across all steps

batched_run()

Run the harness on multiple tasks sequentially.

def batched_run(self, tasks: List[str]) -> List[Any]

Parameters:

tasks (List[str]): List of task prompts to process

Returns: List of results, one per task

get_harness_result()

Return the current state of the harness as a dictionary.

def get_harness_result(self) -> Dict[str, Any]

Returns: Dictionary with id, name, shared_state_path, and conversation (the full conversation history as a dict)

`call()`

Makes the harness callable directly, as a convenience alias for run().

def __call__(self, task: str, *args, **kwargs) -> Any

Parameters:

task (str): A short natural-language description of the desired task
*args, **kwargs: Forwarded to run()

Returns: Same as run()

harness = PlannerGeneratorEvaluator(model_name="gpt-5.4")
result = harness("Write a short guide on sourdough starters")  # same as harness.run(...)

Return Types

The PGE harness produces three structured return types that are also re-exported from swarms.structs so you can type-annotate against them.

StepContract

A negotiated agreement between the Generator and the Evaluator for one step of the plan — the title and acceptance criteria the Generator commits to, plus an approved flag and any amendments the Evaluator pushed back with.

from swarms import StepContract

contract = StepContract(
    step_number=1,
    title="Draft outline",
    acceptance_criteria="Three sections, each with bullet points",
    approved=True,
    amendments="",
)

int

required

1-indexed position in the plan.

str

default:"\"\""

Short step title.

str

default:"\"\""

What the Generator’s output must satisfy for the Evaluator to approve.

bool

default:"False"

Whether the Evaluator approved this contract.

str

default:"\"\""

Evaluator-suggested changes when not approved.

EvaluationReport

Per-step evaluation produced by the Evaluator agent — criterion scores, threshold checks, pass/fail, and feedback used to drive retries.

from swarms import EvaluationReport

report = EvaluationReport(
    step_number=1,
    criterion_scores={"completeness": 9.0, "accuracy": 8.5},
    criterion_thresholds={"completeness": 7.0, "accuracy": 8.0},
    passed=True,
    actionable_feedback="Tighten the second paragraph for clarity",
    summary="Meets thresholds; minor stylistic notes",
    raw_evaluation="...",
)

int

required

Step this report belongs to.

Dict[str, float] | None

default:"None"

Map of lowercased criterion name to score, on the Evaluator’s 1-10 scale.

Dict[str, float] | None

default:"None"

Minimum scores needed to pass each criterion.

bool

default:"False"

Whether the step met every threshold.

str

default:"\"\""

Specific, retry-targeted feedback for the Generator.

str

default:"\"\""

Short prose summary of the evaluation.

str

default:"\"\""

Unparsed Evaluator output. On a retry, this full text is what the Generator receives as feedback.

All three classes are plain Python classes with a to_dict() method. EvaluationReport.to_dict() leaves out raw_evaluation.

HarnessResult

Final result container — populated after run() and accessible via harness.last_result.

from swarms import HarnessResult

result = HarnessResult(
    output_path="pge_shared_state_example.md",
    plan="...",
    step_logs=[{"step_number": 1, "contract": None, "evaluations": [], "retries": 0, "passed": True}],
    total_duration=42.7,
    total_steps_completed=3,
    total_retries=1,
)

str

default:"\"\""

Path to the shared state file, which holds the plan, every contract, work log and evaluation, and the final summary.

str

default:"\"\""

The full plan the harness executed against.

List[Dict[str, Any]] | None

default:"None"

One dict per step with step_number, contract (the StepContract as a dict), evaluations (one EvaluationReport dict per attempt), retries and passed.

float

default:"0.0"

Wall-clock seconds for the full run.

int

default:"0"

Number of steps that hit passed=True.

int

default:"0"

Total retries across all steps.

Usage Examples

Basic Usage

from swarms import PlannerGeneratorEvaluator

harness = PlannerGeneratorEvaluator(
    model_name="gpt-5.4",
    max_steps=3,
    max_retries_per_step=2,
    output_type="final",
    verbose=True,
)

result = harness.run(
    "Write a comprehensive guide on the benefits and risks of intermittent fasting"
)

print(result)
print(f"Steps completed: {harness.last_result.total_steps_completed}")
print(f"Duration: {harness.last_result.total_duration:.1f}s")

Custom Agents with Tools

Pass pre-configured agents with tools so the Generator can write files and the Evaluator can verify them on disk:

from swarms import Agent, PlannerGeneratorEvaluator


def write_file(filename: str, content: str) -> str:
    """Write content to a file."""
    with open(filename, "w") as f:
        f.write(content)
    return f"Written: {filename}"


def read_file(filename: str) -> str:
    """Read content from a file."""
    with open(filename, "r") as f:
        return f.read()


generator = Agent(
    agent_name="PGE-Generator",
    model_name="gpt-5.4",
    max_loops=1,
    tools=[write_file],
)

evaluator = Agent(
    agent_name="PGE-Evaluator",
    model_name="gpt-5.4",
    max_loops=1,
    tools=[read_file],
)

harness = PlannerGeneratorEvaluator(
    model_name="gpt-5.4",
    generator_agent=generator,
    evaluator_agent=evaluator,
    max_steps=3,
)

result = harness.run("Create a Python module for string manipulation utilities")

Evaluator with Playwright MCP (Web App Testing)

For web application development, give the Evaluator browser automation via Playwright MCP so it can test the running app like a real user:

from swarms import Agent, PlannerGeneratorEvaluator

evaluator = Agent(
    agent_name="PGE-Evaluator",
    model_name="gpt-5.4",
    max_loops=1,
    mcp_config={"url": "http://localhost:3000/playwright"},
)

harness = PlannerGeneratorEvaluator(
    model_name="gpt-5.4",
    evaluator_agent=evaluator,
    max_steps=5,
    max_retries_per_step=3,
)

result = harness.run("Build a todo app with React frontend and FastAPI backend")

Custom Thresholds

Provide default score thresholds that apply when the Planner doesn’t define them. They are used when the Evaluator reports inline scores (accuracy: 7/10); a score table row carries its own threshold. See Pass and fail rules:

from swarms import PlannerGeneratorEvaluator

harness = PlannerGeneratorEvaluator(
    model_name="gpt-5.4",
    default_thresholds={
        "accuracy": 8.0,
        "clarity": 7.0,
        "completeness": 7.0,
    },
    max_retries_per_step=4,
)

result = harness.run("Write a technical specification for a rate-limiting middleware")

Architecture Details

Shared State File

All inter-agent communication flows through a single append-only markdown file. Each agent call receives the full file pasted into its task, and its reply is appended as a new timestamped, labeled section:

# PGE Harness Shared State

**Harness ID**: planner-generator-evaluator-...
**Created**: 2026-03-25 10:30:00

---

## User Prompt
[original prompt]

---
### [PLANNER OUTPUT] (2026-03-25 10:30:00)
[plan with steps and evaluation criteria]

---
### [STEP 1 CONTRACT PROPOSAL] (2026-03-25 10:30:15)
[Generator's proposed contract]

---
### [STEP 1 CONTRACT REVIEW] (2026-03-25 10:30:25)
[Evaluator's review -- APPROVED or AMENDMENTS REQUIRED]

---
### [STEP 1 WORK LOG] (2026-03-25 10:30:45)
[Generator's output + self-assessment]

---
### [STEP 1 EVALUATION] (2026-03-25 10:31:00)
[Evaluator's per-criterion scores, findings, and feedback]

---
### [STEP 1 WORK LOG (Retry 1)] (2026-03-25 10:31:30)
[Generator's revised output, if the step failed]

---
### [HARNESS SUMMARY] (2026-03-25 10:40:00)
[Steps passed, total retries, elapsed time, shared state path]

Contract negotiation

Before each step, the Generator proposes a contract and the Evaluator reviews it. The contract counts as approved when the review contains “approved” and not “amendments required”. Execution proceeds either way: an unapproved review is kept as the contract’s amendments, and it stays in the shared state for the Generator to read.

Step count

The harness reads the number of steps from the plan: the highest Step N it finds, else the highest numbered list item, else 3. The result is capped at max_steps.

Pass and fail rules

The Evaluator’s reply is parsed in this order:

Score table rows of the form | criterion | score | threshold | PASS/FAIL |. Each score is compared with the threshold in its own row.
Inline scores such as Clarity: 7/10 or Clarity: 7 out of 10. Each score is compared with the plan’s threshold for that criterion. Plan thresholds come from the Planner’s criteria table, layered over default_thresholds. A criterion with no threshold uses 6.0.
No scores found: an Overall Status: PASS or Overall Status: FAIL line decides. If there is none, the step passes.

Any criterion below its threshold fails the step. On a retry, the Generator receives the Evaluator’s full reply as feedback.

Refine vs. Pivot

When a step fails evaluation, the harness compares the average score of the last two evaluations for that step:

Scores improving — REFINE: keep the current direction, fix specific issues
Scores declining or stagnant — PIVOT: take a fundamentally different approach

This signal is passed to the Generator alongside the Evaluator’s feedback. It needs two scored evaluations, so it first appears on the second retry.

Evaluation Criteria

The Planner defines domain-appropriate criteria as part of the plan. Each criterion has:

Field	Description
Name	Short label (e.g., “accuracy”, “clarity”)
Weight	Relative importance (high, standard, low)
Description	What it measures and what good/bad looks like
Threshold	Minimum passing score (1-10). Any criterion below threshold = step fails

Source Code

View the source code on GitHub

Core Classes

Multi-Agent Harnesses

Tools & Utilities

Deployment

PlannerGeneratorEvaluator

Overview

Installation

Key Features

Attributes

Methods

run()

batched_run()

get_harness_result()

`call()`

Return Types

StepContract

EvaluationReport

HarnessResult

Usage Examples

Basic Usage

Custom Agents with Tools

Evaluator with Playwright MCP (Web App Testing)

Custom Thresholds

Architecture Details

Shared State File

Contract negotiation

Step count

Pass and fail rules

Refine vs. Pivot

Evaluation Criteria

Source Code

​Overview

​Installation

​Key Features

​Attributes

​Methods

​run()

​batched_run()

​get_harness_result()

​__call__()

​Return Types

​StepContract

​EvaluationReport

​HarnessResult

​Usage Examples

​Basic Usage

​Custom Agents with Tools

​Evaluator with Playwright MCP (Web App Testing)

​Custom Thresholds

​Architecture Details

​Shared State File

​Contract negotiation

​Step count

​Pass and fail rules

​Refine vs. Pivot

​Evaluation Criteria

​Source Code

Overview

Installation

Key Features

Attributes

Methods

run()

batched_run()

get_harness_result()

`call()`

Return Types

StepContract

EvaluationReport

HarnessResult

Usage Examples

Basic Usage

Custom Agents with Tools

Evaluator with Playwright MCP (Web App Testing)

Custom Thresholds

Architecture Details

Shared State File

Contract negotiation

Step count

Pass and fail rules

Refine vs. Pivot

Evaluation Criteria

Source Code