Failure Handling and Monitoring

A job that runs every thirty seconds for a week will fail sometimes. A rate limit, a dropped connection, a provider hiccup. The question is not whether it fails but what happens next.

The model

A task that raises is logged and retried at the next interval. It does not take the schedule down. CronJob catches the exception inside the job, records it as a failed execution, and returns normally, so the schedule library moves the job to its next run exactly as it does after a success. A failing job runs no more often than a healthy one: with interval="2seconds", a task that always fails is attempted once every two seconds.

~1s poll ──> due? ──> agent.run(task) ──> raised?
                                            │
                                            ├── no  ──> execution_count += 1
                                            │           consecutive_errors = 0
                                            │           callback runs, if set
                                            │           next_run advances by `interval`
                                            │
                                            └── yes ──> log with traceback
                                                        error_count += 1
                                                        consecutive_errors += 1
                                                        last_error = the exception
                                                        callback is skipped
                                                        next_run advances by `interval`

Earlier versions got this wrong twice. First, one exception set is_running = False and killed the scheduler thread, while run() returned a job object as though nothing had happened. Later, a raised exception left the job due, so a failing task was retried on every one-second poll instead of waiting out its interval. Both are fixed: failures are recorded and the job waits for its next interval.

Error budgets

Retrying forever is right for transient failures and wrong for a misconfigured job hammering a dead endpoint. max_consecutive_errors draws the line.

job = CronJob(
    agent=agent,
    interval="2seconds",
    max_consecutive_errors=10,
)

None (default): never give up. Every failure is logged, and the task runs again at the next interval.
An integer: after that many failures in a row, the job stops and run() raises CronJobExecutionError naming the count and the last error.

The counter resets on any success, so a job that fails occasionally never trips the budget. Only a job failing consistently does.

from swarms.structs.cron_job import CronJob, CronJobExecutionError

try:
    job.run("Fetch the latest reading.")
except CronJobExecutionError as e:
    # Only reached if the budget was exhausted. A clean stop() returns normally.
    alert(f"Schedule died: {e}")

That distinction is the important part: a clean stop() returns, a job that gave up raises. You can tell them apart.

Monitoring while it runs

get_execution_stats() is safe to poll from another thread.

import threading, time

def monitor(job, every=5.0):
    # Started before run(), so wait for the job to come up or the loop below exits at once.
    while not job.is_running:
        time.sleep(0.1)

    while job.is_running:
        time.sleep(every)
        s = job.get_execution_stats()
        print(
            f"ok={s['execution_count']} failed={s['error_count']} "
            f"in-a-row={s['consecutive_errors']} last={s['last_error']}"
        )

threading.Thread(target=monitor, args=(job,), daemon=True).start()
job.run("Fetch the latest reading.")

What to watch

Field	Means	Alert on
`execution_count`	Successful runs	Not increasing, when it should be
`error_count`	Total failures, ever	Ratio to `execution_count`
`consecutive_errors`	Failures since the last success	This one. A rising streak means genuinely broken
`last_error`	Most recent failure as a string	Reading it tells you which dependency
`stopped_due_to_error`	Job gave up	`True` is always worth paging on

error_count on its own is a poor signal: a job running every two seconds for a day will accumulate failures and be perfectly healthy. consecutive_errors is the one that distinguishes noise from breakage.

A complete example

Runs without API keys, since FlakyAgent stands in for an unreliable upstream.

import random
import threading
import time

from swarms.structs.cron_job import CronJob, CronJobExecutionError


class FlakyAgent:
    """Fails roughly half the time."""

    def run(self, task: str = None, **kwargs):
        if random.random() < 0.5:
            raise ConnectionError("upstream API timed out")
        return f"ok: {task}"


def monitor(job, every=5.0):
    while not job.is_running:
        time.sleep(0.1)
    while job.is_running:
        time.sleep(every)
        s = job.get_execution_stats()
        print(
            f"  [monitor] ok={s['execution_count']} failed={s['error_count']} "
            f"in-a-row={s['consecutive_errors']}"
        )


job = CronJob(
    agent=FlakyAgent(),
    interval="2seconds",
    max_consecutive_errors=10,
)

threading.Thread(target=monitor, args=(job,), daemon=True).start()
threading.Timer(60, job.stop).start()

try:
    job.run("Fetch the latest reading.")
except CronJobExecutionError as e:
    print(f"Job gave up: {e}")
else:
    s = job.get_execution_stats()
    print(f"Stopped cleanly: {s['execution_count']} ok, {s['error_count']} failed.")

Over sixty seconds the job makes about thirty attempts, one every two seconds whether the last one failed or not. About half succeed and half fail, and it never stops, because the failures almost never stack ten deep in a row.

Budgets across a fleet

With run_many, budgets are per agent. One agent giving up does not stop its siblings:

CronJob.run_many([
    {"agent": stable_agent, "interval": "1minute", "task": "Check inventory."},
    {
        "agent": flaky_agent,
        "interval": "1minute",
        "task": "Poll the third-party feed.",
        "max_consecutive_errors": 5,   # only this one has a budget
    },
])

If the flaky agent exhausts its five, it stops, the stable one keeps running, and run_many raises once blocking ends, naming which job gave up.

Next steps

Multiple Agents

Fleets on mixed cadences

CronJob Quickstart

Start with a single agent

CronJob Reference

Full parameter and method documentation

Runnable Examples

The example files in the repository

Index

Basic Examples

Model Providers

Single Agent

Tree of Thoughts

Multi-Agent Examples

Auto Agent Builder

Scheduled Agents

Applications

Research

Technical Analysis

Use Cases

Finance

Voice Agents

Integrations

MCP Tool Integrations

Deployment

CLI

Failure Handling and Monitoring

The model

Error budgets

Monitoring while it runs

What to watch

A complete example

Budgets across a fleet

Next steps

Multiple Agents

CronJob Quickstart

CronJob Reference

Runnable Examples

​The model

​Error budgets

​Monitoring while it runs

​What to watch

​A complete example

​Budgets across a fleet

​Next steps

Multiple Agents

CronJob Quickstart

CronJob Reference

Runnable Examples

The model

Error budgets

Monitoring while it runs

What to watch

A complete example

Budgets across a fleet

Next steps