Skip to main content

Overview

The ModelRouter is an intelligent routing system that automatically selects and executes AI models based on task requirements. It leverages a function-calling architecture to analyze tasks and recommend the optimal model and provider combination for each specific use case. Each step makes two LLM calls:
  1. A routing call to gpt-4o with response_format=ModelOutput. The reply is parsed into a ModelOutput that names the model, provider, rewritten task, max_tokens, temperature, and system_prompt to use.
  2. An execution call to {provider}/{model} through LiteLLM, using the settings from that reply.

Key Features

  • Dynamic model selection based on task complexity and requirements
  • Multi-provider support (OpenAI, Anthropic, Google, etc.)
  • Sequential, batch, and thread-pool concurrent execution
  • Optional multi-loop refinement, where each step’s output becomes the next step’s task
  • OpenTelemetry tracing: each run() is a ModelRouter.run span, and concurrent_run() carries the tracing context into its worker threads (see Telemetry)

Installation

  1. Install the latest version of swarms:
  1. Set up your API keys in your .env file:

Attributes

str
default:"model_router_system_prompt"
System prompt for the routing call. It replaces the built-in selection guidance, so describe the models you want the router to choose from. It does not reach the selected model; that model gets the system_prompt written in the routing reply.
int
default:"4000"
Stored as router.max_tokens and shown in repr(). Neither LLM call reads it: the execution call uses the max_tokens from the routing reply.
float
default:"0.5"
Temperature for the routing call. The execution call uses the temperature from the routing reply.
int | str
default:"10"
Thread count for concurrent_run(). Use "auto" for os.cpu_count().
str
default:"None"
Accepted but unused. Set provider keys as environment variables instead.
int
default:"1"
Maximum number of steps per run(). Each step’s output becomes the next step’s task, and the loop stops early if a step returns its input unchanged.
Extra positional and keyword arguments are accepted and ignored.

ModelOutput

The schema the routing call must return. Import it from swarms.structs.model_router.

Methods

step()

Runs a single routing step. It sends the task to the routing model, parses the JSON reply into a ModelOutput, prints the decision, then runs the rewritten task on {provider}/{model} and prints the output. run() calls this internally in a loop up to max_loops times.
Parameters:
  • task (str): The task to be executed
Returns: str - The selected model’s output Raises: pydantic.ValidationError if the routing reply does not parse as a ModelOutput. Errors from the execution call propagate unchanged.

run()

Runs step() up to max_loops times, feeding each output back in as the next task. Recorded as a ModelRouter.run trace span when telemetry is enabled. Calling the router directly, router(task), does the same thing.
Parameters:
  • task (str): The task to be executed
Returns: str - The result of task execution

batch_run()

Executes multiple tasks sequentially with result aggregation.
Parameters:
  • tasks (list): List of task strings to be executed
Returns: list - List of results, one for each task Raises: RuntimeError wrapping the first failure.

concurrent_run()

Runs run() once per task on a thread pool of max_workers threads. The pool carries the caller’s tracing context into each thread, so every task’s span joins the caller’s trace.
Parameters:
  • tasks (list): List of task strings to be executed
Returns: list - Results in task order Raises: RuntimeError wrapping the first failure.

async_run()

async_run() is currently broken and should not be used. Its implementation is return asyncio.create_task(self.run(task, *args, **kwargs)), but run() is a plain sync def — so self.run(...) executes eagerly (blocking the event loop) and returns a plain str before asyncio.create_task() ever sees it. Passing that string to create_task() raises TypeError: a coroutine was expected, which async_run() catches and re-raises as RuntimeError: Async execution failed: .... Every await router.async_run(...) call site fails. Use the synchronous run() (optionally wrapped in asyncio.to_thread() from an async context) or concurrent_run() / batch_run() for parallel execution instead.
Documented (broken) signature:
Working alternative from an async context:

Usage Examples

Basic Usage

Batch Processing

Concurrent Execution

Asynchronous Execution

async_run() is currently broken (see the Warning above). To call ModelRouter from an async context without blocking the event loop, offload the synchronous run() to a thread with asyncio.to_thread():

Financial Analysis System

Healthcare Data Processing Pipeline

NLP Processing Pipeline

Available Models and Use Cases

The default routing prompt embeds these recommendations. The routing model can still name any model.
Most of these are older model names. If your provider no longer serves the model the router picks, the execution call fails. Pass a system_prompt that lists the models you can call.

Provider Capabilities

Performance Optimization Tips

  1. Token Management
    • The routing reply sets max_tokens for each execution call, so ask for a limit in your system_prompt if you need one
    • Raise max_loops only when refinement is worth the extra calls; each step makes two LLM calls
  2. Concurrency Settings
    • Adjust max_workers based on system resources
    • Use "auto" workers for optimal CPU utilization
    • Monitor memory usage with large batch sizes
  3. Temperature Tuning
    • Lower (0.1-0.3) for factual/analytical tasks
    • Higher (0.7-0.9) for creative tasks
    • Mid-range (0.4-0.6) for balanced outputs
  4. System Prompts
    • A custom system_prompt replaces the routing guidance, so list the models and providers you want chosen
    • Include relevant context
    • Define clear output formats

Dependencies

  • asyncio: Asynchronous I/O support
  • concurrent.futures: Thread pool execution
  • pydantic: Data validation
  • litellm: LLM interface standardization

Source Code

View the source code on GitHub