Overview
TheModelRouter is an intelligent routing system that automatically selects and executes AI models based on task requirements. It leverages a function-calling architecture to analyze tasks and recommend the optimal model and provider combination for each specific use case.
Each step makes two LLM calls:
- A routing call to
gpt-4owithresponse_format=ModelOutput. The reply is parsed into aModelOutputthat names the model, provider, rewritten task,max_tokens,temperature, andsystem_promptto use. - An execution call to
{provider}/{model}through LiteLLM, using the settings from that reply.
Key Features
- Dynamic model selection based on task complexity and requirements
- Multi-provider support (OpenAI, Anthropic, Google, etc.)
- Sequential, batch, and thread-pool concurrent execution
- Optional multi-loop refinement, where each step’s output becomes the next step’s task
- OpenTelemetry tracing: each
run()is aModelRouter.runspan, andconcurrent_run()carries the tracing context into its worker threads (see Telemetry)
Installation
- Install the latest version of swarms:
- Set up your API keys in your
.envfile:
Attributes
str
default:"model_router_system_prompt"
System prompt for the routing call. It replaces the built-in selection guidance, so describe the models you want the router to choose from. It does not reach the selected model; that model gets the
system_prompt written in the routing reply.int
default:"4000"
Stored as
router.max_tokens and shown in repr(). Neither LLM call reads it: the execution call uses the max_tokens from the routing reply.float
default:"0.5"
Temperature for the routing call. The execution call uses the
temperature from the routing reply.int | str
default:"10"
Thread count for
concurrent_run(). Use "auto" for os.cpu_count().str
default:"None"
Accepted but unused. Set provider keys as environment variables instead.
int
default:"1"
Maximum number of steps per
run(). Each step’s output becomes the next step’s task, and the loop stops early if a step returns its input unchanged.ModelOutput
The schema the routing call must return. Import it from swarms.structs.model_router.
Methods
step()
Runs a single routing step. It sends the task to the routing model, parses the JSON reply into aModelOutput, prints the decision, then runs the rewritten task on {provider}/{model} and prints the output. run() calls this internally in a loop up to max_loops times.
task(str): The task to be executed
str - The selected model’s output
Raises: pydantic.ValidationError if the routing reply does not parse as a ModelOutput. Errors from the execution call propagate unchanged.
run()
Runsstep() up to max_loops times, feeding each output back in as the next task. Recorded as a ModelRouter.run trace span when telemetry is enabled. Calling the router directly, router(task), does the same thing.
task(str): The task to be executed
str - The result of task execution
batch_run()
Executes multiple tasks sequentially with result aggregation.tasks(list): List of task strings to be executed
list - List of results, one for each task
Raises: RuntimeError wrapping the first failure.
concurrent_run()
Runsrun() once per task on a thread pool of max_workers threads. The pool carries the caller’s tracing context into each thread, so every task’s span joins the caller’s trace.
tasks(list): List of task strings to be executed
list - Results in task order
Raises: RuntimeError wrapping the first failure.
async_run()
Documented (broken) signature:Usage Examples
Basic Usage
Batch Processing
Concurrent Execution
Asynchronous Execution
async_run() is currently broken (see the Warning above). To call ModelRouter from an async context without blocking the event loop, offload the synchronous run() to a thread with asyncio.to_thread():
Financial Analysis System
Healthcare Data Processing Pipeline
NLP Processing Pipeline
Available Models and Use Cases
The default routing prompt embeds these recommendations. The routing model can still name any model.Provider Capabilities
Performance Optimization Tips
-
Token Management
- The routing reply sets
max_tokensfor each execution call, so ask for a limit in yoursystem_promptif you need one - Raise
max_loopsonly when refinement is worth the extra calls; each step makes two LLM calls
- The routing reply sets
-
Concurrency Settings
- Adjust
max_workersbased on system resources - Use
"auto"workers for optimal CPU utilization - Monitor memory usage with large batch sizes
- Adjust
-
Temperature Tuning
- Lower (0.1-0.3) for factual/analytical tasks
- Higher (0.7-0.9) for creative tasks
- Mid-range (0.4-0.6) for balanced outputs
-
System Prompts
- A custom
system_promptreplaces the routing guidance, so list the models and providers you want chosen - Include relevant context
- Define clear output formats
- A custom
Dependencies
asyncio: Asynchronous I/O supportconcurrent.futures: Thread pool executionpydantic: Data validationlitellm: LLM interface standardization