Firecrawl: scrape pages to markdown

Firecrawl turns arbitrary web pages into clean markdown an LLM can actually read: it renders JavaScript, strips navigation and ads, and follows links when you ask it to crawl rather than scrape a single page. It also shows the third authentication shape in this section — the key is a segment of the URL path, not a header or a query parameter.


Server	`https://mcp.firecrawl.dev/{API_KEY}/v2/mcp`
Auth	API key embedded in the URL path (free tier)
Tools	`firecrawl_scrape`, `firecrawl_crawl`, `firecrawl_map`, `firecrawl_search`, `firecrawl_extract`
Model used here	`claude-opus-5`

Crawling is the expensive operation. It is billed per page and can walk a large site quickly. Scrape one page first; crawl only when you mean to.

Prerequisites

Python 3.10+
ANTHROPIC_API_KEY
FIRECRAWL_API_KEY — free tier at firecrawl.dev

Build it

Install and set both keys

pip install -U swarms
export ANTHROPIC_API_KEY="sk-ant-..."
export FIRECRAWL_API_KEY="fc-..."

Fail loudly when the key is missing

Because the key is part of the URL, a missing environment variable produces the URL https://mcp.firecrawl.dev/None/v2/mcp and a confusing connection error. Check for it up front:

import os
import sys

FIRECRAWL_API_KEY = os.getenv("FIRECRAWL_API_KEY")

if not FIRECRAWL_API_KEY:
    sys.exit(
        "FIRECRAWL_API_KEY is not set.\n"
        "Get a free-tier key at https://www.firecrawl.dev/ and export it."
    )

Write a prompt that prefers scraping over crawling

Left alone, a model asked about a site will reach for the broadest tool available. This prompt pushes it toward the cheap one and forbids answering from memory:

SCRAPER_SYSTEM_PROMPT = (
    "You are a web content analyst. Fetch pages before describing them — "
    "never answer from memory about what a site says, because sites change. "
    "Prefer scraping the specific page that answers the question over "
    "crawling a whole site, since crawling is slow and expensive; crawl only "
    "when the user explicitly wants breadth. Quote the page's own wording for "
    "any factual claim, note the URL each fact came from, and say clearly "
    "when a page failed to load or was empty rather than substituting "
    "assumptions."
)

Build the agent

from swarms import Agent

agent = Agent(
    agent_name="Firecrawl-Analyst",
    agent_description="Reads and analyzes live web pages via Firecrawl MCP.",
    system_prompt=SCRAPER_SYSTEM_PROMPT,
    model_name="claude-opus-5",
    # The key is a path segment for Firecrawl. Never log this URL.
    mcp_url=f"https://mcp.firecrawl.dev/{FIRECRAWL_API_KEY}/v2/mcp",
    max_loops=2,
)

Scrape one page and ask for quotes

result = agent.run(
    "Scrape https://modelcontextprotocol.io/introduction and explain, in "
    "the docs' own terms, what problem MCP solves and what the three "
    "core primitives are. Quote the definitions."
)
print(result)

Asking for quotes is a cheap correctness check: if the model paraphrases everything, it probably did not read the page.

The complete script

import os
import sys

from swarms import Agent

FIRECRAWL_API_KEY = os.getenv("FIRECRAWL_API_KEY")

SCRAPER_SYSTEM_PROMPT = (
    "You are a web content analyst. Fetch pages before describing them — "
    "never answer from memory about what a site says, because sites change. "
    "Prefer scraping the specific page that answers the question over "
    "crawling a whole site, since crawling is slow and expensive; crawl only "
    "when the user explicitly wants breadth. Quote the page's own wording for "
    "any factual claim, note the URL each fact came from, and say clearly "
    "when a page failed to load or was empty rather than substituting "
    "assumptions."
)

if not FIRECRAWL_API_KEY:
    sys.exit(
        "FIRECRAWL_API_KEY is not set.\n"
        "Get a free-tier key at https://www.firecrawl.dev/ and export it."
    )

agent = Agent(
    agent_name="Firecrawl-Analyst",
    agent_description="Reads and analyzes live web pages via Firecrawl MCP.",
    system_prompt=SCRAPER_SYSTEM_PROMPT,
    model_name="claude-opus-5",
    mcp_url=f"https://mcp.firecrawl.dev/{FIRECRAWL_API_KEY}/v2/mcp",
    max_loops=2,
)

if __name__ == "__main__":
    result = agent.run(
        "Scrape https://modelcontextprotocol.io/introduction and explain, in "
        "the docs' own terms, what problem MCP solves and what the three "
        "core primitives are. Quote the definitions."
    )
    print(result)

Which Firecrawl tool for which job

Tool	Use it when	Cost
`firecrawl_scrape`	You know the URL that answers the question.	one page
`firecrawl_map`	You need the site’s URL structure before deciding what to read.	cheap
`firecrawl_search`	You need to find pages on a topic first.	per search
`firecrawl_extract`	You want structured fields out of a page, not prose.	one page
`firecrawl_crawl`	You genuinely need breadth across a site.	per page, unbounded

The order matters when you write the task: map then scrape is usually both cheaper and more accurate than crawl, because you choose which pages get read.

Pairing it with search

Firecrawl reads pages well but finding the right page is a different job. A common two-stage setup gives the finder and the reader their own servers:

from swarms import Agent, SequentialWorkflow

finder = Agent(
    agent_name="Finder",
    model_name="gpt-5.4",
    mcp_url=f"https://mcp.exa.ai/mcp?exaApiKey={os.getenv('EXA_API_KEY')}",
    max_loops=2,
)

reader = Agent(
    agent_name="Reader",
    model_name="claude-opus-5",
    mcp_url=f"https://mcp.firecrawl.dev/{FIRECRAWL_API_KEY}/v2/mcp",
    max_loops=2,
)

workflow = SequentialWorkflow(agents=[finder, reader], max_loops=1)

See MCP in a multi-agent workflow for why one server per agent beats giving both servers to one agent.

Source: examples/mcp/agents/10_firecrawl_web_scraping.py

Firecrawl via swarms-tools — the crawl-a-site tool without MCP.
Authentication patterns — all five ways a server can take a credential.

Index

Basic Examples

Model Providers

Single Agent

Multi-Agent Examples

Auto Agent Builder

Scheduled Agents

Applications

Research

Technical Analysis

Use Cases

Finance

Voice Agents

Integrations

MCP Tool Integrations

Deployment

CLI

Firecrawl: scrape pages to markdown

Prerequisites

Build it

The complete script

Which Firecrawl tool for which job

Pairing it with search

Next

​Prerequisites

​Build it

​The complete script

​Which Firecrawl tool for which job

​Pairing it with search

​Next

Prerequisites

Build it

The complete script

Which Firecrawl tool for which job

Pairing it with search

Next