Firecrawl

Recently shipped web-scale /monitor for agent-driven web search and the Research Index for AI/ML literature.

Web-scale /monitor watches the entire web, not just pages. Define search queries and a plain-English goal, and Firecrawl searches on your schedule, alerting via webhook or email when new results match. The same endpoint handles search, deduplication, filtering, and alerts — replaces the DIY stack.

Research Index targets AI/ML research loops. Search 3M+ arXiv papers and GitHub artifacts (issues, merged PRs, READMEs) refreshed daily. 53.3% recall on arXivQA beats the next best provider by 18%. Agents retrieve papers, verify claims against full text, and pull code in one loop. Works with Codex, Claude Code, and any agent harness.

Three token-saving formats for /scrape. highlights returns only matching sentences, code blocks, and table rows (up to 100x fewer tokens). question returns grounded answers with prompt-injection isolation, skipping the scrape-parse-prompt pipeline. deterministicJson returns structured JSON without an LLM per request — generates reusable extractors per schema and site.

Lockdown Mode and /parse for security-first document ingestion. lockdown: true on /scrape serves results exclusively from Firecrawl's index with zero outbound requests and zero data retention. /parse uploads PDFs, Word docs, and spreadsheets up to 50 MB via a Rust engine (up to 5x faster), returning clean Markdown, JSON, or summaries.

Keyless access for core endpoints. /scrape, /search, /interact, and /parse work without an API key from official MCP, CLI, and SDK clients. Automatic PII redaction (redactPII) strips names, emails, phone numbers, addresses, and secrets from scraped content.

The open-source web-agent framework lets agents search, scrape, and interact with the web using any model, with parallel sub-agents and teachable Skill playbooks.

Video discovery on any page. The video format now finds videos on any page, returning URL, title, description, and thumbnail — works for embedded, <video> elements, and streaming platforms.