--- name: GitHub Releases slug: firecrawl-github-releases type: github source_url: https://github.com/firecrawl/firecrawl/releases organization: Firecrawl organization_slug: firecrawl total_releases: 33 latest_version: v2.9.0 latest_date: 2026-04-10 last_updated: 2026-04-18 tracking_since: 2024-09-05 canonical: https://releases.sh/firecrawl/firecrawl-github-releases organization_url: https://releases.sh/firecrawl --- Firecrawl expanded its interactive and agentic capabilities, shifting from static scraping toward dynamic browser control and AI-powered workflows. Version 2.9.0 introduced the `/interact` endpoint for in-browser actions—clicking, form-filling, navigation—with natural language prompts or Playwright code, plus persistent browser profiles for reusing state across sessions. The release also added `query` format for direct natural-language answers and `audio` output support. Version 2.8.0 brought parallel agent execution via the new Spark 1 Fast model, a CLI tool covering scrape through crawl operations, and a Firecrawl Skill for Claude and other AI agents to autonomously invoke the platform. # Firecrawl v2.9.0 ## Improvements - **Browser Interaction via `/interact` endpoint** — Scrape a page, then call `/interact` to take actions on it — click buttons, fill forms, navigate deeper, or extract dynamic content. Describe what you want in natural language via `prompt`, or write Playwright code (Node.js, Python) and Bash (`agent-browser`) for full control. Sessions persist across calls, with live view and interactive live view URLs for real-time browser streaming. Persistent profiles let you save and reuse browser state (cookies, localStorage) across scrapes. Available in JS, Python, Java, and Rust SDKs. - **`query` format** — Added `query` format to the `/scrape` endpoint — pass a natural-language prompt and get a direct answer back in `data.answer`. - **`audio` format** — Added `audio` format option to scrape responses, returning audio output as a field on the document. - **`onlyCleanContent` parameter** — Added `onlyCleanContent` parameter to the `/scrape` endpoint, which strips navigation, ads, cookie banners, and other non-semantic content from markdown output. - **PDF parsing modes** — Added PDF parsing modes (`fast`, `auto`, `ocr`) and a `maxPages` option to control extraction depth and OCR behavior. - **Java and Elixir SDKs** — Added official Java and Elixir SDKs with full v2 API support. - **Legacy `.doc` file support** — Added support for parsing legacy `.doc` files. - **Wikimedia engine** — Added a dedicated engine for scraping Wikipedia and Wikimedia pages with improved output quality. - **`contentType` in scrape responses** — Added `contentType` to scrape responses for PDFs and documents. - **PDF pipeline improvements** — Improved PDF pipeline with better table detection, header/footer stripping, mixed PDF handling, inline image parsing, and magic byte detection. - **Branding extraction** — Improved branding extraction to skip hidden DOM elements for cleaner output. - **HTML-to-markdown performance** — Improved HTML-to-markdown conversion performance and fixed code blocks losing content during conversion. - **Concurrency queue** — New concurrency queue system with reconciler and backfill for more reliable job scheduling. - **Rust SDK v2** — Added v2 API namespace with agent support to the Rust SDK. - Fixed Python SDK parameters `timeout`, `max_retries`, and `backoff_factor` — these were previously accepted but silently ignored. - Capped job timeouts at 48 hours to prevent runaway jobs from consuming resources. - Added retry limits to prevent scrape loops. - Binary content types are now rejected early in the scrape pipeline to avoid wasted processing. ## Fixes - Fixed empty responses when using the `o3-mini` model on extract jobs. - Fixed revoked API keys remaining valid for up to 10 minutes after deletion. - Fixed a race condition in extract jobs that caused "Job not found" crashes. - Fixed `time_taken` in `/v1/map` always returning ~0. - Fixed crawl status responses now surfacing a `failed` status with an error message and partial data when a crawl-level failure occurs. - Fixed `maxPages` not being passed to the PDF extractor — previously, full PDF content was returned while only charging for the limited page count. - Fixed free request credits being incorrectly consumed and billed on agent jobs exceeding the `maxCredits` threshold. - Fixed dashboard displaying incorrect concurrency limits due to stale reads. - Fixed branding `colors.secondary` not being populated. - Fixed `removeBase64Images` running after `deriveDiff` in the transformer pipeline, causing diff issues. - Fixed GCS fetch using wrong row index for cache info lookups. - Fixed unhandled `ZodError` in `/v1/search` controller. - Resolved multiple CVEs across dependencies including `handlebars`, `path-to-regexp`, `fast-xml-parser`, `rollup` (CVE-2026-27606), `undici`, and others. - Hardened the Playwright service against SSRF attacks. ## API - Added `GET /v2/team/activity` endpoint for listing recent scrape, crawl, and extract jobs with cursor-based pagination (last 24 hours, up to 100 results per page, filterable by endpoint type). - Added `regexOnFullURL` parameter on crawl requests to apply `includePaths`/`excludePaths` filtering against the full URL including query parameters. Available in JS, Python, Java, and Elixir SDKs. - Added `deduplicateSimilarURLs` parameter on crawl requests. Available in JS, Python, Java, and Elixir SDKs. - Deprecated the `extract` endpoint — use the `/agent` endpoint instead. Existing `extract` methods in JS and Python SDKs are marked deprecated. - Renamed `persistentSession` to `profile` on browser/interact requests (`writeMode` is now `saveChanges`). The old parameter name remains functional but is no longer documented. --- ## New Contributors * @misza-one made their first contribution in https://github.com/firecrawl/firecrawl/pull/2660 * @madmikeross made their first contribution in https://github.com/firecrawl/firecrawl/pull/2948 * @rowinsg made their first contribution in https://github.com/firecrawl/firecrawl/pull/3065 * @Bortlesboat made their first contribution in https://github.com/firecrawl/firecrawl/pull/3243 * @dagecko made their first contribution in https://github.com/firecrawl/firecrawl/pull/3249 * @cokemine made their first contribution in https://github.com/firecrawl/firecrawl/pull/3262 * @paulonasc made their first contribution in https://github.com/firecrawl/firecrawl/pull/3275 ## Contributors * @nickscamara * @mogery * @amplitudesxd * @abimaelmartell * @ericciarla * @rafaelsideguide * @delong3 * @devhims * @Chadha93 * @tomsideguide * @charlietlamb * @developersdigest * @micahstairs * @rhys-firecrawl * @firecrawl-spring * @devin-ai-integration * @misza-one * @madmikeross * @rowinsg * @Bortlesboat * @dagecko * @cokemine * @paulonasc --- **Full Changelog**: https://github.com/firecrawl/firecrawl/compare/v2.8.0...v2.9.0 # v2.8.0 ## Firecrawl v2.8.0 is here! Firecrawl v2.8.0 brings major improvements to agent workflows, developer tooling, and self-hosted deployments across the API and SDKs, including our new Skill. * Parallel Agents for running thousands of `/agent` queries simultaneously, powered by our new Spark 1 Fast model. * Firecrawl CLI with full support for scrape, search, crawl, and map commands. * Firecrawl Skill for enabling AI agents (Claude Code, Codex, OpenCode) to use Firecrawl autonomously. * Three new models powering /agent: Spark 1 Fast for instant retrieval (currently only available in Playground), Spark 1 Mini for complex research queries, and Spark 1 Pro for advanced extraction tasks. * Agent enhancements including webhooks, model selection, and new MCP Server tools. * Platform-wide performance improvements including faster search execution and optimized Redis calls. * SDK improvements including Zod v4 compatibility. And much more, check it out below! ## New Features * **Parallel Agents** Execute thousands of `/agent` queries in parallel with automatic failure handling and intelligent waterfall execution. Powered by Spark 1-Fast for instant retrieval, automatically upgrading to Spark 1 Mini for complex queries requiring full research. * **Firecrawl CLI** New command-line interface for Firecrawl with full support for scrape, search, crawl, and map commands. Install with `npm install -g firecrawl-cli`. * **Firecrawl Skill** Enables agents like Claude Cursor, Codex, and OpenCode to use Firecrawl for web scraping and data extraction, installable via `npx skills add firecrawl/cli.` * **Spark Model Family** Three new models powering /agent: **Spark 1 Fast** for instant retrieval (currently available in Playground), **Spark 1 Mini** (default) for everyday extraction tasks at 60% lower cost, and **Spark 1 Pro** for complex multi-domain research requiring maximum accuracy. Spark 1 Pro achieves ~50% recall while Mini delivers ~40% recall, both significantly outperforming tools costing 4-7x more per task. * **Firecrawl MCP Server Agent Tools** New `firecrawl_agent` and `firecrawl_agent_status` tools for autonomous web data gathering via MCP-enabled agents. * **Agent Webhooks** Agent endpoint now supports webhooks for real-time notifications on job completion and progress. * **Agent Model Selection** Agent endpoint now accepts a `model` parameter and includes model info in status responses. * **Multi-Arch Docker Images** Self-hosted deployments now support `linux/arm64` architecture in addition to `amd64`. * **Sitemap-Only Crawl Mode** New crawl option to exclusively use sitemap URLs without following links. * **`ignoreCache` Map Parameter** New option to bypass cached results when mapping URLs. * **Custom Headers for `/map`** Map endpoint now supports custom request headers. * **Background Image Extraction** Scraper now extracts background images from CSS styles. * **Improved Error Messages** All user-facing error messages now include detailed explanations to help diagnose issues. --- ## API Improvements * Search without concurrency limits — scrapes in search now execute directly without queue overhead. * Return `400` for unsupported actions with clear errors when requested actions aren't supported by available engines. * Job ID now included in search metadata for easier tracking. * Metadata responses now include detected timezone. * Backfill metadata title from `og:title` or `twitter:title` when missing. * Preserve `gid` parameter when rewriting Google Sheets URLs. * Fixed v2 path in batch scrape status pagination. * Validate team ownership when appending to existing crawls. * Screenshots with custom viewport or quality settings now bypass cache. * Optimized Redis calls across endpoints. * Reduced excessive `robots.txt` fetching and parsing. * Minimum request timeout parameter now configurable. --- ## SDK Improvements ### JavaScript SDK * **Zod v4 Compatibility** — schema conversion now works with Zod v4 with improved error detection. * **Watcher Exports** — `Watcher` and `WatcherOptions` now exported from the SDK entrypoint. * **Agent Webhook Support** — new webhook options for agent calls. * **Error Retry Polling** — SDK retries polling after transient errors. * **Job ID in Exceptions** — error exceptions now include `jobId` for debugging. ### Python SDK * Manual pagination helpers for iterating through results. * Agent webhook support added to agent client. * Agent endpoint now accepts model selection parameter. * Metadata now includes concurrency limit information. * Fixed `max_pages` handling in crawl requests. --- ## Dashboard Improvements * Dark mode is now supported. * On the usage page, you can now view credit usage broken down by day. * On the activity logs page, you can now filter by the API key that was used. * The "images" output format is now supported in the Playground. * All admins can now manage their team's subscriptions. --- ## Quality & Performance * Skip markdown conversion checks for large HTML documents. * Export Google Docs as HTML instead of PDF for improved performance. * Improved branding format with better logo detection and error messages for PDFs and documents. * Improved `lopdf` metadata loading performance. * Updated `html-to-markdown` module with multiple bug fixes. * Increased markdown service body limit and added request ID logging. * Better Sentry filtering for cancelled jobs and engine errors. * Fixed extract race conditions and RabbitMQ poison pill handling. * Centralized Firecrawl configuration across the codebase. * Multiple security vulnerability fixes, including CVE-2025-59466 and lodash prototype pollution. --- ## Self-Hosted Improvements * CLI custom API URL support via `firecrawl --api-url http://localhost:3002` for local instances. * ARM64 Docker support via multi-arch images for Apple Silicon and ARM servers. * Fixed docker-compose database credentials out of the box. * Fixed Playwright service startup caused by Chromium path issues. * Updated Node.js to major version 22 instead of a pinned minor. * Added RabbitMQ health check endpoint. * Fixed PostgreSQL port exposure in docker-compose. --- ## New Contributors * @gemyago * @loganaden * @pcgeek86 * @dmlarionov --- **Full Changelog**: https://github.com/firecrawl/firecrawl/compare/v2.7.0...v2.8.0 ## What's Changed * refactor(api): centralize firecrawl config by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2496 * fix(config): add .catch to NUQ worker port defaults for error handling by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2505 * (sdk)fix/same timeout as api now by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2503 * (sdks)feat/added concurrency info to metadata by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2502 * fix: make srcset URLs absolute in HTML transformation by @Chadha93 in https://github.com/firecrawl/firecrawl/pull/2515 * feat(api/admin/crawl-monitor): add endpoint for monitoring crawl system by @mogery in https://github.com/firecrawl/firecrawl/pull/2518 * feat(api/logRequest): associate requests with API keys by @mogery in https://github.com/firecrawl/firecrawl/pull/2519 * Fix Config Load on Tests by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2506 * feat(api): update model usage to gpt-4o-mini by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2520 * feat(api/scrapeURL): engpicker integ by @mogery in https://github.com/firecrawl/firecrawl/pull/2523 * fix(playwright-service-ts): wasn't starting up due to the lack of chromium under /tmp/.cache by @dmlarionov in https://github.com/firecrawl/firecrawl/pull/2512 * added timezone to metadata response by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2526 * Increase Go Service Write Timeout by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2489 * (python-sdk)fix/max_pages by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2527 * Use invoiced billing for certain expansion packs by @micahstairs in https://github.com/firecrawl/firecrawl/pull/2532 * Fix PostgreSQL port exposure in docker-compose by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2530 * (feat/partners) Allow email to be optional for partners API by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2533 * Advanced model for recursive schemas by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2535 * feat(api): update gpt-4o usage to gpt-4.1 by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2536 * fix(api): cost tracking by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2537 * Update Sentry for ZDR compliance by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2529 * sanitize null-byte strings and report robustInsert failures to Sentry by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2538 * Dont log Feature Flog Errors to Sentry by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2540 * Debug logs to Extract Updates by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2539 * Update test site build by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2543 * feat: increase precrawl limits by @delong3 in https://github.com/firecrawl/firecrawl/pull/2544 * fix(api): engines for robots and scrape + reduced sitemap limit by @delong3 in https://github.com/firecrawl/firecrawl/pull/2545 * Webhook dispatcher by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2534 * (feat/partner-integrations) Rotate endpoint by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2547 * fix: extract race condition by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2548 * fix: correct property name from 'success' to 'is_successful' by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2550 * feat: update root endpoint to return JSON with documentation URL by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2552 * Branding Format Improvements by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2438 * Add more Sentry filtering by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2556 * Revert "feat(crawl): implement URL modification handling in crawl con… by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2558 * fix: export URL schema for external usage by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2559 * feat: add configurable harness startup timeout by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2560 * fix(api): scrapeURL/index-metrics logging by @delong3 in https://github.com/firecrawl/firecrawl/pull/2561 * fix(api): excessive robots.txt fetching/parsing by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2562 * feat(api): optimize redis calls by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2564 * Add strictJsonSchema on Branding Format LLM Call by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2567 * Fix docker-compose db credentials by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2568 * Fix Deps Audit by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2569 * feat(api): ab/b by @mogery in https://github.com/firecrawl/firecrawl/pull/2570 * feat(api): ab/bs by @mogery in https://github.com/firecrawl/firecrawl/pull/2571 * feat(api): ab/bsrch by @mogery in https://github.com/firecrawl/firecrawl/pull/2572 * Update html-to-markdown module by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2573 * python-sdk: Update Agent Client by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2579 * fix(extract): rabbitmq + poison pill handling by @mogery in https://github.com/firecrawl/firecrawl/pull/2581 * chore(api): re-route fire-1 by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2584 * feat: add bypassCreditChecks team flag for infinite graceful credit checks by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2578 * fix(api/search): saner billing logic by @mogery in https://github.com/firecrawl/firecrawl/pull/2585 * feat(api): custom header support for /map by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2593 * fix(api): index metrics backwards compat by @delong3 in https://github.com/firecrawl/firecrawl/pull/2598 * Add redis.sadd("billed_teams", team_id) to clearACUCTeam for centralized tracking by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2602 * Throw error when v3-beta is passed into /extract by @micahstairs in https://github.com/firecrawl/firecrawl/pull/2601 * Fixes #2583 add RabbitMQ health check and update API dependencies by @pcgeek86 in https://github.com/firecrawl/firecrawl/pull/2605 * js-sdk: Retry polling after errors, add jobId to error exception by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2608 * Allow formats on agent schema by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2603 * feat(api): extract background images by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2611 * Don't log EngineError to Sentry by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2613 * Ignore Cancelled Jobs on Sentry by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2614 * feat(api): include jobId in search response metadata by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2618 * Update html-to-markdown version by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2620 * Add request_id to Markdown Logs by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2621 * Fix NPM Audit by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2623 * Add error details to Go Markdown Service by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2624 * Add Request ID to Markdown Service Logger by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2630 * Increase Markdown Service Body Limit by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2629 * Multi-arch images for playwright and api with linux/arm64 support by @gemyago in https://github.com/firecrawl/firecrawl/pull/2555 * fix(nuq): zombie pg clients by @mogery in https://github.com/firecrawl/firecrawl/pull/2633 * Webhooks for agent by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2628 * Update firecrawl/html-to-markdown by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2634 * fix: add Number() coercion to prevent string concatenation in credit calculations by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2635 * Revert "fix: add Number() coercion to prevent string concatenation in credit calculations" by @devhims in https://github.com/firecrawl/firecrawl/pull/2636 * fix(api): validate team ownership when appending to an existing crawl by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2637 * Add debug logging for 402 credit check failures by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2638 * Implement num_results for searxng by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2645 * Bump auth_credit_usage_chunk_38 to auth_credit_usage_chunk_39 by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2646 * Attempt fix infinite loop by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2644 * Update lopdf to use load_metadata method for better performance by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2642 * fix: backfill metadata title from og:title or twitter:title when missing by @devhims in https://github.com/firecrawl/firecrawl/pull/2650 * fix(security): fix audit-ci vulnerabilities and clean up allowlists by @mogery in https://github.com/firecrawl/firecrawl/pull/2657 * chore(api): export gdocs as HTML instead of PDF by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2652 * Skip Markdown Check for Big HTML Documents by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2622 * feat(sdk): add model parameter to agent endpoint by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2663 * fix(security): add new hono vulnerabilities to audit allowlist by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2664 * Update Audit Workflow by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2667 * fix(nuq): align RabbitMQ expiration with lock reaper to prevent oscillation by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2672 * fix(nuq-postgres): tune checkpoint and autovacuum to reduce job prefetch stalls by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2673 * feat(billing): skip credit checks for organization teams by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2674 * fix(cache): skip cache for screenshots with custom viewport or quality settings by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2677 * .github/workflows: Migrate workflows to Blacksmith runners by @blacksmith-sh[bot] in https://github.com/firecrawl/firecrawl/pull/2680 * fix(ci): allowlist new vulnerabilities in npm audit by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2681 * fix(nuq-postgres): aggressive checkpoint tuning to prevent queue stalls by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2679 * chore(ci): tune runners by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2683 * chore(ci): tune runners by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2685 * fix(js-sdk): detect mistaken use of Zod schema.shape and provide helpful error by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2684 * fix(ci): allowlist low severity npm audit vulnerabilities by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2682 * fix(ci): add new vulnerabilities to audit allowlists by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2688 * feat(api): add ignoreCache parameter to map endpoint by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2686 * chore(api): update A/B test to use FIRE_ENGINE_AB_URL instead of FIRE… by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2689 * chore: update audit runner from 4c to 2c by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2690 * fix(scraper): preserve postprocessor markdown in transformer by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2694 * Fix CVE-2025-59466 by @loganaden in https://github.com/firecrawl/firecrawl/pull/2695 * fix(docker): use node 22 major version instead of pinned 22.22 by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2696 * feat(search): remove NuQ queue and execute scrapes directly without concurrency limits by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2668 * fix: preserve gid parameter when rewriting Google Sheets URLs by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2693 * feat(api): fire-engine action metadata by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2699 * Don't Expose Internal Errors by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2700 * docs: fix typos in README by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2703 * feat(sdk): add agent webhook support to Node.js and Python SDKs by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2705 * Return 400 when actions are requested but no engines support then by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2704 * feat: add fire engine A/B comparison by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2706 * chore: remove DB webhook logic by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2707 * fix: a/b test logic by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2708 * feat: allow slight variance on test comparison by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2709 * feat: HTML to markdown conversion in A/B comparison by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2714 * feat: request timeout parameter min by @delong3 in https://github.com/firecrawl/firecrawl/pull/2710 * feat: word jaccard diff by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2720 * fix(api): use correct v2 path in batch scrape status next URL by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2722 * fix(js-sdk): add Zod v4 compatibility for schema conversion by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2724 * feat(crawl): add sitemap-only support by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2726 * fix(deps): resolve lodash prototype pollution vulnerability by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2728 * Add manual pagination helpers for Python SDK by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2727 * feat(agent): include model in status responses by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2717 * feat(errors): improve all user-facing error messages with detailed explanations by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2697 * chore(codeowners): add abimaelmartell as owner for branding files by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2735 * fix: handle 415 Unsupported Media Type without retrying by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2737 * fix(branding): use spread operator instead of Array.from for Set conversion by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2746 * fix(js-sdk): export Watcher and WatcherOptions from SDK entrypoint by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2754 * fix(audit): Fix pnpm audit issues by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2757 * fix(audit): Upgrade eslint to fix GHSA-p5wg-g6qr-c7cg vulnerability in ingestion-ui by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2765 * feat(api): add enhanced proxy option as alias for stealth by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2759 * feat(api): only load more pages if we already have one by @tomsideguide in https://github.com/firecrawl/firecrawl/pull/2767 * fix(api): update notification email from address by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2769 * conc boost for agent interop by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2770 * chore(api): Remove jest-junit test reporting by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2768 * chore(api): Add logging to flaky map redirect test by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2771 * fix(branding): Improve Logo Detection by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2749 * feat(api): promote jobs in concurrency queue backfill by @tomsideguide in https://github.com/firecrawl/firecrawl/pull/2773 * fix(security): allowlist fast-xml-parser vulnerability GHSA-37qj-frw5-hhjh by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2775 * fix(branding): improve error messages for PDFs and documents by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2776 * chore: remove firecrawl_jobs cleanup from ZDR cleaner by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2777 * fix(billing): bump to update_tally_8_team by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2779 * fix(api): add 'enhanced' to proxy enum in OpenAPI specs by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2780 * chore: bump auth_credit_usage_chunk_39 to auth_credit_usage_chunk_40 by @firecrawl-spring[bot] in https://github.com/firecrawl/firecrawl/pull/2782 * feat(rust-sdk): add v2 API namespace with agent support by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2778 ## Firecrawl v2.7.0 is here! - ZDR Search support for enterprise customers. - Improved Branding Format with better detection. - Partner Integrations API now in closed beta. - Faster and more accurate screenshots. - Self-hosted improvements And a lot more enhacements, check it out below! ## **New Features** - **Improved Branding Extract** Better logo and color detection for more accurate brand extraction results. - **NOQ Scrape System (Experimental)** New scrape pipeline with improved stability and integrated concurrency checks. - **Enhanced Redirect Handling** URLs now resolve before mapping, with safer redirect-chain detection and new abort timeouts. - **Enterprise Search Parameters** New enterprise-level options available for the `/search` endpoint. - **Integration-Based User Creation** Users can now be automatically created when coming from referring integrations. - **`minAge` Scrape Parameter** Allows requiring a minimum cached age before re-scraping. - **Extract Billing Credits** Extract jobs now use the same credit billing system as other endpoints. - **Self-Host: Configurable Crawl Concurrency** Self-hosted deployments can now set custom concurrency limits. - **Sentry Enhancements** Added Vercel AI integration, configurable sampling rates, and improved exception filtering. - **UUIDv7 IDs** All new resources use lexicographically sortable UUIDv7. ## **API Improvements** - **DNS Resolution Errors Now Return 200** for more consistent failure handling. - **Improved URL Mapping Logic** including sitemap `maxAge` fixes, recursive sitemap support, Vue/Angular router normalization, and skipping subdomain logic for IP addresses. - **Partial Results for Multi-Source Search** instead of failing all sources. - **Concurrency Metadata Added** to scrape job responses. - **Enhanced Metrics** including total wait time, LLM usage, and format details. - **Batch Scrape Upgrades** - Added missing `/v2/batch/scrape/:jobId/errors` endpoint - Fixed pagination off-by-one bug - **More Robust Error Handling** for PDF/document engines, pydantic parsing, Zod validation, URL validation, and billing edge cases. --- ## **SDK Improvements** ### **JavaScript SDK** - Returns **job ID** from synchronous methods. - Improved **WebSocket `document` event** handling. - Fixed **types**, **Deno WS**, and added support for `ignoreQueryParameter`. - Version bump with internal cleanup. ### **Python SDK** - Added extra metadata fields. - Improved batch validation handling. --- ## **Quality & Performance** - Reduced log file size and improved tmp file cleanup. - Updated Express version and patched vulnerable packages. - Disabled markdown conversion for sitemap scrapes for improved performance. - Better precrawl logging and formatting. - Skip URL rewriting for published Google Docs. - Prevent empty cookie headers during webhook callbacks. --- ## **Self-Hosted Improvements** - Disabled concurrency limit enforcement for self-hosted mode. - PostgreSQL credentials now configurable via environment variables. - Docker-compose build instructions fixed. --- ## 👥 New Contributors - @omahs - @davidkhala - @DraPraks - @devhims --- **Full Changelog:** https://github.com/firecrawl/firecrawl/compare/v2.6.0...v2.7.0 ## What's Changed * (feat/dns) DNS Resolution errors should be a 200 by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2402 * Improve Logo and Color Detection on Branding Extract by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2362 * (js-sdk) fix: ws 'document' event implementation by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2415 * (js-sdk): Return job ID from synchronous methods by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2414 * (js-sdk): Fix types by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2416 * (js-sdk): Bump Version by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2417 * fix(api): disable gcs logging without db auth by @delong3 in https://github.com/firecrawl/firecrawl/pull/2418 * feat: noq scrape system by @delong3 in https://github.com/firecrawl/firecrawl/pull/2419 * Muv2 exp add more logs by @tomkosm in https://github.com/firecrawl/firecrawl/pull/2421 * feat(api): noq concurrency check integration by @delong3 in https://github.com/firecrawl/firecrawl/pull/2424 * Fix concurrency backfill bug by @micahstairs in https://github.com/firecrawl/firecrawl/pull/2425 * feat: redirect to docs when hitting main api endpoint by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2426 * feat(api): total wait time in request metrics by @delong3 in https://github.com/firecrawl/firecrawl/pull/2428 * (fix/search) rm legacy external search apis by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2420 * fix(api): various vulnerable packages (2025/11/21) by @mogery in https://github.com/firecrawl/firecrawl/pull/2431 * fix(api): pdf + document engines not respecting skipTlsVerification flag and error handling for uncidi by @delong3 in https://github.com/firecrawl/firecrawl/pull/2435 * fix(api): /map returning less urls with sitemap include by @delong3 in https://github.com/firecrawl/firecrawl/pull/2440 * feat: resolve redirects before mapping urls by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2439 * fix(api): tally system rework by @mogery in https://github.com/firecrawl/firecrawl/pull/2430 * fix(api): update URL handling of resolved redirects by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2442 * fix(api): opaque fire engine delete + poll interval by @delong3 in https://github.com/firecrawl/firecrawl/pull/2443 * feat(api): add abort timeout for resolveRedirects by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2444 * (python-sdk) feat: added extra fields to metadata by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2441 * fix(api): handle case with no billed teams in tallyBilling function by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2446 * fix: Add support for ignoreQueryParameter in map SDKs by @Chadha93 in https://github.com/firecrawl/firecrawl/pull/2429 * fix(api): vue + angular router url normalization by @delong3 in https://github.com/firecrawl/firecrawl/pull/2447 * feat(api): usedLlm + formats in request metrics by @delong3 in https://github.com/firecrawl/firecrawl/pull/2448 * feat(api): switch to uuidv7 by @mogery in https://github.com/firecrawl/firecrawl/pull/2449 * Add Sentry Settings by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2451 * Filter Sentry Exceptions by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2453 * Add minAge parameter to scrape (ENG-4073) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2452 * Fix typos by @omahs in https://github.com/firecrawl/firecrawl/pull/2457 * fix docker-compose service build instructions by @davidkhala in https://github.com/firecrawl/firecrawl/pull/2406 * Cleanup tmp files from downloadFile by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2455 * Optimize Logs File Size by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2456 * Update express version by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2465 * Annotate test failures on CI by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2462 * fix(api/precrawl): precrawl logging + format + skip index by @delong3 in https://github.com/firecrawl/firecrawl/pull/2466 * fix(go-html-to-md): request body max 60MB by @delong3 in https://github.com/firecrawl/firecrawl/pull/2467 * fix(api): dns + crawl denial errors by @delong3 in https://github.com/firecrawl/firecrawl/pull/2469 * Disable markdown conversion for sitemap scrapes by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2461 * Add missing /v2/batch/scrape/:jobId/errors endpoint by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2471 * fix: improve pydantic parsing error handling | ENG-4070 by @Chadha93 in https://github.com/firecrawl/firecrawl/pull/2450 * fix: Make PostgreSQL credentials configurable via environment variables by @DraPraks in https://github.com/firecrawl/firecrawl/pull/2388 * feat(api): create users via referring integrations by @mogery in https://github.com/firecrawl/firecrawl/pull/2463 * feat: muv2 exp apikey env by @tomkosm in https://github.com/firecrawl/firecrawl/pull/2472 * (feat/search) Enterprise params by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2412 * Validate UUID from URL in Requests by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2392 * Disable Concurrency Limit on Self Hosted by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2475 * (js sdk)fix/ws deno by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2476 * fix(api): sitemap max age for map requests by @delong3 in https://github.com/firecrawl/firecrawl/pull/2479 * fix(api): sitemap max age for recursive sitemaps by @delong3 in https://github.com/firecrawl/firecrawl/pull/2480 * feat: new app database shape by @mogery in https://github.com/firecrawl/firecrawl/pull/2445 * chore(api): disable x-powered-by by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2483 * Skip subdomain logic for IP addresses by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2477 * Attempt Fix Search Tests by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2478 * fix(api): don't bill where stealth proxy was unsupported by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2484 * feat(extract): port to billing credits by @mogery in https://github.com/firecrawl/firecrawl/pull/2482 * feat: [self-host] - add support to configure concurrency for crawl by @Chadha93 in https://github.com/firecrawl/firecrawl/pull/2193 * Update AI SDK to Latest Version by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2369 * Fix Zod Error Handling on V0 by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2488 * fix: off-by-one bug in batch scrape pagination by @devhims in https://github.com/firecrawl/firecrawl/pull/2492 * fix: make auto-recharge email show actual credit amount instead of hardcoded 1000 by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2485 * fix: allow partial results when searching multiple sources by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2490 * gitignore test results xml by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2493 * Fix latest advisory issues by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2494 * feat(api): add sentry vercel ai integration by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2473 * feat(api/sentry): make sampling rates configurable via environment va… by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2495 * Python sdk fix/batch validate limit by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2399 * chore(sentry): set default TRACE_SAMPLE_RATE to 0 by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2498 * chore(api): disable vercel input/output tracing by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2499 * feat(api): concurrency limit info in scrape metadata by @delong3 in https://github.com/firecrawl/firecrawl/pull/2497 * engpicker POC by @mogery in https://github.com/firecrawl/firecrawl/pull/2501 * fix: skip URL rewriting for published Google Docs by @devhims in https://github.com/firecrawl/firecrawl/pull/2500 * fix: prevent empty cookie header in webhook callbacks by @devhims in https://github.com/firecrawl/firecrawl/pull/2504 ## New Contributors * @omahs made their first contribution in https://github.com/firecrawl/firecrawl/pull/2457 * @davidkhala made their first contribution in https://github.com/firecrawl/firecrawl/pull/2406 * @DraPraks made their first contribution in https://github.com/firecrawl/firecrawl/pull/2388 * @devhims made their first contribution in https://github.com/firecrawl/firecrawl/pull/2492 **Full Changelog**: https://github.com/firecrawl/firecrawl/compare/v2.6.0...v2.7.0 # v2.6.0 ## Highlights - **Unified Billing Model** - Credits and tokens merged into single system. Extract now uses credits (15 tokens = 1 credit), existing tokens work everywhere. - **Full Release of Branding Format** - Full support across Playground, MCP, JS and Python SDKs. - **Change Tracking** - Faster and more reliable detection of web page content updates. - **Reliability and Speed Improvements** - All endpoints significantly faster with improved reliability. - **Instant Credit Purchases** - Buy credit packs directly from dashboard without waiting for auto-recharge. - **Improved Markdown Parsing** - Enhanced markdown conversion and main content extraction accuracy. - **Core Stability Fixes** - Fixed change-tracking issues, PDF timeouts, and improved error handling. ## What's Changed * fix(mu): Bug fix on v2 exp by @tomkosm in https://github.com/firecrawl/firecrawl/pull/2345 * Allow index use with waitFor (ENG-3481) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2346 * Fix autoCharge return, add top level guard by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2341 * fix: import MAX_MAP_LIMIT from types.ts to resolve 1000 URL cap by @prashu0705 in https://github.com/firecrawl/firecrawl/pull/2333 * chore: improve llm extract logging by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2348 * fix: error truncation by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2349 * feat(go-html-to-md): enhance markdown conversion with robust PRE and … by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2321 * feat(billing): merge credits and tokens by @mogery in https://github.com/firecrawl/firecrawl/pull/2352 * chore: update geoip database by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2354 * Implement Branding Format by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2326 * Filter non-HTTP(S) protocols with separate error message by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2357 * Add branding format support to JS and Python SDKs by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2360 * Fix: Handle invalid favicon URLs gracefully in metadata extraction by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2361 * feat: allow disabling webhook delivery by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2367 * feat: add engine forcing by domain pattern by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2371 * revert nuq commits by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2376 * CI: Remove npm audit from server tests by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2385 * ci: Fix dependency audit by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2386 * (fix/ctracking) Fix change tracking issues by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2391 * update: Adds support for recursive schema for `python-sdk` with model selection by @Chadha93 in https://github.com/firecrawl/firecrawl/pull/2266 * fix: image search field mapping in Python SDK by @naaa760 in https://github.com/firecrawl/firecrawl/pull/2244 * fix(api/scrape): document + pdf scrape loop by @delong3 in https://github.com/firecrawl/firecrawl/pull/2396 ## New Contributors * @prashu0705 made their first contribution in https://github.com/firecrawl/firecrawl/pull/2333 * @naaa760 made their first contribution in https://github.com/firecrawl/firecrawl/pull/2244 **Full Changelog**: https://github.com/firecrawl/firecrawl/compare/v2.5.0...v2.6.0 ## v2.5.0 - The World's Best Web Data API ## v2.5.0 - The World's Best Web Data API We now have the highest quality and most comprehensive web data API available powered by our new semantic index and custom browser stack. See the benchmarks below: image ## New Features - Implemented scraping for `.xlsx` (Excel) files. - Introduced new crawl architecture and NUQ concurrency tracking system. - Per-owner/group concurrency limiting + dynamic concurrency calculation. - Added group backlog handling and improved group operations. - Added `/search` pricing update - Added team flag to skip country check. - Always populate NUQ metrics for improved observability. - New test-site app for improved CI testing. - Extract metadata from document head for richer output. ### Enhancements & Improvements - Improved blocklist loading and unsupported site error messages. - Updated x402-express version. - Improved includePaths handling for subdomains. - Updated self-hosted search to use DuckDuckGo. - JS & Python SDKs no longer require API key for self-hosted deployments. - Python SDK timeout handling improvements. - Rust client now uses `tracing` instead of `print`. - Reduced noise in auto-recharge Slack notifications. ### Fixes - Ensured crawl robots.txt warnings surface reliably. - Resolved concurrency deadlocks and duplicate job handling. - Fixed search country defaults and pricing logic bugs. - Fixed port conflicts in harness environments. - Fixed viewport dimension support and screenshot behavior in Playwright. - Resolved CI test flakiness (playwright cache, prod tests). ### 👋 New Contributors - @delong3 - @c4nc - @codetheweb **Full diff:** https://github.com/firecrawl/firecrawl/compare/v2.4.0...v2.5.0 ## What's Changed * More verbose blocklist loading errors by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2277 * Update x402-express Version by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2279 * Revise unsupported site error message by @micahstairs in https://github.com/firecrawl/firecrawl/pull/2286 * feat: index precrawl by @delong3 in https://github.com/firecrawl/firecrawl/pull/2289 * fix: ensure includePaths apply to subdomains when allowSubdomains is enabled by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2278 * Fix search country parameter to default to undefined when location is set by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2283 * Fix Port Conflict in Harness by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2285 * js-sdk: require API key only for cloud API (not self-hosted) by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2237 * feat: Implement Scraping Excel xlsx files by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2284 * feat(nuq): concurrency tracking by @mogery in https://github.com/firecrawl/firecrawl/pull/2291 * fix(crawl): surface robots.txt warning reliably by @ftonato in https://github.com/firecrawl/firecrawl/pull/2287 * feat(nuq): add source for max_concurrency by @mogery in https://github.com/firecrawl/firecrawl/pull/2293 * feat(nuq/concurrency-tracking): fix deadlock by @mogery in https://github.com/firecrawl/firecrawl/pull/2295 * Replace self-hosted Google with DDG search (ENG-3499) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2225 * python-sdk: Fix timeout handling across api calls by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2288 * python-sdk: Don't require API Key when running Self Hosted by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2290 * Add team flag to skip country check by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2300 * Update /search endpoint pricing to 2 credits per 10 search results by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2299 * Fix search pricing bug by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2301 * feat(nuq): per-owner-per-group concurrency limiting by @mogery in https://github.com/firecrawl/firecrawl/pull/2302 * update: handle circular refs as well in recursive schema by @Chadha93 in https://github.com/firecrawl/firecrawl/pull/2298 * feat(nuq): dynamically calculate current concurrency by @mogery in https://github.com/firecrawl/firecrawl/pull/2305 * feat(nuq): group_id, job backlogs, and group add operations by @mogery in https://github.com/firecrawl/firecrawl/pull/2309 * feat(ci): new test-site app + updated jest tests by @delong3 in https://github.com/firecrawl/firecrawl/pull/2312 * feat: new crawl architecture by @mogery in https://github.com/firecrawl/firecrawl/pull/2320 * Moved index for backlog query after the table creation by @c4nc in https://github.com/firecrawl/firecrawl/pull/2323 * fix(ci): playwright cache + prod tests by @delong3 in https://github.com/firecrawl/firecrawl/pull/2314 * Improve slack notifications for scale auto-recharges by @micahstairs in https://github.com/firecrawl/firecrawl/pull/2325 * Make auto-recharge notifications less noisy by @micahstairs in https://github.com/firecrawl/firecrawl/pull/2327 * fix: viewport dimension support for Playwright engine screenshots by @ftonato in https://github.com/firecrawl/firecrawl/pull/2329 * feat: always populate nuq metrics by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2328 * fix: scrape viewport test by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2330 * Revert "Merge pull request #2329 from firecrawl/devin/ENG-3639-175924… by @micahstairs in https://github.com/firecrawl/firecrawl/pull/2332 * fix(nuq): per-instance listen channel ID by @mogery in https://github.com/firecrawl/firecrawl/pull/2336 * fix(auto_charge): add a cooldown to the new recharge route by @mogery in https://github.com/firecrawl/firecrawl/pull/2338 * chore: update last scrape rpc by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2339 * Rust client: use `tracing` instead of print by @codetheweb in https://github.com/firecrawl/firecrawl/pull/2324 * Extract metadata from document head (ENG-3822) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2342 * fix(nuq,concurrency-limit): handle if there are duplicate jobs in the concurrency queue by @mogery in https://github.com/firecrawl/firecrawl/pull/2343 ## New Contributors * @delong3 made their first contribution in https://github.com/firecrawl/firecrawl/pull/2289 * @c4nc made their first contribution in https://github.com/firecrawl/firecrawl/pull/2323 * @codetheweb made their first contribution in https://github.com/firecrawl/firecrawl/pull/2324 **Full Changelog**: https://github.com/firecrawl/firecrawl/compare/v2.4.0...v2.5.0 ## v2.4.0 ### New Features - **New PDF Search Category** - You can now search for only pdfs via our v2/search endpoints by specifying .pdf category - **Gemini 2.5 Flash CLI Image Editor** — Create and edit images directly in the CLI using Firecrawl + Gemini 2.5 Flash integration ([#2172](https://github.com/firecrawl/firecrawl/pull/2172)) - **x402 Search Endpoint (`/v2/x402`)** — Added a next-gen search API with improved accuracy and speed ([#2218](https://github.com/firecrawl/firecrawl/pull/2218)) - **RabbitMQ Event System** — Firecrawl jobs now support event-based communication and prefetching from Postgres ([#2230](https://github.com/firecrawl/firecrawl/pull/2230), [#2233](https://github.com/firecrawl/firecrawl/pull/2233)) - **Improved Crawl Status API** — More accurate and real-time crawl status reporting using the new `crawl_status_2` RPC ([#2239](https://github.com/firecrawl/firecrawl/pull/2239)) - **Low-Results & Robots.txt Warnings** — Users now receive clear feedback when crawls are limited by robots.txt or yield few results ([#2248](https://github.com/firecrawl/firecrawl/pull/2248)) - **Enhanced Tracing (OpenTelemetry)** — Much-improved distributed tracing for better observability across services ([#2219](https://github.com/firecrawl/firecrawl/pull/2219)) - **Metrics & Analytics** — Added request-level metrics for both Scrape and Search endpoints ([#2216](https://github.com/firecrawl/firecrawl/pull/2216)) - **Self-Hosted Webhook Support** — Webhooks can now be delivered to private IP addresses for self-hosted environments ([#2232](https://github.com/firecrawl/firecrawl/pull/2232)) ### Improvements - **Reduced Docker Image Size** — Playwright service image size reduced by **1 GB** by only installing Chromium ([#2210](https://github.com/firecrawl/firecrawl/pull/2210)) - **Python SDK Enhancements** — Added `"cancelled"` job status handling and poll interval fixes ([#2240](https://github.com/firecrawl/firecrawl/pull/2240), [#2265](https://github.com/firecrawl/firecrawl/pull/2265)) - **Faster Node SDK Timeouts** — Axios timeouts now propagate correctly, improving reliability under heavy loads ([#2235](https://github.com/firecrawl/firecrawl/pull/2235)) - **Improved Crawl Parameter Previews** — Enhanced prompts and validation for crawl parameter previews ([#2220](https://github.com/firecrawl/firecrawl/pull/2220)) - **Zod Schema Validation** — Stricter API parameter validation with rejection of extra fields ([#2058](https://github.com/firecrawl/firecrawl/pull/2058)) - **Better Redis Job Handling** — Fixed edge cases in `getDoneJobsOrderedUntil` for more stable Redis retrieval ([#2258](https://github.com/firecrawl/firecrawl/pull/2258)) - **Markdown & YouTube Fixes** — Fixed YouTube cache and empty markdown summary bugs ([#2226](https://github.com/firecrawl/firecrawl/pull/2226), [#2261](https://github.com/firecrawl/firecrawl/pull/2261)) - **Updated Docs & Metadata** — README updates and new metadata fields added to the JS SDK ([#2250](https://github.com/firecrawl/firecrawl/pull/2250), [#2254](https://github.com/firecrawl/firecrawl/pull/2254)) - **Improved API Port Configuration** — The API now respects environment-defined ports ([#2209](https://github.com/firecrawl/firecrawl/pull/2209)) ### Fixes - Fixed recursive `$ref` schema validation edge cases ([#2238](https://github.com/firecrawl/firecrawl/pull/2238)) - Fixed enum arrays being incorrectly converted to objects ([#2224](https://github.com/firecrawl/firecrawl/pull/2224)) - Fixed harness timeouts and self-hosted `docker-compose.yaml` issues ([#2242](https://github.com/firecrawl/firecrawl/pull/2242), [#2252](https://github.com/firecrawl/firecrawl/pull/2252)) ### New Contributors - @Chadha93 ([#2155](https://github.com/firecrawl/firecrawl/pull/2155)) - @MAVRICK-1 ([#2172](https://github.com/firecrawl/firecrawl/pull/2172)) - @bernie43 ([#2210](https://github.com/firecrawl/firecrawl/pull/2210)) - @abimaelmartell ([#2209](https://github.com/firecrawl/firecrawl/pull/2209)) - @th3w1zard1 ([#2252](https://github.com/firecrawl/firecrawl/pull/2252)) **🔗 Full Changelog:** [v2.3.0 → v2.4.0](https://github.com/firecrawl/firecrawl/compare/v2.3.0...v2.4.0) ## What's Changed * fix: add missing `poll_interval` param in watcher by @Chadha93 in https://github.com/firecrawl/firecrawl/pull/2155 * feat: Add Firecrawl + Gemini 2.5 Flash Image CLI Editor by @MAVRICK-1 in https://github.com/firecrawl/firecrawl/pull/2172 * Add environment variable to disable blocklist by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2197 * Fix ARM builds by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2198 * fix(v1/search): if f-e search is available, only use that by @mogery in https://github.com/firecrawl/firecrawl/pull/2199 * Upgrade html-to-markdown dependency (ENG-3563) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2195 * feat(map): add crawler and scrape options to job logging by @ftonato in https://github.com/firecrawl/firecrawl/pull/2203 * refactor: integrate facilitator in payment middleware by @ftonato in https://github.com/firecrawl/firecrawl/pull/2213 * (feat/metrics) Scrape and Search Request Metrics by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2216 * (feat/big-query) Big Query by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2217 * feat(api): add x402 search endpoint to /v2 by @ftonato in https://github.com/firecrawl/firecrawl/pull/2218 * feat(api/otel): much improved tracing by @mogery in https://github.com/firecrawl/firecrawl/pull/2219 * fix: Add Zod validation to reject additionalProperties in schema parameters by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2058 * Reduce playwright-service image size by 1 GB by installing only Chromium by @bernie43 in https://github.com/firecrawl/firecrawl/pull/2210 * fix: enum arrays being converted to objects by @Chadha93 in https://github.com/firecrawl/firecrawl/pull/2224 * feat(nuq): RabbitMQ support for job finish events and waiting by @mogery in https://github.com/firecrawl/firecrawl/pull/2230 * fix: Use port from env.PORT for API by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2209 * feat(nuq/rabbitmq): add prefetching jobs from psql to rabbitmq by @mogery in https://github.com/firecrawl/firecrawl/pull/2233 * fix: skip summary generation when markdown is empty by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2226 * Propagate timeout to Axios in Node SDK (ENG-3474) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2235 * feat(api/crawl-status): use crawl_status_2 RPC by @mogery in https://github.com/firecrawl/firecrawl/pull/2239 * Allow self-hosted webhook delivery to private IP addresses by @abimaelmartell in https://github.com/firecrawl/firecrawl/pull/2232 * Update harness timeout by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2242 * python-sdk: include "cancelled" in CrawlJob.status and exit wait loop on cancel (fixes #2190) by @Jeelislive in https://github.com/firecrawl/firecrawl/pull/2240 * feat(api/ci): test with RabbitMQ on prod by @mogery in https://github.com/firecrawl/firecrawl/pull/2241 * (fix/crawl-params) Enhance crawl param preview prompt further by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2220 * build(deps): bump actions/checkout from 3 to 5 by @dependabot[bot] in https://github.com/firecrawl/firecrawl/pull/2115 * fix: harness by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2249 * Fix a self-hosted docker-compose.yaml bug caused by a recent firecrawl change by @th3w1zard1 in https://github.com/firecrawl/firecrawl/pull/2252 * fix: handle `$ref` for recursive schema validation by @Chadha93 in https://github.com/firecrawl/firecrawl/pull/2238 * Add missing metadata fields to JS SDK (ENG-3439) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2250 * Update README.md by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2254 * fix: handle edge case in getDoneJobsOrderedUntil function for Redis job retrieval by @ftonato in https://github.com/firecrawl/firecrawl/pull/2258 * Fix YouTube cache markdown bug by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2261 * feat(api): add warnings for low results and robots.txt restrictions in map and crawl controllers by @ftonato in https://github.com/firecrawl/firecrawl/pull/2248 * Test new mu alternative by @tomkosm in https://github.com/firecrawl/firecrawl/pull/2263 * chore(python-sdk): Bump version to 4.3.7 for poll_interval fix by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2265 * Feat/test new mu alt by @tomkosm in https://github.com/firecrawl/firecrawl/pull/2267 * (feat/search-index) Search Index by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2268 * Feat/test new mu alt by @tomkosm in https://github.com/firecrawl/firecrawl/pull/2270 * (feat/search-index) Separate service by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2271 * fix: additional `queue_scrape` for nuq schema by @Chadha93 in https://github.com/firecrawl/firecrawl/pull/2272 * (feat/search) Pdf search category by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2276 ## New Contributors * @Chadha93 made their first contribution in https://github.com/firecrawl/firecrawl/pull/2155 * @MAVRICK-1 made their first contribution in https://github.com/firecrawl/firecrawl/pull/2172 * @bernie43 made their first contribution in https://github.com/firecrawl/firecrawl/pull/2210 * @abimaelmartell made their first contribution in https://github.com/firecrawl/firecrawl/pull/2209 * @th3w1zard1 made their first contribution in https://github.com/firecrawl/firecrawl/pull/2252 **Full Changelog**: https://github.com/firecrawl/firecrawl/compare/v2.3.0...v2.4.0 ## v2.3.0 ### New Features - **YouTube Support:** You can now get YouTube transcripts - **Enterprise Auto-Recharge:** Added enterprise support for auto-recharge - **odt and .rtf**: Now support odt and rtf file parsing - **Docx Parsing**: 50x faster docx parsing - **K8s Deployment:** Added NuQ worker deployment example - **Self Host**: Tons of improvements for our self host users ### Improvements & Fixes - **Stability:** Fixed timeout race condition, infinite scrape loop, and location query bug - **Tooling:** Replaced ts-prune with knip, updated pnpm with minimumReleaseAge - **Docs:** Added Rust to CONTRIBUTING and fixed typos - **Security:** Fixed `pkgvuln` issue ## What's Changed * Update blocklist by @micahstairs in https://github.com/firecrawl/firecrawl/pull/2150 * docs: fix typo and punctuation in CONTRIBUTING.md by @jarrensj in https://github.com/firecrawl/firecrawl/pull/2149 * Fix timeout error message race condition for ENG-3372 by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2144 * Add exceptions to blocklist by @micahstairs in https://github.com/firecrawl/firecrawl/pull/2156 * fix: pkgvuln by @mogery in https://github.com/firecrawl/firecrawl/pull/2158 * Replace ts prune with knip (ENG-3540) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2148 * feat(auto-recharge): enterprise by @mogery in https://github.com/firecrawl/firecrawl/pull/2127 * feat(scrapeURL/index): index metrics by @mogery in https://github.com/firecrawl/firecrawl/pull/2160 * Update pnpm and add minimumReleaseAge (ENG-3560) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2162 * feat(api/scrapeURL): add special support for YouTube watch pages by @mogery in https://github.com/firecrawl/firecrawl/pull/2157 * fix(scrapeURL/index): locations array querying bug by @mogery in https://github.com/firecrawl/firecrawl/pull/2164 * Fix infinite loop when scraping a forbidden webpage (ENG-3339) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2147 * Add Rust to CONTRIBUTING by @oalsing in https://github.com/firecrawl/firecrawl/pull/2180 * feat(scrapeURL/summary): use gpt-5-mini by @mogery in https://github.com/firecrawl/firecrawl/pull/2174 * Custom Rust document parser (ENG-3489) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2159 * feat: add NuQ worker deployment to Kubernetes examples by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2163 * feat(api): move blocklist to DB by @mogery in https://github.com/firecrawl/firecrawl/pull/2186 ## New Contributors * @jarrensj made their first contribution in https://github.com/firecrawl/firecrawl/pull/2149 * @oalsing made their first contribution in https://github.com/firecrawl/firecrawl/pull/2180 **Full Changelog**: https://github.com/firecrawl/firecrawl/compare/v2.2.0...v2.3.0 ## v2.2.0 ### Features - MCP version 3 is live. Stable support for cloud mcp with HTTP Transport and SSE modes. Compatible with v2 and v1 from. - Webhooks: Now we support signatures + extract support + event failures - Map is now 15x faster + supports more urls - Search reliability improvements - Usage is now tracked by API Key - Support for additional locations (CA, CZ, IL, IN, IT, PL, and PT) - Queue status endpoint - Added `maxPages` parameter to v2 scrape API for pdf parsing ### Improvements - **API:** - New `/team/queue-status` endpoint. - Added `nuq` feature. - Added `VIASOCKET` integration. - Historical credit/token usage endpoints with expanded data. - **Student Program**: Support for more universities + students to get free credits through our student program - **Map**: 15x faster and increased the limit to 100k - **Scrape API:** Added `maxPages` parameter for PDF parser. - **Python SDK:** - Added `get_queue_status` to aio + normalization of docs in search results. - **SDKs:** Added next cursor pagination and integration param support. - **Infrastructure:** Added static IP proxy pool + proxy location support. - **Webhooks:** Implemented signatures, refactored sending, added scrape error events. - **Performance:** Optimized map, converted Rust natives to single NAPI library. - **CI/CD:** Revamped CI, added pre-commit hooks, cross-platform harness. ### 🐛 Fixes - Corrected concurrency limit scaling. - Fixed search result links/descriptions and retry mechanism for empty results. - Re-signed expired screenshot URLs. - Trimmed null chars from PDF titles + fixed encoding. - Fixed sitemap parsing and added `.gz` sitemap support. - Fixed js-sdk `zod-to-json-schema` import. - Fixed webhook data format regression. - Improved credit handling in account object. ### 🛠️ Chores & Other - Removed unused dependencies, updated CONTRIBUTING.md. - Added debug logging, ignored scripts during CI build. - Various dependency bumps and build improvements. **🔗 Full Changelog:** [v2.1.0...v2.2.0](https://github.com/firecrawl/firecrawl/compare/v2.1.0...v2.2.0) ## What's Changed * feat(sdks): next cursor pagination by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2067 * feat: add maxPages parameter to PDF parser in v2 scrape API by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2047 * fix(concurrency-limit): scale! by @mogery in https://github.com/firecrawl/firecrawl/pull/2071 * feat(api): add /team/queue-status endpoint by @mogery in https://github.com/firecrawl/firecrawl/pull/2063 * build(deps): bump actions/checkout from 3 to 5 by @dependabot[bot] in https://github.com/firecrawl/firecrawl/pull/1998 * build(deps): bump actions/setup-python from 4 to 5 by @dependabot[bot] in https://github.com/firecrawl/firecrawl/pull/2028 * build(deps): bump docker/login-action from 1 to 3 by @dependabot[bot] in https://github.com/firecrawl/firecrawl/pull/1996 * build(deps): bump docker/build-push-action from 5 to 6 by @dependabot[bot] in https://github.com/firecrawl/firecrawl/pull/1995 * build(deps): bump actions/setup-node from 3 to 4 by @dependabot[bot] in https://github.com/firecrawl/firecrawl/pull/1997 * feat: historical credit/token usage endpoints + more data in existing usage endpoints by @mogery in https://github.com/firecrawl/firecrawl/pull/2077 * fix(api/tsconfig): remove baseUrl by @mogery in https://github.com/firecrawl/firecrawl/pull/2078 * fix(search): get links and descriptions correctly by @mogery in https://github.com/firecrawl/firecrawl/pull/2076 * fix(api/crawler/sitemap): bump sitemap limit by @mogery in https://github.com/firecrawl/firecrawl/pull/2079 * fix(api/scrapeURL/index): re-sign expired screenshot URLs by @mogery in https://github.com/firecrawl/firecrawl/pull/2080 * fix(python-sdk): added missing get_queue_status in aio and added to t… by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2081 * Fix go-html-to-md on Windows (ENG-3398) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2082 * fix(js-sdk): zod-to-json-schema import by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2083 * Replace custom address validation functions with ipaddr.js (ENG-3404) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2084 * fix(api/native/pdf-parser): trim null chars out of pdf titles by @mogery in https://github.com/firecrawl/firecrawl/pull/2086 * Fix sitemap parsing (ENG-3361) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2085 * Implement webhook signatures (ENG-3018) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2087 * Format api and add pre-commit hooks (ENG-3408) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2088 * Fix pre-commit hook by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2089 * Ignore scripts during CI build by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2090 * Add more debug logging to crawler by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2091 * Add proxy location support to crawl and map endpoints (ENG-3361) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2092 * post-incident changes by @mogery in https://github.com/firecrawl/firecrawl/pull/2095 * feat(python-sdk): normalize docs in search results by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2098 * Feat(sdks): integration param by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2096 * Refactor webhook sending (ENG-3426) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2094 * Fix webhook data format regression by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2106 * Update Type Annotations for v2 Async Search (SearchResponse → SearchData) by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2097 * Fire webhook events for scrape/batch scrape errors (ENG-3463) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2107 * Add static IP proxy pool (ENG-3420) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2103 * Convert all Rust natives to a single library using NAPI (ENG-3397) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2105 * Filter out invalidated index records (ENG-3396) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2102 * feat(sdk): added agent option by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2108 * feat(api): nuq by @mogery in https://github.com/firecrawl/firecrawl/pull/1984 * Make harness cross-platform compatible (ENG-3477) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2110 * Update error messages for self-hosted instances by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/2119 * Revise /extract's error message when no content could be fetched from URLs by @micahstairs in https://github.com/firecrawl/firecrawl/pull/2109 * Fix graceful credit handling in account object (ENG-3495) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2120 * fix(scrapeURL/f-e/scrape): bad failed schema by @mogery in https://github.com/firecrawl/firecrawl/pull/2123 * feat(ci): revamp by @mogery in https://github.com/firecrawl/firecrawl/pull/2124 * fix(api/native/pdf): get title with proper encoding by @mogery in https://github.com/firecrawl/firecrawl/pull/2125 * chore: remove unused dependencies + various CI fixes by @mogery in https://github.com/firecrawl/firecrawl/pull/2128 * Perform watching inside of harness (ENG-3514) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2131 * Add gzipped sitemap support (ENG-3520) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2132 * Remove .xml.gz from `include` entries by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2134 * fix(api): rearchitect crawl kickoff by @mogery in https://github.com/firecrawl/firecrawl/pull/2133 * Update CONTRIBUTING.md by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2141 * Update CONTRIBUTING.md by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2142 * (fix/search) Implement retry mechanisms for empty results by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2140 * Optimize map (ENG-3526) by @amplitudesxd in https://github.com/firecrawl/firecrawl/pull/2138 * feat(api): add VIASOCKET integration by @ftonato in https://github.com/firecrawl/firecrawl/pull/2143 **Full Changelog**: https://github.com/firecrawl/firecrawl/compare/v2.1.0...v2.2.0 # Firecrawl v2.1.0 is here! ## ✨ New Features - **Search Categories**: Filter search results by specific categories using the `categories` parameter: - `github`: Search within GitHub repositories, code, issues, and documentation - `research`: Search academic and research websites (arXiv, Nature, IEEE, PubMed, etc.) - More coming soon - **Image Extraction:** Added image extraction support to the v2 scrape endpoint. - **Data Attribute Scraping:** Now supports extraction of `data-*` attributes. - **Hash-Based Routing:** Crawl endpoints now handle hash-based routes. - **Improved Google Drive Scraping:** Added ability to scrape TXT, PDF, and Sheets from Google Drive. - **PDF Enhancements:** Extracts PDF titles and shows them in metadata. - **API Enhancements:** - Map endpoint supports up to **100k results**. - **Helm Chart:** Initial Helm chart added for Firecrawl deployment. - **Security:** Improved protection against XFF spoofing. ## 🛠 Fixes - Fixed UTF-8 encoding in Google search scraper. - Restored crawl status in preview mode. - Fixed missing methods in Python SDK. - Corrected JSON response handling for v2 search with `scrapeOptions.formats`. - Fixed field population for `credits_billed` in v0 scrape. - Improved document field overlay in v2 search. ## 👥 New Contributors - @kelter-antunes - @vishkrish200 - @ieedan 🔗 [**Full Changelog**](https://github.com/firecrawl/firecrawl/compare/v2.0.1...v2.1.0) ## What's Changed * fix: handle UTF-8 encoding properly in Google search scraper by @kelter-antunes in https://github.com/firecrawl/firecrawl/pull/1924 * feat(api): add image extraction support to v2 scrape endpoint by @vishkrish200 in https://github.com/firecrawl/firecrawl/pull/2008 * feat(api): support extraction of data-* attributes in scrape endpoints by @vishkrish200 in https://github.com/firecrawl/firecrawl/pull/2006 * feat: add initial Helm chart for Firecrawl deployment by @JakobStadlhuber in https://github.com/firecrawl/firecrawl/pull/1262 * feat(api/crawl): support hash-based routing by @mogery in https://github.com/firecrawl/firecrawl/pull/2031 * fix(python-sdk): missing methods in client by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/2050 * feat(countryCheck): better protection against XFF spoofing by @mogery in https://github.com/firecrawl/firecrawl/pull/2051 * fix: include json in v2 /search response when using scrapeOptions.formats by @ieedan in https://github.com/firecrawl/firecrawl/pull/2052 * feat(scrapeURL/rewrite): scrape Google Drive TXT/PDF files and sheets by @mogery in https://github.com/firecrawl/firecrawl/pull/2053 * Update README.md by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2060 * (fix/crawl) Re-enable crawl status in preview mode by @nickscamara in https://github.com/firecrawl/firecrawl/pull/2061 * feat(pdf-parser): get PDF title and show in metadata by @mogery in https://github.com/firecrawl/firecrawl/pull/2062 * fix(v2/search): overlay doc fields via spread operator by @mogery in https://github.com/firecrawl/firecrawl/pull/2054 * feat(api): propagate api_key_id towards billing function by @mogery in https://github.com/firecrawl/firecrawl/pull/2049 * feat(api/map): use new RPCs + set limit max to 100k by @mogery in https://github.com/firecrawl/firecrawl/pull/2065 * fix(api/v0/scrape): populate credits_billed field by @mogery in https://github.com/firecrawl/firecrawl/pull/2066 ## New Contributors * @kelter-antunes made their first contribution in https://github.com/firecrawl/firecrawl/pull/1924 * @vishkrish200 made their first contribution in https://github.com/firecrawl/firecrawl/pull/2008 * @ieedan made their first contribution in https://github.com/firecrawl/firecrawl/pull/2052 **Full Changelog**: https://github.com/firecrawl/firecrawl/compare/v2.0.1...v2.1.0 This release fixes the "SSRF Vulnerability via malicious webhook" security advisory. It is recommended that people using the self-hosted version of Firecrawl update to v2.0.1 immediately. More info in the advisory: https://github.com/firecrawl/firecrawl/security/advisories/GHSA-p2wg-prhf-jx79 # Introducing v2.0.0 ### Key Improvements * **Faster by default**: Requests are cached with `maxAge` defaulting to 2 days, and sensible defaults like `blockAds`, `skipTlsVerification`, and `removeBase64Images` are enabled. * **New summary format**: You can now specify `"summary"` as a format to directly receive a concise summary of the page content. * **Updated JSON extraction**: JSON extraction and change tracking now use an object format: `{ type: "json", prompt, schema }`. The old `"extract"` format has been renamed to `"json"`. * **Enhanced screenshot options**: Use the object form: `{ type: "screenshot", fullPage, quality, viewport }`. * **New search sources**: Search across `"news"` and `"images"` in addition to web results by setting the `sources` parameter. * **Smart crawling with prompts**: Pass a natural-language `prompt` to crawl and the system derives paths/limits automatically. Use the new crawl-params-preview endpoint to inspect the derived options before starting a job. ## Quick migration checklist * Replace v1 client usage with v2 clients: * JS: `const firecrawl = new Firecrawl({ apiKey: 'fc-YOUR-API-KEY' })` * Python: `firecrawl = Firecrawl(api_key='fc-YOUR-API-KEY')` * API: use the new `https://api.firecrawl.dev/v2/` endpoints. * Update formats: * Use `"summary"` where needed * JSON mode: Use `{ type: "json", prompt, schema }` for JSON extraction * Screenshot and Screenshot\@fullPage: Use screenshot object format when specifying options * Adopt standardized async flows in the SDKs: * Crawls: `startCrawl` + `getCrawlStatus` (or `crawl` waiter) * Batch: `startBatchScrape` + `getBatchScrapeStatus` (or `batchScrape` waiter) * Extract: `startExtract` + `getExtractStatus` (or `extract` waiter) * Crawl options mapping (see below) * Check crawl `prompt` with `crawl-params-preview` ## SDK surface (v2) ### JS/TS #### Method name changes (v1 → v2) **Scrape, Search, and Map** | v1 (FirecrawlApp) | v2 (Firecrawl) | | --------------------- | ------------------------- | | `scrapeUrl(url, ...)` | `scrape(url, options?)` | | `search(query, ...)` | `search(query, options?)` | | `mapUrl(url, ...)` | `map(url, options?)` | **Crawling** | v1 | v2 | | --------------------------- | ------------------------------- | | `crawlUrl(url, ...)` | `crawl(url, options?)` (waiter) | | `asyncCrawlUrl(url, ...)` | `startCrawl(url, options?)` | | `checkCrawlStatus(id, ...)` | `getCrawlStatus(id)` | | `cancelCrawl(id)` | `cancelCrawl(id)` | | `checkCrawlErrors(id)` | `getCrawlErrors(id)` | **Batch Scraping** | v1 | v2 | | --------------------------------- | ----------------------------------- | | `batchScrapeUrls(urls, ...)` | `batchScrape(urls, opts?)` (waiter) | | `asyncBatchScrapeUrls(urls, ...)` | `startBatchScrape(urls, opts?)` | | `checkBatchScrapeStatus(id, ...)` | `getBatchScrapeStatus(id)` | | `checkBatchScrapeErrors(id)` | `getBatchScrapeErrors(id)` | **Extraction** | v1 | v2 | | ----------------------------- | ---------------------- | | `extract(urls?, params?)` | `extract(args)` | | `asyncExtract(urls, params?)` | `startExtract(args)` | | `getExtractStatus(id)` | `getExtractStatus(id)` | **Other / Removed** | v1 | v2 | | --------------------------------- | --------------------- | | `generateLLMsText(...)` | (not in v2 SDK) | | `checkGenerateLLMsTextStatus(id)` | (not in v2 SDK) | | `crawlUrlAndWatch(...)` | `watcher(jobId, ...)` | | `batchScrapeUrlsAndWatch(...)` | `watcher(jobId, ...)` | #### Type name changes (v1 → v2) **Core Document Types** | v1 | v2 | | ---------------------------- | ---------------------------- | | `FirecrawlDocument` | `Document` | | `FirecrawlDocumentMetadata` | `DocumentMetadata` | **Scrape, Search, and Map Types** | v1 | v2 | | ---------------------------- | ---------------------------- | | `ScrapeParams` | `ScrapeOptions` | | `ScrapeResponse` | `Document` | | `SearchParams` | `SearchRequest` | | `SearchResponse` | `SearchData` | | `MapParams` | `MapOptions` | | `MapResponse` | `MapData` | **Crawl Types** | v1 | v2 | | ---------------------------- | ---------------------------- | | `CrawlParams` | `CrawlOptions` | | `CrawlStatusResponse` | `CrawlJob` | **Batch Operations** | v1 | v2 | | ---------------------------- | ---------------------------- | | `BatchScrapeStatusResponse` | `BatchScrapeJob` | **Action Types** | v1 | v2 | | ---------------------------- | ---------------------------- | | `Action` | `ActionOption` | **Error Types** | v1 | v2 | | ---------------------------- | ---------------------------- | | `FirecrawlError` | `SdkError` | | `ErrorResponse` | `ErrorDetails` | *** ### Python (sync) #### Method name changes (v1 → v2) **Scrape, Search, and Map** | v1 | v2 | | ----------------- | ------------- | | `scrape_url(...)` | `scrape(...)` | | `search(...)` | `search(...)` | | `map_url(...)` | `map(...)` | **Crawling** | v1 | v2 | | ------------------------- | ----------------------- | | `crawl_url(...)` | `crawl(...)` (waiter) | | `async_crawl_url(...)` | `start_crawl(...)` | | `check_crawl_status(...)` | `get_crawl_status(...)` | | `cancel_crawl(...)` | `cancel_crawl(...)` | **Batch Scraping** | v1 | v2 | | ------------------------------ | ------------------------------ | | `batch_scrape_urls(...)` | `batch_scrape(...)` (waiter) | | `async_batch_scrape_urls(...)` | `start_batch_scrape(...)` | | `get_batch_scrape_status(...)` | `get_batch_scrape_status(...)` | | `get_batch_scrape_errors(...)` | `get_batch_scrape_errors(...)` | **Extraction** | v1 | v2 | | ------------------------- | ------------------------- | | `extract(...)` | `extract(...)` | | `start_extract(...)` | `start_extract(...)` | | `get_extract_status(...)` | `get_extract_status(...)` | **Other / Removed** | v1 | v2 | | ------------------------------------ | ---------------------- | | `generate_llms_text(...)` | (not in v2 SDK) | | `get_generate_llms_text_status(...)` | (not in v2 SDK) | | `watch_crawl(...)` | `watcher(job_id, ...)` | *** ### Python (async) * `AsyncFirecrawl` mirrors the same methods (all awaitable). ## Formats and scrape options * Use string formats for basics: `"markdown"`, `"html"`, `"rawHtml"`, `"links"`, `"summary"`. * Instead of `parsePDF` use `parsers: [ { "type": "pdf" } | "pdf" ]`. * Use object formats for JSON, change tracking, and screenshots: ### JSON format ```bash curl -X POST https://api.firecrawl.dev/v2/scrape \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -d '{ "url": "https://docs.firecrawl.dev/", "formats": [{ "type": "json", "prompt": "Extract the company mission from the page." }] }' ``` ### Screenshot format ```bash cURL curl -X POST https://api.firecrawl.dev/v2/scrape \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -d '{ "url": "https://docs.firecrawl.dev/", "formats": [{ "type": "screenshot", "fullPage": true, "quality": 80, "viewport": { "width": 1280, "height": 800 } }] }' ``` ## Crawl options mapping (v1 → v2) | v1 | v2 | | ----------------------- | ---------------------------------------------------- | | `allowBackwardCrawling` | (removed) use `crawlEntireDomain` | | `maxDepth` | (removed) use `maxDiscoveryDepth` | | `ignoreSitemap` (bool) | `sitemap` (e.g., `"only"`, `"skip"`, or `"include"`) | | (none) | `prompt` | ## Crawl prompt + params preview See crawl params preview examples: ```bash curl -X POST https://api.firecrawl.dev/v2/crawl-params-preview \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer YOUR_API_KEY' \ -d '{ "url": "https://docs.firecrawl.dev", "prompt": "Extract docs and blog" }' ``` ## What's Changed * Add a couple exceptions to our blocked list by @micahstairs in https://github.com/firecrawl/firecrawl/pull/1816 * fix(api/v1/types): depth check throws error if URL is invalid by @mogery in https://github.com/firecrawl/firecrawl/pull/1821 * (feat/rtxt) Improved robots control on scrape via flags by @nickscamara in https://github.com/firecrawl/firecrawl/pull/1820 * fix/actions dict attributeError by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/1824 * feat(types): add JSON schema validation to schema options in extractO… by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/1803 * fix(python-sdk): add max_age parameter to scrape_url validation by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1825 * Restore Rust link filtering logic by @mogery in https://github.com/firecrawl/firecrawl/pull/1822 * feat: modified docker image to be runable by any UID by @expruc in https://github.com/firecrawl/firecrawl/pull/1819 * fix(api): update vulnerable pkgs by @mogery in https://github.com/firecrawl/firecrawl/pull/1829 * fix: more packaging vulns by @mogery in https://github.com/firecrawl/firecrawl/pull/1831 * chore(deps): bump form-data from 4.0.0 to 4.0.4 in /apps/js-sdk by @dependabot[bot] in https://github.com/firecrawl/firecrawl/pull/1832 * chore(deps): bump axios from 1.6.8 to 1.11.0 in /apps/js-sdk by @dependabot[bot] in https://github.com/firecrawl/firecrawl/pull/1833 * chore(deps): bump esbuild and tsx in /apps/js-sdk by @dependabot[bot] in https://github.com/firecrawl/firecrawl/pull/1834 * fix(js-sdk): remaining pkg vulns by @mogery in https://github.com/firecrawl/firecrawl/pull/1835 * fix(crawl): only extend URLs at pre-finish if changeTracking is on (ENG-2900) by @mogery in https://github.com/firecrawl/firecrawl/pull/1837 * fix(queue-worker): clean up stalled jobs to not get crawls stuck (ENG-2902) by @mogery in https://github.com/firecrawl/firecrawl/pull/1838 * fix(queue-worker): improve stalled job cleaner (ENG-2907) by @mogery in https://github.com/firecrawl/firecrawl/pull/1839 * feat: rewrite sitemap XML parsing from JavaScript to Rust (ENG-2904) by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1840 * Fix robots.txt parser panic with content type validation by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1843 * Add allowTeammateInvites flag to TeamFlags type by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1847 * Fix ignoreQueryParameters being ignored in URL deduplication - ENG-2804 by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1846 * ENG-2829: Fix isSubdomain bug by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1845 * fix(crawler/sitemap): improvements by @mogery in https://github.com/firecrawl/firecrawl/pull/1842 * Add __experimental_omceDomain flag for debugging and benchmarking by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1844 * feat: Add iframe selector transformation for includeTags and excludeTags by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1850 * fix(worker/antistall/kickoff): bad check (ENG-2936) by @mogery in https://github.com/firecrawl/firecrawl/pull/1859 * Update batch_billing.ts by @nickscamara in https://github.com/firecrawl/firecrawl/pull/1860 * hotfix by @mogery in https://github.com/firecrawl/firecrawl/pull/1861 * fix(queue-service): get rid of done billing jobs much faster by @mogery in https://github.com/firecrawl/firecrawl/pull/1862 * further logging by @mogery in https://github.com/firecrawl/firecrawl/pull/1863 * more logs by @mogery in https://github.com/firecrawl/firecrawl/pull/1864 * Fixes #1870: correct maxConcurrency calculation by @wwhurley in https://github.com/firecrawl/firecrawl/pull/1871 * fix(html-to-markdown): reinitialize converter lib for every conversion (ENG-2956) by @mogery in https://github.com/firecrawl/firecrawl/pull/1872 * Revert go version in Dockerfile by @mogery in https://github.com/firecrawl/firecrawl/pull/1873 * fix(crawl-status): move count_jobs_of_crawl_team after checks by @mogery in https://github.com/firecrawl/firecrawl/pull/1875 * update koffi by @mogery in https://github.com/firecrawl/firecrawl/pull/1876 * feat(v2): add natural language prompt support to crawl API by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1877 * Fix robots.txt HTML filtering to check content structure by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1880 * fix(go): add mutex to prevent concurrent access issues in html-to-markdown by @mogery in https://github.com/firecrawl/firecrawl/pull/1883 * feat: better log attribute propagation (ENG-2934) by @mogery in https://github.com/firecrawl/firecrawl/pull/1857 * fix(js-sdk): add retry logic for socket hang up errors in monitorJobStatus (ENG-3029) by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1893 * Fix Pydantic field name shadowing issues causing import NameError by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1800 * feat: add crawlTtlHours team flag to replace teamIdsExcludedFromExpiry by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1899 * feat(crawler): replace robotstxt library with texting_robots for ENG-3016 by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1895 * queue-worker: move scrape worker to separate threads (ENG-3008) by @mogery in https://github.com/firecrawl/firecrawl/pull/1879 * fix(crawl-redis): attempt to cleanup crawl memory post finish by @mogery in https://github.com/firecrawl/firecrawl/pull/1901 * fix: convert timeout from milliseconds to seconds in Python SDK by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1894 * feat(python-sdk): implement missing crawl_entire_domain parameter by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1896 * Improve error handling in Python SDK for non-JSON responses by @rafaelsideguide in https://github.com/firecrawl/firecrawl/pull/1827 * feat: queue multiplexing by @mogery in https://github.com/firecrawl/firecrawl/pull/1902 * fix(docs): correct link to Map by @ChetanGoti in https://github.com/firecrawl/firecrawl/pull/1904 * feat(v2): parsers + merge w/ main by @mogery in https://github.com/firecrawl/firecrawl/pull/1907 * (feat/index) Domain frequency aggregator by @nickscamara in https://github.com/firecrawl/firecrawl/pull/1908 * fix(api): add database fallback to crawl errors endpoint by @mogery in https://github.com/firecrawl/firecrawl/pull/1909 * feat(v2): Add viewport parameter for screenshots by @mogery in https://github.com/firecrawl/firecrawl/pull/1910 * feat(scrape-v2): Implement skipTlsVerification support for fetch and playwright engines by @mogery in https://github.com/firecrawl/firecrawl/pull/1911 * feat(v2): Set default maxAge to 4 hours for scrape endpoint by @mogery in https://github.com/firecrawl/firecrawl/pull/1915 * Fix v1 API JSON/extract format backward compatibility on v2 by @mogery in https://github.com/firecrawl/firecrawl/pull/1917 * Fix: Handle --max-old-space-size flag for worker threads by @Josh-M42 in https://github.com/firecrawl/firecrawl/pull/1922 * feat: langfuse integration by @mogery in https://github.com/firecrawl/firecrawl/pull/1928 * ENG-3088: Change parsers parameter from object to array format by @mogery in https://github.com/firecrawl/firecrawl/pull/1931 * feat(python-sdk): add agent parameter support to scrape_url method by @devin-ai-integration[bot] in https://github.com/firecrawl/firecrawl/pull/1919 * fix: prevent PDF scrapes with parsePDF:false from being indexed by @mogery in https://github.com/firecrawl/firecrawl/pull/1933 * feat(crawl-redis): reduce `crawl::visited` size in Redis by 16x by @mogery in https://github.com/firecrawl/firecrawl/pull/1936 * feat: adding on-demand search endpoint by @ftonato in https://github.com/firecrawl/firecrawl/pull/1852 * fix(queue-worker): turn off useWorkerThreads in sandboxed scrape worker by @mogery in https://github.com/firecrawl/firecrawl/pull/1941 * fix(scrapeURL/pdf): better timeout error for PDF scrapes by @mogery in https://github.com/firecrawl/firecrawl/pull/1942 * feat(scrape-worker): reintroduce OTEL for accurate LLM cost tracking by @mogery in https://github.com/firecrawl/firecrawl/pull/1943 * feat(api/ai-sdk): add labels to vertex/google calls by @mogery in https://github.com/firecrawl/firecrawl/pull/1944 * test(v2/crawl): implement tests for crawl API with prompt parameter by @mogery in https://github.com/firecrawl/firecrawl/pull/1929 * feat(ci): audit NPM packages on PR by @mogery in https://github.com/firecrawl/firecrawl/pull/1947 * fix(tests/scrape): make `maxAge: 0` explicit in Index tests by @mogery in https://github.com/firecrawl/firecrawl/pull/1946 * ENG-3089: Support both string and object format inputs in v2 scrape API by @mogery in https://github.com/firecrawl/firecrawl/pull/1932 * feat(v2/timeout): initial new waterfalling system (ENG-2922) by @mogery in https://github.com/firecrawl/firecrawl/pull/1950 * feat(v2): extract by @mogery in https://github.com/firecrawl/firecrawl/pull/1955 * feat(v2): error handling by @mogery in https://github.com/firecrawl/firecrawl/pull/1957 * Test new mu version by @tomkosm in https://github.com/firecrawl/firecrawl/pull/1958 * Update mu by @tomkosm in https://github.com/firecrawl/firecrawl/pull/1959 * feat(api/admin): prometheus metrics about cc limit queue by @mogery in https://github.com/firecrawl/firecrawl/pull/1963 * fix(api): stop using bulljobs_teams table by @mogery in https://github.com/firecrawl/firecrawl/pull/1962 * fix(queue-service): reduce redis connections to BullMQ by @mogery in https://github.com/firecrawl/firecrawl/pull/1966 * fix(crawl-status, extract-status): reduce the use of BullMQ getState by @mogery in https://github.com/firecrawl/firecrawl/pull/1968 * feat(api): add OTEL everywhere by @mogery in https://github.com/firecrawl/firecrawl/pull/1969 * feat(crawl-status): refactor by @mogery in https://github.com/firecrawl/firecrawl/pull/1971 * (feat/api) v2 by @nickscamara in https://github.com/firecrawl/firecrawl/pull/1841 ## New Contributors * @expruc made their first contribution in https://github.com/firecrawl/firecrawl/pull/1819 * @wwhurley made their first contribution in https://github.com/firecrawl/firecrawl/pull/1871 * @ChetanGoti made their first contribution in https://github.com/firecrawl/firecrawl/pull/1904 * @Josh-M42 made their first contribution in https://github.com/firecrawl/firecrawl/pull/1922 **Full Changelog**: https://github.com/firecrawl/firecrawl/compare/v1.15.0...v2.0.0 ## Firecrawl v1.15.0 is here! - SSO for enterprise - Improved scraping reliability - Search params added to activity logs - FireGEO example - And over 50 PRs merged for bug & improvements 🔥 ### Improvements - OMCE support in `scrapeURL` and HTML transformer - Improved logging (search params, cache age) - New `created_at` field in `/crawl/active` response - Case-insensitive URL protocol checks - `filterLinks` ported to Rust ### Fixes - HTML transformer stability (Arabic, base tag, panic) - `scrapeURL` index bug & waitFor exclusion - PDF billing bug with `parsePDF=false` - Crawl returning only 1 result (edge case) - Crawler no-sections bug - Logger method naming - Timeout handling in API - `crawl-status` resilience for ejected jobs ### SDK & Infra - SDK header param fix & async error handling - Express port now configurable via env var - Temporary crawl expiry exemption ### Docs - Kubernetes setup update ## What's Changed * Make worker Express server port configurable via environment variable by @devin-ai-integration[bot] in https://github.com/mendableai/firecrawl/pull/1748 * fix(logger): correct method names in logger.child calls by @ahnafatef in https://github.com/mendableai/firecrawl/pull/1731 * feat(html-transformer, scrapeURL): omce support by @mogery in https://github.com/mendableai/firecrawl/pull/1764 * Add created_at field to /crawl/active endpoint response by @devin-ai-integration[bot] in https://github.com/mendableai/firecrawl/pull/1718 * feat: test RunPod MU new version by @tomkosm in https://github.com/mendableai/firecrawl/pull/1771 * docs: kubernetes simple update by @mogery in https://github.com/mendableai/firecrawl/pull/1772 * fix(api): f-e timeout handling by @mogery in https://github.com/mendableai/firecrawl/pull/1774 * fix(api/html-transformer): stop panicing on arabic sites by @mogery in https://github.com/mendableai/firecrawl/pull/1773 * fix(api/html-transformer): relative base tag URL handling by @mogery in https://github.com/mendableai/firecrawl/pull/1776 * html-transformer: never panic by @mogery in https://github.com/mendableai/firecrawl/pull/1778 * feat(search): improve param logging by @mogery in https://github.com/mendableai/firecrawl/pull/1777 * fix(scrapeURL/index): horrible no-good very bad index url bug by @mogery in https://github.com/mendableai/firecrawl/pull/1780 * feat(index): store request frequency for precrawling by @mogery in https://github.com/mendableai/firecrawl/pull/1782 * feat(scrapeURL): log cache age in request frequency by @mogery in https://github.com/mendableai/firecrawl/pull/1784 * fix(scrapeURL/index): exclude waitfor by @mogery in https://github.com/mendableai/firecrawl/pull/1787 * feat: make URL protocol checks case-insensitive by @devin-ai-integration[bot] in https://github.com/mendableai/firecrawl/pull/1788 * Revert "Add temporary exception for specific team to bypass job expiration" by @micahstairs in https://github.com/mendableai/firecrawl/pull/1789 * fix(extract): improve enforcement by @mogery in https://github.com/mendableai/firecrawl/pull/1790 * insert omce jobs upon scrape by @mogery in https://github.com/mendableai/firecrawl/pull/1786 * [sdk] fixes missing headers param in scrape_url by @rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1795 * Add temporary exemption for crawl expiry by @micahstairs in https://github.com/mendableai/firecrawl/pull/1796 * fix(crawl-status): keep working even if jobs are ejected from bullmq by @mogery in https://github.com/mendableai/firecrawl/pull/1799 * feat: precrawl worker by @mogery in https://github.com/mendableai/firecrawl/pull/1783 * sdk-fix: ensure async error handling in AsyncFirecrawlApp methods, up… by @rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1802 * feat(crawler): port filterLinks to Rust by @mogery in https://github.com/mendableai/firecrawl/pull/1801 * Fix search endpoint PDF billing when parsePDF=false by @devin-ai-integration[bot] in https://github.com/mendableai/firecrawl/pull/1806 * Fix bug which sometimes caused crawl to only return 1 result by @micahstairs in https://github.com/mendableai/firecrawl/pull/1810 * (fix/crawler) No sections edge case fix by @nickscamara in https://github.com/mendableai/firecrawl/pull/1814 ## New Contributors * @ahnafatef made their first contribution in https://github.com/mendableai/firecrawl/pull/1731 **Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.14.0...v1.15.0 ## v1.14.0 Release We're excited to announce the release of Firecrawl v1.14.0, packed with cool updates. - Authenticated scraping (Join the waitlist [here](https://firecrawl.dev/authenticated-scraping)) - Zero data retention for enterprise (Email us at [help@firecrawl.com](mailto:help@firecrawl.com) to enable it) - Improved p75 speeds - New MCP version w/ maxAge + better tool calling - Open Researcher Example (Open Source Researcher). See [repo](https://github.com/mendableai/open-researcher) - And so much more 🔥 ## What's Changed * fix json format on search (ENG-2529) by @mogery in https://github.com/mendableai/firecrawl/pull/1729 * proxy used improvement by @mogery in https://github.com/mendableai/firecrawl/pull/1727 * add pdf prefetch log for debugging (ENG-2542) by @mogery in https://github.com/mendableai/firecrawl/pull/1734 * (feat/ledger) Ledger events by @nickscamara in https://github.com/mendableai/firecrawl/pull/1728 * fix(auto_charge): bad hourly counter logic by @mogery in https://github.com/mendableai/firecrawl/pull/1736 * feat: implement IDN support with Punycode encoding by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1735 * feat(scrapeURL): ask user to increase timeout if there's a DOM.getDocument or queryAXTree error by @mogery in https://github.com/mendableai/firecrawl/pull/1739 * feat(queue-worker): decrease job lock duration to pick up jobs on dead workers faster by @mogery in https://github.com/mendableai/firecrawl/pull/1737 * feat(crawl-status): better creditsUsed field by @mogery in https://github.com/mendableai/firecrawl/pull/1738 * (fix/sdk) Fixes Extract with show_sources=True causes JSON parsing error by @nickscamara in https://github.com/mendableai/firecrawl/pull/1740 * fix(map): pass timeout to sitemap fetch by @mogery in https://github.com/mendableai/firecrawl/pull/1741 * feat(api): zero data retention (ENG-2376) by @mogery in https://github.com/mendableai/firecrawl/pull/1687 * Fix protocol change in firecrawl services by @nickscamara in https://github.com/mendableai/firecrawl/pull/1742 * Fix queue worker liveness endpoint for self-hosted environments (ENG-2587) by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1747 * feat(firecrawl): add integration parameter support and enhance kwargs handling by @ftonato in https://github.com/mendableai/firecrawl/pull/1715 * bugfix zero_data_retention param and certifi dependency by @rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1749 * Add health check endpoint for Playwright service (ENG-2585) by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1752 * Add URL depth validation to crawl requests (ENG-2617) by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1753 * fix(api/deep-research): bad default values leading to bug on self-host by @mogery in https://github.com/mendableai/firecrawl/pull/1754 * Make error message for backwards crawling more clear by @micahstairs in https://github.com/mendableai/firecrawl/pull/1758 * Implement JavaScript-only base href handling (ENG-2302) by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1756 * fix(html-transformer): check base tag when resolving relative URLs by @mogery in https://github.com/mendableai/firecrawl/pull/1761 * html-transformer improvements by @mogery in https://github.com/mendableai/firecrawl/pull/1762 * Implement fallback mechanism for onlyMainContent scraping (ENG-2499) by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1759 * Add end-to-end test for deep research functionality (ENG-2627) by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1760 * feat: add waitFor validation rule to enforce waitFor <= timeout/2 by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1751 * fix: replace hardcoded search billing with calculateCreditsToBeBilled by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1714 * feat(scrapeURL/fire-engine): start AB testing by @mogery in https://github.com/mendableai/firecrawl/pull/1763 **Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.13.0...v1.14.0 ## Firecrawl v1.13.0 Changelog - Added AU, FR, DE to [Stealth Mode](https://docs.firecrawl.dev/features/proxies#location-based-proxy-selection) - Crawl subdomains with [allowSubdomains](https://docs.firecrawl.dev/api-reference/endpoint/crawl-post#body-allow-subdomains) - Google slides scraping - Generate a PDF of the current page. See [docs](https://docs.firecrawl.dev/api-reference/endpoint/scrape#generate-pdf) - Higher res screenshots with [quality param](https://docs.firecrawl.dev/api-reference/endpoint/scrape#screenshot) - Weekly view for usage on the dashboard - Fireplexity Example (Open Source Perplexity). See [repo](https://github.com/mendableai/fireplexity) - And more! ### Features - Support cookies in `scrapeURL` via `safeFetch` - Add Google Slides scraping - Add PDF actions to API, Python & JS SDKs - Screenshot quality config - `parsePDF` support in Python & JS SDKs - Allow international URLs - Enable subdomain crawling in SDKs - Worker liveness check before job queueing ### Improvements & Fixes - Removed old cache systems (Redis, PDF cache, etc.) - Enhanced error messages (TLS, DNS) - Improved job count query & error handling - Fixed `pdf`, `credits` scope, `ignoreInvalidURLs` bugs - Reduced over-logging in concurrency limits - Added special job expiration bypass for Faire team - Unified fallback logic in JS SDK ## What's Changed * Remove old cache mechanisms (redis cache, PDF cache, crawl maps, etc.) (FIR-2266) by @mogery in https://github.com/mendableai/firecrawl/pull/1667 * feat(scrapeURL): support cookies in safeFetch by @mogery in https://github.com/mendableai/firecrawl/pull/1688 * fix(search): respect parsePDF in pricing by @mogery in https://github.com/mendableai/firecrawl/pull/1690 * feat(scrape): support Google Slides by @meetsoni1214 in https://github.com/mendableai/firecrawl/pull/1693 * fix(api): CI by @mogery in https://github.com/mendableai/firecrawl/pull/1692 * fix(api): improve error logging with structured error object by @mogery in https://github.com/mendableai/firecrawl/pull/1697 * fix(api): enhance error handler with optional ACUC data by @mogery in https://github.com/mendableai/firecrawl/pull/1698 * fix(api): handle errors better in redis-less crawl status by @mogery in https://github.com/mendableai/firecrawl/pull/1699 * feat(api): optimize job count query and improve error handling by @mogery in https://github.com/mendableai/firecrawl/pull/1700 * fix(api/v1/crawl/ongoing): only crawls, no batch scrape by @mogery in https://github.com/mendableai/firecrawl/pull/1701 * feat(api): pdf action + housekeeping by @mogery in https://github.com/mendableai/firecrawl/pull/1702 * fix(v1): check credits variable scope collision by @mogery in https://github.com/mendableai/firecrawl/pull/1703 * fix(api): pdf bug + testing bugs by @mogery in https://github.com/mendableai/firecrawl/pull/1704 * fix(api/batch/scrape): maxConcurrency field support when using ignoreInvalidURLs by @mogery in https://github.com/mendableai/firecrawl/pull/1705 * fix(api): instantiate Storage only once by @mogery in https://github.com/mendableai/firecrawl/pull/1706 * feat(api/ci): idmux by @mogery in https://github.com/mendableai/firecrawl/pull/1707 * fix(concurrency-limit): scan instead of taking jobs by @mogery in https://github.com/mendableai/firecrawl/pull/1708 * fix(concurrency-limit): overlogging by @mogery in https://github.com/mendableai/firecrawl/pull/1709 * fix(api): cached acuc didn't have the is_extract flag set (ENG-2468) by @mogery in https://github.com/mendableai/firecrawl/pull/1712 * fix(js-sdk/extract): use same zod fallback logic by @mogery in https://github.com/mendableai/firecrawl/pull/1711 * Add temporary exception for Faire team ID to bypass job expiration by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1716 * feat(python-sdk): add parsePDF parameter support by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1713 * Allow international URLs to pass validation by @micahstairs in https://github.com/mendableai/firecrawl/pull/1717 * feat: Screenshot quality by @nickscamara in https://github.com/mendableai/firecrawl/pull/1721 * feat(sdk/python): add pdf action (ENG-2515) by @mogery in https://github.com/mendableai/firecrawl/pull/1722 * feat(scrapeURL/skipTlsVerification): improve error message by @mogery in https://github.com/mendableai/firecrawl/pull/1723 * feat: improve DNS resolution error message by @mogery in https://github.com/mendableai/firecrawl/pull/1724 * Fix unreachable allowSubdomains code in crawler filterURL method by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1719 * Add parsePDF parameter to JS SDK by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1720 * feat(api/worker): liveness check in queueing -- don't take jobs when the worker is dying (ENG-2519) by @mogery in https://github.com/mendableai/firecrawl/pull/1725 * (feat/sdk) Allow Subdomains in the sdks by @nickscamara in https://github.com/mendableai/firecrawl/pull/1726 ## New Contributors * @meetsoni1214 made their first contribution in https://github.com/mendableai/firecrawl/pull/1693 * @micahstairs made their first contribution in https://github.com/mendableai/firecrawl/pull/1717 **Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.12.0...v1.13.0 ## Firecrawl v1.12.0 is here! - New Concurrency System - Specify max concurrency by request for better control. [See docs](https://docs.firecrawl.dev/api-reference/endpoint/crawl-post#body-max-concurrency). - Crawl Entire Domain Param - Follow internal links to sibling or parent URLs, not just child paths (prev. allowBackwardLinks). [See docs](https://docs.firecrawl.dev/api-reference/endpoint/crawl-post#body-crawl-entire-domain). - Google Docs Scraping - We now officially support scraping Google Docs files - Improved Activity Logs - Better support for FIRE-1 requests. [See your logs here.](https://www.firecrawl.dev/app/logs) - /search Playground Enhanced - Location Params added. [Check out the playground.](https://www.firecrawl.dev/playground?mode=search) - Firestarter Example - Open Source Chatbot building platform. [Repo here.](https://github.com/mendableai/firestarter) - Plus tons of performance improvements and bug fixes.  P.S. Have feedback or ideas for v1.13.0? Hit reply and let us know. We're always listening to our community to build the features you need most. ## What's Changed * feat(vertex): fix vertex ai provider bug and update model references… by @tomkosm in https://github.com/mendableai/firecrawl/pull/1668 * Improve URL filtering error messages with specific denial reasons (FIR-2352) by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1676 * feat(index): store short titles and descriptions (FIR-2356) by @mogery in https://github.com/mendableai/firecrawl/pull/1677 * feat: Test mu v3 by @tomkosm in https://github.com/mendableai/firecrawl/pull/1678 * Add deployment type field to bug report template by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1681 * feat(crawl-status): refactor to work after a redis flush (FIR-2271) by @mogery in https://github.com/mendableai/firecrawl/pull/1664 * fix(scrape): log FIRE-1 credits billed on failures properly (FIR-2331) by @mogery in https://github.com/mendableai/firecrawl/pull/1682 * feat(scrape, extract): creditsUsed, tokensUsed fields (FIR-2336) by @mogery in https://github.com/mendableai/firecrawl/pull/1683 * Concurrency limit refactor + `maxConcurrency` parameter (FIR-2191) by @mogery in https://github.com/mendableai/firecrawl/pull/1643 * feat(scrape): support Google Docs (FIR-1365) by @mogery in https://github.com/mendableai/firecrawl/pull/1686 * feat: add followInternalLinks parameter as semantic replacement for allowBackwardLinks by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1684 **Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.11.0...v1.12.0 ## Firecrawl v1.11.0 is here! ### Major Features - Launched our Firecrawl Index - Speed up scrapes 5x if opted in - Improved Activity Logs - View webhook events - Active crawl management - Fire Enrich Example (Open Source Clay) - Community Java SDK - and a lot more ### Features - Improved Playwright tests and webhook test coverage - Added `GET /crawl/ongoing` endpoint - Introduced tag support in change tracking - Added `integration` field to jobs and propagated through queue worker - Parallel testing for runpod v2 and updated `mu` - Ported `queryIndexAtSplitLevel` to RPC - Enhanced SDK with index and missing parameters - Removed redundant GCS check to improve performance - Added `credits_billed` field across pipeline - Enabled domain-level index splitting for better map querying - Used index in search and extract operations - Removed unused index columns ### Fixes & Improvements - Fixed crawl pre-finishing logic - Refactored `callWebhook` and added logging - Improved index testing (FIR-2214) - Fixed JS SDK tests - Clarified scrape options usage in README - Fixed missing `PLAYWRIGHT_MICROSERVICE_URL` in env example - Improved concurrency limit notification emails - Removed query param sanitization that broke extract ### Contributors - @Arkit003 made their first contribution - @pulbyte made their first contribution **Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.10.0...v1.11.0 ## What's Changed * feat(api/tests/scrape): Playwright test improvements by @mogery in https://github.com/mendableai/firecrawl/pull/1626 * feat(queue-worker): fix crawl pre-finishing logic (FIR-2171) by @mogery in https://github.com/mendableai/firecrawl/pull/1628 * feat(api/tests): add webhook tests + refactor batch scrape lib (FIR-2219) by @mogery in https://github.com/mendableai/firecrawl/pull/1630 * feat(api): GET /crawl/ongoing (FIR-2189) by @mogery in https://github.com/mendableai/firecrawl/pull/1620 * feat(changeTracking): support tags (FIR-1940) by @mogery in https://github.com/mendableai/firecrawl/pull/1631 * feat(api): add integration field to jobs and update related controllers and types by @ftonato in https://github.com/mendableai/firecrawl/pull/1632 * feat(api): propagate integration field in queue worker job processing by @ftonato in https://github.com/mendableai/firecrawl/pull/1635 * feat(scraper): runpod v2 parallel testing by @tomkosm in https://github.com/mendableai/firecrawl/pull/1636 * feat: update mu by @tomkosm in https://github.com/mendableai/firecrawl/pull/1639 * feat(api/index): port queryIndexAtSplitLevel to RPC (FIR-2241) by @mogery in https://github.com/mendableai/firecrawl/pull/1640 * feat(webhook): refactor callWebhook and add logWebhook (FIR-2218) by @mogery in https://github.com/mendableai/firecrawl/pull/1629 * Index testing improvements (FIR-2214) by @mogery in https://github.com/mendableai/firecrawl/pull/1637 * feat(sdk): Index parameters + other missing parameters (FIR-2240) by @mogery in https://github.com/mendableai/firecrawl/pull/1638 * feat(gcs-jobs): ditch exists check to cut lookup time in half by @mogery in https://github.com/mendableai/firecrawl/pull/1641 * fix(js-sdk/tests): fix the testing situation (FIR-2253) by @mogery in https://github.com/mendableai/firecrawl/pull/1644 * fix(readme): clarify that scrape_options must be a ScrapeOptions instance crawl_url and typo fixes by @Arkit003 in https://github.com/mendableai/firecrawl/pull/1647 * fix: PLAYWRIGHT_MICROSERVICE_URL in apps/api/.env.example by @pulbyte in https://github.com/mendableai/firecrawl/pull/1654 * Improve concurrency limit email notifications by @nickscamara in https://github.com/mendableai/firecrawl/pull/1658 * feat: add credits_billed everywhere (FIR-2286) by @mogery in https://github.com/mendableai/firecrawl/pull/1655 * fix(api): remove query parameter sanitization that was breaking extracts (FIR-2307) by @mogery in https://github.com/mendableai/firecrawl/pull/1661 * feat(index): remove unused columns by @mogery in https://github.com/mendableai/firecrawl/pull/1662 * feat: use index in search and extract (FIR-2267) by @mogery in https://github.com/mendableai/firecrawl/pull/1660 * feat(index): add domain splitting for improved map querying by @mogery in https://github.com/mendableai/firecrawl/pull/1666 ## New Contributors * @Arkit003 made their first contribution in https://github.com/mendableai/firecrawl/pull/1647 * @pulbyte made their first contribution in https://github.com/mendableai/firecrawl/pull/1654 **Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.10.0...v1.11.0 ## Introducing Search - v1.10.0 We’re excited to announce the launch of our new Search API endpoint that combines web search with Firecrawl’s powerful scraping capabilities. ### Search Features: - Search the web and get full content from results in one API call - Choose specific output formats (markdown, HTML, links, screenshots) - Customize search parameters (language, country, time range, number of results) - Full SDK support for Python and Node.js ### More Features - Auto mode proxy for scraping (`scrapeURL`, `js-sdk`) [#1551](https://github.com/mendableai/firecrawl/pull/1551), [#1602](https://github.com/mendableai/firecrawl/pull/1602) - Timeout handling and content type improvements for `scrapeURL/pdf` [#1570](https://github.com/mendableai/firecrawl/pull/1570), [#1604](https://github.com/mendableai/firecrawl/pull/1604), [#1592](https://github.com/mendableai/firecrawl/pull/1592) - Redis improvements: separate non-eviction Redis support [#1600](https://github.com/mendableai/firecrawl/pull/1600) - Search improvements: `ignoreBlockedURLs`, ignore concurrency limit [#1580](https://github.com/mendableai/firecrawl/pull/1580), [#1617](https://github.com/mendableai/firecrawl/pull/1617) - New `/cclog` endpoint for concurrency logging [#1589](https://github.com/mendableai/firecrawl/pull/1589) - Metadata extraction now includes `itemprop` attributes [#1624](https://github.com/mendableai/firecrawl/pull/1624) - Self-hosted: deployable Playwright image [#1625](https://github.com/mendableai/firecrawl/pull/1625) ### Fixes & Improvements - Better subdomain handling for `LLMs.txt` + bypass option [#1557](https://github.com/mendableai/firecrawl/pull/1557) - Improved URL validation and special character handling [#1547](https://github.com/mendableai/firecrawl/pull/1547) - Zombie worker cleanup + TTL handling for extract status [#1575](https://github.com/mendableai/firecrawl/pull/1575), [#1599](https://github.com/mendableai/firecrawl/pull/1599) - Fix concurrency queue logic and rate limiter override [#1595](https://github.com/mendableai/firecrawl/pull/1595), [#1593](https://github.com/mendableai/firecrawl/pull/1593) - Better logging for search pagination and robust fetch [#1572](https://github.com/mendableai/firecrawl/pull/1572), [#1588](https://github.com/mendableai/firecrawl/pull/1588) - Minor fixes: `og:locale:alternate`, adblock toggle, Playwright-only logic, malformed metadata arrays [#1597](https://github.com/mendableai/firecrawl/pull/1597), [#1616](https://github.com/mendableai/firecrawl/pull/1616), [#1574](https://github.com/mendableai/firecrawl/pull/1574) ### Testing & Docs - Add `MAX_RAM` and `MAX_CPU` environment variable docs [#1581](https://github.com/mendableai/firecrawl/pull/1581) - Testing infrastructure improvements [#1623](https://github.com/mendableai/firecrawl/pull/1623) ## What's Changed * Fix LLMs.txt cache bug with subdomains and add bypass option by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1557 * FIR-1951: Fix URL validation for special characters in query parameters by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1547 * feat(scrapeURL): proxy auto mode (FIR-1853) by @mogery in https://github.com/mendableai/firecrawl/pull/1551 * feat(scrapeURL/pdf/mu): add timeout and created_at (FIR-2008) by @mogery in https://github.com/mendableai/firecrawl/pull/1570 * fix(auto_charge): fix ACUC clear (FIR-1805) by @mogery in https://github.com/mendableai/firecrawl/pull/1571 * fix(api/search): log page options correctly (FIR-2015) by @mogery in https://github.com/mendableai/firecrawl/pull/1572 * Update docker-compose.yaml comment by @emircanerkul in https://github.com/mendableai/firecrawl/pull/1566 * hotfix: kill zombie workers, respect timeouts better (FIR-2034) by @mogery in https://github.com/mendableai/firecrawl/pull/1575 * Fix: Concatenate metadata arrays into strings with exceptions by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1574 * Fix sdk/undefined response handle error by @rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1578 * feat(python-sdk/CrawlWatcher): remove max payload size from WebSocket (FIR-2038) by @mogery in https://github.com/mendableai/firecrawl/pull/1577 * FIR-2006: Fix maxUrls and timeLimit parameters in Deep Research API by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1569 * docs: add MAX_RAM and MAX_CPU environment variables documentation by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1581 * feat(search): ignoreBlockedURLs (FIR-1954) by @mogery in https://github.com/mendableai/firecrawl/pull/1580 * fix(queue-worker): finish crawl if all addable URLs were already locked (FIR-1936) by @mogery in https://github.com/mendableai/firecrawl/pull/1582 * feat(api/extract): show extract as origin for scrapes originating from it (FIR-2061) by @mogery in https://github.com/mendableai/firecrawl/pull/1584 * feat(api/v1/extract): ignoreInvalidURLs (FIR-1948) by @mogery in https://github.com/mendableai/firecrawl/pull/1585 * fix(robustFetch): selective logging (FIR-2072) by @mogery in https://github.com/mendableai/firecrawl/pull/1588 * feat(scrapeURL, logJob): log pdf page count to db (FIR-2068) by @mogery in https://github.com/mendableai/firecrawl/pull/1587 * feat(concurrency-log): add cclog endpoint (FIR-2067) by @mogery in https://github.com/mendableai/firecrawl/pull/1589 * feat: parse PDFs on fc side and reject if too long for timeout (FIR-2083) by @mogery in https://github.com/mendableai/firecrawl/pull/1592 * feat(queue-worker/afterJobDone): improved ccq insert logic (FIR-2082) by @mogery in https://github.com/mendableai/firecrawl/pull/1595 * fix(v1): avoid overwriting rateLimiterMode with FIRE-1 rate limiter (FIR-2090) by @mogery in https://github.com/mendableai/firecrawl/pull/1593 * fix(html-transformer): bad outName for og:locale:alternate (FIR-2101) by @mogery in https://github.com/mendableai/firecrawl/pull/1597 * fix(extract-status): be able to get extract status even after TTL lapses by @mogery in https://github.com/mendableai/firecrawl/pull/1599 * feat(scrapeURL): add unnormalizedSourceURL for url matching DX (FIR-2137) by @mogery in https://github.com/mendableai/firecrawl/pull/1601 * feat(apps/api): add support for a separate, non-eviction Redis by @mogery in https://github.com/mendableai/firecrawl/pull/1600 * feat(js-sdk): auto mode proxy (FIR-2145) by @mogery in https://github.com/mendableai/firecrawl/pull/1602 * feat(scrapeURL): handle contentType JSON better in markdown conversion (FIR-2159) by @mogery in https://github.com/mendableai/firecrawl/pull/1604 * feat(scrapeURL/pdf): bill n credits per page (FIR-1934) by @mogery in https://github.com/mendableai/firecrawl/pull/1553 * [rust-sdk] webhook param for crawl by @palsp in https://github.com/mendableai/firecrawl/pull/1609 * feat(search): ignore concurrency limit for search (FIR-2187) by @mogery in https://github.com/mendableai/firecrawl/pull/1617 * fix(scrapeURL): only allow disabling the adblock on playwright (FIR-2200) by @mogery in https://github.com/mendableai/firecrawl/pull/1616 * feat(api/scrape): credits_billed column + handle billing for `/scrape` calls on worker side with stricter timeout enforcement (FIR-2162) by @mogery in https://github.com/mendableai/firecrawl/pull/1607 * Bypass billing on search preview by @nickscamara in https://github.com/mendableai/firecrawl/pull/1622 * feat: enhance metadata extraction by including 'itemprop' attribute in HTML by @ftonato in https://github.com/mendableai/firecrawl/pull/1624 * feat(selfhost): deploy a playwright image by @mogery in https://github.com/mendableai/firecrawl/pull/1625 * Testing improvements (FIR-2209) by @mogery in https://github.com/mendableai/firecrawl/pull/1623 * Index (FIR-2177) by @mogery in https://github.com/mendableai/firecrawl/pull/1605 ## New Contributors * @emircanerkul made their first contribution in https://github.com/mendableai/firecrawl/pull/1566 * @palsp made their first contribution in https://github.com/mendableai/firecrawl/pull/1609 **Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.9.0...v.10.0 ## Firecrawl v1.9.0 Release ### **What's New:** **Self-Host Improvements** - Supabase client fixes - Fixed support for LLM Providers - Crawl is much faster now - Global adoption of cacheable lookup system - Easier setup **MCP Improvements (v1.11.0)** - Tons of improvements to it (prompts, examples, and how to use params properly) **SDK & API Enhancements** - Added change tracking to SDK 2.0 - Crawl delay support with per-crawl concurrency limiting - New Qwen3 crawler example via OpenRouter - Cancel batch scrape endpoint **Performance & Limits** - Global adoption of cacheable lookup system - Increased map endpoint limit from 5,000 to 30,000 links - Search schema limit increased from 50 to 100 **Fixes & Stability** - Better error handling for SSL failures - Optional chaining bug fixes - WaitAction field validation in firecrawl-py - Concurrency queue reworked to prioritize by time, not priority **Dashboard (Cloud version)** - New activity logs ## What's Changed * (sdk/py) Change Tracking Added to 2.0 by @nickscamara in https://github.com/mendableai/firecrawl/pull/1506 * Fix sdk/schemas by @rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1507 * Fix: Handle both dict and model instances in actions parameter by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1508 * [feat] Implement GCS storage option for scrape results across controllers an… by @rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1500 * (batch/scrape) Cancel batch scrape endpoint by @nickscamara in https://github.com/mendableai/firecrawl/pull/1509 * Add Qwen3 web crawler example using OpenRouter by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1510 * Add crawl delay functionality with per-crawl concurrency limiting (FIR-249) by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1413 * Add delay parameter to crawl options in all SDKs by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1514 * refactor: maximum links limit for map endpoint from 5000 to 30000 by @ftonato in https://github.com/mendableai/firecrawl/pull/1523 * fix(concurrency-limit): rework cc queue to work by time not priority (FIR-1890) by @mogery in https://github.com/mendableai/firecrawl/pull/1526 * fix(queue-worker, scrape): stealth proxy pricing change (FIR-1847) by @mogery in https://github.com/mendableai/firecrawl/pull/1521 * feat(acuc): propagate team flags (FIR-1879) by @mogery in https://github.com/mendableai/firecrawl/pull/1522 * [Bug Fix] Make WaitAction milliseconds field optional in firecrawl-py by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1533 * create openAI provider using base url parameter by @y0hnn in https://github.com/mendableai/firecrawl/pull/1480 * refactor: increase max limit for search request schema from 50 to 100 by @ftonato in https://github.com/mendableai/firecrawl/pull/1545 * feat(scrapeURL): better error for SSL failures by @mogery in https://github.com/mendableai/firecrawl/pull/1552 * Fix/optional chaining operators missing by @tribixbite in https://github.com/mendableai/firecrawl/pull/1549 * feat(api/extract/fire-0): error logging (FIR-1971) by @mogery in https://github.com/mendableai/firecrawl/pull/1556 * feat: use cacheable lookup everywhere by @mogery in https://github.com/mendableai/firecrawl/pull/1559 * Mog/cachable lookup by @mogery in https://github.com/mendableai/firecrawl/pull/1560 * Add caching for RunPod PDF markdown results in GCS by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1561 * Fix Supabase client configuration errors when USE_DB_AUTHENTICATION is false by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1534 ## New Contributors * @y0hnn made their first contribution in https://github.com/mendableai/firecrawl/pull/1480 * @tribixbite made their first contribution in https://github.com/mendableai/firecrawl/pull/1549 **Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.8.0...v1.9.0 ## Launch Week III - v.1.8.0 ## Day 7 – Integration Day ### Overview For the final day of Launch Week III, we’re rolling out new and updated integrations that make it easier to connect Firecrawl to the tools and platforms you already use. From automation platforms to AI pipelines, Firecrawl now integrates with 20+ services, giving you a faster path from web data to workflow execution. ### Key Integration Updates - **Discord Bot:** Trigger scrapes and receive structured results right inside your server. [Learn more](https://github.com/mendableai/firecrawl-discord-bot) - **Make Integration:** Visually build workflows powered by Firecrawl's scraping and extraction. [Learn more](https://www.make.com/en/integrations/firecrawl) - **n8n Integration:** Connect Firecrawl to custom automation flows. [Learn more](https://n8n.io/) - **Langflow:** Seamlessly embed Firecrawl agents into Langflow pipelines. [Learn more](https://docs.langflow.org/) - **LlamaIndex:** Use Firecrawl with LlamaIndex to enrich and retrieve data intelligently. [Learn more](https://www.llamaindex.ai/) - **Dify:** Integrate Firecrawl with Dify to automate AI workflows. [Learn more](https://dify.ai/) More integrations are on the way — and if there’s one you’re missing, we’d love to hear about it. ## Day 6 – Firecrawl MCP ### Overview Today we’re launching a major upgrade to our Firecrawl MCP server, our implementation of the Model Context Protocol for LLM-connected scraping workflows. This release brings FIRE-1 agent support to the MCP, letting you unlock data hidden behind interaction barriers like logins and buttons — all via scrape and extract endpoints. We’re also introducing Server-Sent Events (SSE) support for local use, making setup and real-time integration easier than ever. These updates make it simpler to stream web data into LLM pipelines, with intelligent agents handling the heavy lifting. ### Key Features * FIRE-1 Support: Use the web action agent via MCP’s scrape and extract endpoints. * Interaction-Aware: Automatically navigates complex web barriers. * SSE Support: Stream output in real time — locally or from our hosted MCP server. * Plug-and-Play: Minimal setup needed for integration with your stack. ## Day 5 – Developer Day ### Overview Today is all about developers. We’re rolling out upgrades that make building with Firecrawl smoother and more scalable — whether you’re working in Python, Rust, or your favorite editor. We’ve introduced a fully async Python SDK with named params and return types, powerful new features in the Rust SDK, expanded team support on every plan, and a brand new Firecrawl Dark Theme for VSCode and compatible editors. ### Key Features - **Major Python SDK Improvements**: Named params, return types, and Async added to our Python SDK (**VERSION 2.0**) - **Rust SDK Upgrades**: Batch scraping, job cancellation, llms.txt generation, smarter search. - **Firecrawl Dark Theme**: A sleek new theme for VSCode, Cursor, and Windsurf. Download it here. - **QoL Improvements**: Plenty of small but meaningful changes to improve dev experience like no more team caps. All plans now include up to 20 seats. ## Day 4 – LLMstxt.new ### Overview Today we’re announcing [http://llmstxt.new](http://llmstxt.new) — the fastest way to turn any website into a clean, consolidated text file for LLMs. Just add `llmstxt.new/` in front of any URL, and you’ll get back a plain `.txt` file, optimized for AI training and inference. No boilerplate, no noise — just useful content. Built on top of Firecrawl, this tool makes it effortless to prepare real-world web content for use in LLM pipelines. ### Key Features - **Instant Usage**: Just add `llmstxt.new/` before any URL. - **Two Outputs**: `llms.txt` for concise summaries, `llms-full.txt` for full content. - **API Friendly**: Call via `http://llmstxt.new/{YOUR_URL}` or with a Firecrawl API key for full output. - **LLM-Ready Format**: Designed for both training and inference contexts. ## Day 3 – /extract v2 ### Overview Today we’re launching **/extract v2**, a major upgrade to our extraction system — powered by the FIRE-1 agent. With full support for **pagination, multi-step flows, and dynamic interactions**, extract v2 goes way beyond what we shipped back in January. It’s also now possible to extract data **without a URL**, using a built-in search layer to find the content you’re after. We’ve rebuilt the internals from the ground up — improved models, better architecture, and significantly better performance across our internal benchmarks. ### Key Features - **FIRE-1 Integration**: Automatically handles page actions, navigation, and interaction. - **Multi-Page Extraction**: Extract data across paginated or step-based flows. - **No URL Required**: Use built-in search to extract content without a direct link. - **Improved Accuracy**: New models and architecture mean better, faster results. ## Day 2 - FIRE-1 Agent ### Overview Meet **FIRE-1**, Firecrawl's first AI Agent built to take web scraping to the next level. With intelligent navigation and interaction capabilities, FIRE-1 can go far beyond traditional scraping methods. From handling pagination to interacting with dynamic site elements like buttons and links, FIRE-1 allows for powerful, context-aware scraping and extraction workflows. ### Key Features - **Smart Navigation**: Move through paginated or multi-step content automatically. - **Dynamic Interaction**: Click buttons, follow links, and interact with JavaScript-rendered components. ## Day 1 - Change Tracking ### Overview Change tracking is a powerful feature that allows you to monitor and detect changes in web content over time. It is available in both the JavaScript and Python SDKs. ### Key Features - **Detect Changes**: Identify if a webpage has changed since the last scrape. - **View Specific Changes**: Access detailed information about what has changed between scrapes. - **Structured Data**: Receive structured data about the changes. - **Visibility Control**: Manage the visibility of changes on your website. Using the `changeTracking` format, you can effectively monitor changes on a website and receive comprehensive information about the timestamp of the previous scrape, the result of the comparison between the two page versions, and the visibility of the current page/URL. ## Day 0 - Firecrawl Editor Theme ### Overview We're excited to release our official Firecrawl Editor Theme! Available now for most editors including Cursor, Windsurf, and more. ### Key Features The Firecrawl Editor Theme provides a clean, focused coding experience for everyone. Our color palette emphasizes readability while maintaining the Firecrawl brand identity. ### Download You can download the editor theme on the VS Code Marketplace [here](https://firecrawl.link/lw3-editor-theme). ## What's Changed * fix(js-sdk): isows import issues (FIR-1586) (FIR-1536) by @mogery in https://github.com/mendableai/firecrawl/pull/1411 * fix(rust-sdk): remove rustfmt by @washanhanzi in https://github.com/mendableai/firecrawl/pull/1392 * feat(log_job): start saving jobs to GCS by @mogery in https://github.com/mendableai/firecrawl/pull/1424 * Rename 'compare' format and property to 'changeTracking' by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1423 * feat(crawl-status): retrieve job data from GCS by @mogery in https://github.com/mendableai/firecrawl/pull/1427 * read from GCS (again) by @mogery in https://github.com/mendableai/firecrawl/pull/1433 * Add examples/ Llama 4 Maverick Crawler by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1416 * Add examples/llama4-maverick-web-extractor by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1418 * feat(api/search): add search endpoint and update request limits by @ftonato in https://github.com/mendableai/firecrawl/pull/1436 * feat(log_job): stop putting docs in the db by @mogery in https://github.com/mendableai/firecrawl/pull/1438 * feat(scrapeURL): reintroduce default timeout for simple queries by @mogery in https://github.com/mendableai/firecrawl/pull/1440 * ACUC: Dynamic Limits (FIR-1641) by @mogery in https://github.com/mendableai/firecrawl/pull/1434 * (feat/deep-research) Improvements to final analysis by @nickscamara in https://github.com/mendableai/firecrawl/pull/1443 * feat: incorporate user preferences and notification categories by @ftonato in https://github.com/mendableai/firecrawl/pull/1419 * (feat/change-tracking) Change Tracking Modes by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1445 * Add change tracking support to Python and JS SDKs by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1448 * Add waitFor of 5000ms for changeTracking format by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1450 * Add examples/gpt-4.1-crawler by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1460 * feat(scrape): get job result from GCS, avoid Redis by @mogery in https://github.com/mendableai/firecrawl/pull/1461 * (feat/fire-1) FIRE-1 by @nickscamara in https://github.com/mendableai/firecrawl/pull/1462 * Set default timeout to 120s when proxy is stealth by @devin-ai-integration in https://github.com/mendableai/firecrawl/pull/1464 * feat(extract): cost limit by @mogery in https://github.com/mendableai/firecrawl/pull/1473 * feat(rust): update rust sdk to support new features by @kkharji in https://github.com/mendableai/firecrawl/pull/1446 * [python-SDK] improvs/async by @rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1337 * Add examples/gpt-4.1 Company Researcher by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1477 * Python sdk/v2.1.0 by @rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1479 * [fix/sdk] kwargs params by @rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1490 * (fix/search) Search logs fix by @nickscamara in https://github.com/mendableai/firecrawl/pull/1491 * RM sb by @nickscamara in https://github.com/mendableai/firecrawl/pull/1492 * [feat] added second scrapeURLWithFireEngine by @rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1494 * Use `_async_monitor_job_status` in `AsyncFirecrawlApp` by @jmbledsoe in https://github.com/mendableai/firecrawl/pull/1498 * Add examples/o3 Web Crawler by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1487 * Add examples/o4-mini web crawler by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1489 * [SDK] fix: Update version to 2.4.0 and enhance ExtractResponse model with additi… by @rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1501 * [python-sdk] Webhook param for batch scrape by @arvida in https://github.com/mendableai/firecrawl/pull/1505 ## New Contributors * @washanhanzi made their first contribution in https://github.com/mendableai/firecrawl/pull/1392 * @kkharji made their first contribution in https://github.com/mendableai/firecrawl/pull/1446 * @jmbledsoe made their first contribution in https://github.com/mendableai/firecrawl/pull/1498 * @arvida made their first contribution in https://github.com/mendableai/firecrawl/pull/1505 **Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.7.0...v1.8.0 ## v1.7.0 - Release Notes ### New Features - **Deep Research Open Alpha**: Structured outputs + customizability. - **llmstxt.new**: Generate an llms.txt for any website by just appending its url: llmstxt.new/firecrawl.dev - **Concurrent Browsers**: Improved rate limits for all users. - **Compare Beta**: Figure what has changed in the website directly in /scrape and /crawl endpoints. Currently in closed beta. - **/extract**: URLs are now optional. - **/scrape**: Warns if concurrency-limited. - **New Firecrawl Examples**: Featuring models like Claude 3.7, Gemini 2.5, Deepseek V3, Mistral 3.1, and more. - **Crawl**: `maxDiscoveryDepth` option added. ### Fixes & Improvements - Fixed **circular JSON error** in search. - Reworked new **tally** system. - Fixed sitemaps poisoning crawler with unrelated links. - Crawler status retries added on failure (up to 3 times). - Credit check now snaps to remaining credits if exceeded. - Fixed path filtering bug in Map. - Removed unsupported schema in `llmExtract`. ## What's Changed * fix: resolve circular JSON structure error in search function by @invarrow in https://github.com/mendableai/firecrawl/pull/1330 * feat(crawl): add maxDiscoveryDepth by @mogery in https://github.com/mendableai/firecrawl/pull/1329 * tally rework api switchover by @mogery in https://github.com/mendableai/firecrawl/pull/1328 * (feat/pricing) Concurrent Browsers - Improve rate limits by @nickscamara in https://github.com/mendableai/firecrawl/pull/1331 * Fix SearxNG categories by @loorisr in https://github.com/mendableai/firecrawl/pull/1319 * (fix/map) Map failed to filter by path if indexed by @nickscamara in https://github.com/mendableai/firecrawl/pull/1333 * fix(crawler): sitemaps poisoning crawls with unrelated links by @mogery in https://github.com/mendableai/firecrawl/pull/1334 * fix(llmExtract): remove unsupported JSON schema properties (FIR-1246) by @mogery in https://github.com/mendableai/firecrawl/pull/1335 * Add/ Claude 3.7 implementation by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1336 * fix(js-sdk/crawl,batch-scrape): retry status call if it returns an error up to 3 times by @mogery in https://github.com/mendableai/firecrawl/pull/1343 * Increase maxurls for generate /llmstxt by @ericciarla in https://github.com/mendableai/firecrawl/pull/1349 * fix(v1/checkCredits): snap crawl limit to remaining credits if over without erroring out (FIR-1450) by @mogery in https://github.com/mendableai/firecrawl/pull/1350 * feat(scrape): add warning to document if it was concurrency limited by @mogery in https://github.com/mendableai/firecrawl/pull/1348 * (feat/extract) URLs can now be optional in /extract by @nickscamara in https://github.com/mendableai/firecrawl/pull/1346 * [SDKs] Added 403s to sdk error handlers by @rafaelsideguide in https://github.com/mendableai/firecrawl/pull/1357 * feat(scrape/actions/click): add all parameter (FIR-1443) by @mogery in https://github.com/mendableai/firecrawl/pull/1342 * (feat/deep-research) Add Analysis Prompt by @nickscamara in https://github.com/mendableai/firecrawl/pull/1351 * Add examples/ mistral-small-3.1-crawler by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1366 * Add examples/mistral 3.1 company researcher by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1369 * Add example/Deep-research Apartment finder by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1378 * (feat/deep-research) Deep Research Alpha v1 - Structured Outputs + Customizability by @nickscamara in https://github.com/mendableai/firecrawl/pull/1365 * Add examples/gemini-2.5-extractor by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1381 * Add examples/gemini-2.5-pro crawler by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1380 * feat(scrapeURL): return js returns from f-e (FIR-1535) by @mogery in https://github.com/mendableai/firecrawl/pull/1385 * Add examples/deepseek-v3-crawler by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1383 * Add examples/ Deepseek V3 Company Researcher by @aparupganguly in https://github.com/mendableai/firecrawl/pull/1384 * feat(queue-jobs): update notification logic for concurrency limits and add parameter (jsdocs) to batchScrapeUrls by @ftonato in https://github.com/mendableai/firecrawl/pull/1398 * feat(notification): add notification message for concurrency limit reached by @ftonato in https://github.com/mendableai/firecrawl/pull/1404 * compare format (FIR-1560) by @mogery in https://github.com/mendableai/firecrawl/pull/1405 * feat(queue-jobs): add function to determine job type and update notification logic for concurrency limits by @ftonato in https://github.com/mendableai/firecrawl/pull/1409 ## New Contributors * @invarrow made their first contribution in https://github.com/mendableai/firecrawl/pull/1330 **Full Changelog**: https://github.com/mendableai/firecrawl/compare/v1.6.0...v1.7.0