The /llmstxt endpoint allows you to transform any website into clean, LLM-ready text files. Simply provide a URL, and Firecrawl will crawl the site and generate both llms.txt and llms-full.txt files that can be used for training or analysis with any LLM.
Docs here: https://docs.firecrawl.dev/features/alpha/llmstxt
The /deep-research endpoint enables AI-powered deep research and analysis on any topic. Simply provide a research query, and Firecrawl will autonomously explore the web, gather relevant information, and synthesize findings into comprehensive insights.
Join the waitlist here: https://www.firecrawl.dev/deep-research
Introducing the Firecrawl MCP Server. Give Cursor, Windsurf, Claude enhanced web extraction capabilities. Big thanks to @vrknetha, @cawstudios for the initial implementation!
See here: https://github.com/mendableai/firecrawl-mcp-server
Full Changelog: v1.5.0...v1.6.0
Full Changelog: https://github.com/mendableai/firecrawl/compare/v1.5.0...v1.6.0
SELF_HOST.md and docker-compose.yaml have been updated for clarity and compatibility/search endpoint (#1193)For the complete details, check out the full changelog.
Full Changelog: https://github.com/mendableai/firecrawl/compare/v1.4.4...v1.5.0
🚀 Features & Enhancements
✏️ Examples
🐛 Fixes
Full Changelog: https://github.com/mendableai/firecrawl/compare/v1.4.3...v1.4.4
Full Changelog: https://github.com/mendableai/firecrawl/compare/v1.4.2...v1.4.3
We're excited to announce several new features and improvements:
enableWebSearch parameter__experimental_showSources parametertimeout parameter for map endpointblockAds parameter (enabled by default)Full Changelog: https://github.com/mendableai/firecrawl/compare/1.4.1...v1.4.2 Firecrawl website changelog: https://firecrawl.dev/changelog
We've significantly enhanced our data extraction capabilities with several key updates:
Full Changelog: https://github.com/mendableai/firecrawl/compare/v1.4.0...1.4.1
We’re excited to announce the release of /extract - get data from any website with just a prompt. With /extract, you can retrieve any information from anywhere on a website without being limited by scraping roadblocks or the typical context constraints of LLMs.
No more manual copy-pasting, broken scraping scripts, or debugging LLM calls. - it’s never been easier to enrich your data, create datasets, or power AI applications with clean, structured data from any website.
Companies are already using extract to:
Instead of spending hours manually researching, fixing broken scrapers, or piecing together data from multiple sources, simply specify what information you need and the target website, and let the Firecrawl handle the entire retrieval process.
Specifically, you can:
This versatility translates into a wide range of real-world applications—enabling you to enrich web data for just about any use case.
Curious to try /extract out for yourself? Visit our playground to try out /extract - you get 500,000 tokens for free Dive into our Extract Beta documentation for detailed technical guidance and API reference Want a no-code solution? Connect /extract to thousands of applications through our enhanced Zapier integration
That's all for now! Happy Extracting from the whole Firecrawl team 🔥
Full Changelog: https://github.com/mendableai/firecrawl/compare/v.1.3.0...v1.4.0
Full Changelog: https://github.com/mendableai/firecrawl/compare/v1.2.1...v.1.3.0
We have updated the /extract endpoint to now be asynchronous. When you make a request to /extract, it will return an ID that you can use to check the status of your extract job. If you are using our SDKs, there are no changes required to your code, but please make sure to update the SDKs to the latest versions as soon as possible.
For those using the API directly, we have made it backwards compatible. However, you have 10 days to update your implementation to the new asynchronous model.
For more details about the parameters, refer to the docs sent to you.
Full Changelog: https://github.com/mendableai/firecrawl/compare/v1.2.0...v1.2.1
Changelog: https://www.firecrawl.dev/changelog#/extract-changes
The search endpoint combines web search with Firecrawl’s scraping capabilities to return full page content for any query.
Include scrapeOptions with formats: ["markdown"] to get complete markdown content for each search result otherwise it defaults to getting SERP results (url, title, description).
More info here /v1/search docs
Full Changelog: https://github.com/mendableai/firecrawl/compare/v1.1.1...v1.2.0
Full Changelog: https://github.com/mendableai/firecrawl/compare/v1.1.0...v1.1.1
Starting today we are going to be posting weekly releases here and on firecrawl.dev/changelog. This release is just a summary of all the improvements and fixes we pushed since v1 release here. Thank you all for the contributions!
/scrape and /crawl./v1. by @shige in https://github.com/mendableai/firecrawl/pull/635dotenv and uuid to devDependencies, fix zod import by @MonsterDeveloper in https://github.com/mendableai/firecrawl/pull/614SELF_HOST.md by @Mefisto04 in https://github.com/mendableai/firecrawl/pull/799satisfies instead of as for ts example by @twlite in https://github.com/mendableai/firecrawl/pull/831WebScraper refactor into scrapeURL by @mogery in https://github.com/mendableai/firecrawl/pull/714/map to only return links present in the sitemap by @nickscamara in https://github.com/mendableai/firecrawl/pull/901Full Changelog: https://github.com/mendableai/firecrawl/compare/v1.0.0...v1.1.0
We're gonna be updating the release every week now and will be updating it on firecrawl.dev/changelog too!
Firecrawl V1 is here! With that we introduce a more reliable and developer friendly API.
August 29th, 2024
Here is what’s new:
Learn more about it here
Start using v1 right away at https://firecrawl.dev
Full Changelog: https://github.com/mendableai/firecrawl/commits/v1.0.0