Firecrawl opens its scrapers to files and third-party data feeds

One endpoint for everything you want to feed an agent

Firecrawl's headline move this week was Universal Scrape, which broadens the familiar /scrape endpoint well past HTML. Point it at a dynamic website and it still works, but the same call now reads PDFs, images, Word documents, spreadsheets, and presentations by URL — the formats agents usually choke on because they need a separate parsing pipeline. More interesting is the second half: the endpoint can also call Alexandria provider capabilities using a Firecrawl key and credits, so a single request can pull in people and company data from Apollo, FullEnrich, and Data Legion, financial data from Fiscal.ai and Benzinga, or podcast data from Particle. Some providers require an org admin to accept terms before the first call, so teams wiring this into production should sort out billing permissions before writing code. For anyone building agents on top of scraped context, this collapses what used to be several integrations into one billing surface.

Firecrawl also kept expanding what Alexandria itself covers, adding fantasy sports data so agents in ChatGPT or Claude can research lineups by combining rankings, projections, player and team stats, betting odds, and news. A demo lets you paste a roster and compare projections across scoring formats to see suggested start/sit changes — a narrow vertical, but a clear illustration of the provider model Universal Scrape now exposes more generally.

Apify's SDKs head toward 4.x

Apify's Python SDK picked up idempotency keys for Actor.charge(), letting actors retry a charge without double-billing the same run — the kind of thing that matters once you're charging per event in a scraper that may itself retry. The same release lets the SDK read Actor input from the working directory when the Apify store has none, which simplifies local development and testing. Both changes point at the same goal: making the Python SDK behave predictably outside the platform as well as inside it.

Crawlee, meanwhile, is deep in a breaking 4.x transition. The release candidate splits Crawler-only code out of @crawlee/core into @crawlee/basic, moves @crawlee/jsdom and @crawlee/linkedom out of the monorepo entirely, and delegates input handling to the Apify SDK, which now requires apify >= 4.0.0-beta.40. Two behavioral changes deserve attention: the request queue is no longer emptied between run() calls, and statistics naming plus the IRequestManager and IBrowserPool contracts have shifted. A new DomCrawler also lands in this cut. If you maintain a Crawlee-based scraper, the request-queue change is the one most likely to alter existing run semantics, so it's worth testing before the stable release.