Scrape API & Extraction Improvements
Features & Enhancements: Added action and wait time validation for scrape API, improved PDF/image sub-link detection and text extraction via Gemini, enhanced multi-entity extraction prompts, moved source tracking out of experimental, added Serper and Search API env vars to docker-compose, updated credit system terminology from "credits" to "tokens".
Examples: Implemented Gemini 2.0 crawler example, Gemini TrendFinder, and Open Deep Research.
Fixes: Updated HTML transformer free_string parameter type, improved Gemini crawler library and PDF link extraction, fixed crawl queue worker page counting, fixed relative URL conversion, and enforced scrape rate limits in batch scraping.
Fetched April 11, 2026