Apify

Crawlee 4.0 hit release candidate while the Python SDK shipped 4.0 and 4.1, adding actor restart, task start, and idempotent charging.

Crawlee 4.0 nears stable, with breaking changes.

• @crawlee/jsdom and @crawlee/linkedom moved out of the monorepo, and crawler-only code moved from @crawlee/core to @crawlee/basic with more internals made private.
• Input handling moved to the Apify SDK, which now requires apify >= 4.0.0-beta.40.
• The request queue is no longer emptied between run() calls, so queued requests carry over.
• New DomCrawler with a pluggable DOM parser, OpenTelemetry instrumentation via @crawlee/otel, skipRequest and afterCommit context helpers, an initialConcurrency option, and extractMicrodata.

An upgrade guide covers the migration path.

Python SDK 4.1 added run lifecycle control. Actor.resurrect restarts a finished run against its original storages, and Actor.start_task starts a task without waiting for it to finish. Actor.charge() takes an idempotency_key so retried charges don't double-bill, and actors read input from the working directory when the store has none.

Python SDK 4.0 GA opened up the platform environment. Actor.get_env() exposes platform env vars that were previously unreachable, and apify-client errors now re-export from apify.errors. Scrapy proxy errors log correctly and requests that fail to convert skip instead of crashing the run.

Reliability fixes landed across both. Shared request queue locks are prolonged and tracked to prevent duplicate processing, and Scrapy no longer drops in-flight requests or redirects. Browser-pool fixes survive a page.close() that never settles and track updated Cloudflare challenge markup. Storage already in use is no longer purged mid-run, robots.txt 404s now mean allow-all, and HTTP redirects carry credentials, cookies, and request bodies correctly.