Features

• add @crawlee/stagehand package for AI-powered browser automation (#3331) (a89cb5a), closes #3064
• Add a counter of in-flight rendering type detections (#3355) (565fc34)
• implements async iterators (#3352) (7f7a4ab), closes #3338
• make handleCloudflareChallenge more configurable (#3247) (629daf8), closes #3127
• utils: add discoverValidSitemaps utility (#3339) (29f52ed)

Performance Improvements

• drop tsbuildinfo from published packages (#3243) (3450f27), closes #3239

3.15.3 (2025-11-10)

Bug Fixes

• await retries inside _timeoutAndRetry (#3206) (9c1cf6d), closes /github.com/apify/crawlee/pull/3188#discussion_r2410256271
• cli: support creating projects with names that start with a number (#3219) (3f37845), closes #3213
• use shared enqueue links wrapper in AdaptivePlaywrightCrawler (#3188) (9569d19)

Features

• support custom userAgent with respectRobotsTxtFile (#3226) (354252d), closes #3222

3.15.2 (2025-10-23)

Bug Fixes

• correctly apply launchOptions with useIncognitoPages (#3181) (84a4b70), closes /github.com/apify/crawlee/issues/3173#issuecomment-3346728227 #3173 #3173
• enable systemInfoV2 by default (#3208) (617a343)
• Re-export MemoryStorage from the crawlee package (#3200) (e183ea9)

Features

• export cheerio types in all crawler packages (#3204) (f05790b)

Performance Improvements

• don't await crawler.setStatusMessage (#3207) (1a67ffb)

3.15.1 (2025-09-26)

Bug Fixes

• use correct config for storage classes to avoid memory leaks (#3144) (911a2eb)

Performance Improvements

• Improve glob performance by reusing minimatch objects (#3168) (e5632e2)

3.15.0 (2025-09-17)

Bug Fixes

• ImpitHttpClient respects the internal Request timeout (#3103) (a35376d)
• proxyUrls list can contain null (#3142) (dc39cc2), closes #3136
• don't fail exportData calls on empty datasets (#3115) (298f170), closes #2734
• respect maxCrawlDepth with a custom enqueueLinks transformRequestFunction (#3159) (e2ecb74)

Features

• add collectAllKeys option for BasicCrawler.exportData (#3129) (2ddfc9c), closes #3007
• add TandemRequestProvider for combined RequestList and RequestQueue usage (#2914) (4ca450f), closes #2499

3.14.1 (2025-08-05)

Note: Version bump only for package @crawlee/root

3.14.0 (2025-07-25)

Bug Fixes

• don't retire browsers with long-running pre|postLaunchHooks prematurely (#3062) (681660e)
• respect exclude option in enqueueLinksByClickingElements (#3058) (013eb02)
• retry on blocked status codes in HttpCrawler (#3060) (b5fcd79), closes /github.com/apify/crawlee/blob/f68d2a95d67cc6230122dc1a5226c57ca23d0ae7/packages/browser-crawler/src/internals/browser-crawler.ts#L481-L486 #3029
• validation of iterables when adding requests to the queue (#3091) (529a1dd), closes #3063

Features

• add maxCrawlDepth crawler option (#3045) (0090df9), closes #2633
• Add llms.txt and llms-full.txt generator (#3061) (79f3ba5), closes #3046

3.13.10 (2025-07-09)

Bug Fixes

• call onSkippedRequest for AdaptivePlaywrightCrawler.enqueueLinks (#3043) (fc23d34), closes #3026 #3039
• improve enqueueLinks limit checking (#3038) (2774124), closes #3037

3.13.9 (2025-06-27)

Bug Fixes

• Do not log 'malformed sitemap content' on network errors in Sitemap.tryCommonNames (#3015) (64a090f), closes #2884
• Fix link filtering in enqueueLinks in AdaptivePlaywrightCrawler (#3021) (8a3b6f8), closes #2525

Features

• Accept (Async)Iterables in addRequests methods (#3013) (a4ab748), closes #2980
• Report links skipped because of various filter conditions (#3026) (5a867bc), closes #3016

3.13.8 (2025-06-16)

Bug Fixes

• Do not enqueue more links than what the crawler is capable of processing (#2990) (ea094c8), closes #2728
• Persist rendering type detection results in AdaptivePlaywrightCrawler (#2987) (76431ba), closes #2899

Features

• dataset: add collectAllKeys option for full CSV export (#2945) (#3007) (3b629da)
• support KVS.listKeys() prefix and collection parameters (#3001) (5c4726d), closes #2974

3.13.7 (2025-06-06)

Bug Fixes

• use merged cookies correctly in GotScrapingHttpClient (#3000) (a2985f2), closes #2991

3.13.6 (2025-06-05)

Bug Fixes

• enable full cookie support for ImpitHttpClient (#2991) (120f0a7)
• ensure PlaywrightGotoOptions won't result in unknown when playwright is not installed (#2995) (93eba38), closes #2994
• extract only body from iframe elements (#2986) (c36166e), closes #2979

3.13.5 (2025-05-20)

Features

• add MinimumSpeedStream and ByteCounterStream helpers (#2970) (921c4ee)
• Allow the AdaptivePlaywrightCrawler result comparator to signal an inconclusive result (#2975) (7ba8906)

3.13.4 (2025-05-14)

Bug Fixes

• core: respect systemInfoV2 in snapshotter (#2961) (4100eab), closes #2958
• core: use short timeouts for periodic KVS.setRecord calls (#2962) (d31d90e)
• core: optimize request unlocking to get rid of unnecessary unlock calls (#2963) (a433037)
• social: extract emails from each text node separately (#2952) (799afc1)

Features

• impit-client: bump impit version to v0.4 (#2954) (7a1bf10)

3.13.3 (2025-05-05)

Bug Fixes

• await _createPageForBrowser in browser pool (#2950) (27ba74b), closes #2789
• convert [@apilink](https://github.com/apilink) to [@link](https://github.com/link) on build (#2949) (abe1dee), closes #2717
• disable default fingerprints in the Camoufox template (#2928) (6dffa00)
• Fix trailing slash removal in BrowserPool (#2921) (c1fc439), closes #2878
• Fix useState behavior in adaptive crawler (#2941) (5282381)
• Persist SitemapRequestList state periodically (#2923) (e6e7a9f), closes #2897
• respect autoscaledPoolOptions.isTaskReadyFunction option (#2948) (fe2d206), closes #2922
• statistics: track actual request.retryCount in Statistics (#2940) (c9f7f54)

Features

• pass response to FileDownload.streamHandler (#2930) (008c4c7)

3.13.2 (2025-04-08)

Features

• add onSkippedRequest option (#2916) (764f992), closes #2910

3.13.1 (2025-04-07)

Bug Fixes

• don't double increment session usage count in BrowserCrawler (#2908) (3107e55), closes #2851
• rename RobotsFile to RobotsTxtFile (#2913) (3160f71), closes #2910
• treat 406 as other 4xx status codes in HttpCrawler (#2907) (b0e6f6d), closes #2892

Features

• add respectRobotsTxtFile crawler option (#2910) (0eabed1)

3.13.0 (2025-03-04)

Bug Fixes

• cheerio: don't decode HTML entities in context.body (#2838) (32d6d0e), closes #2401
• install browser in the camoufox template correctly (#2864) (a9d008c), closes #2863
• Make log message in RequestQueue.isFinished more accurate (#2848) (3d124ae)
• Simplified RequestQueueV2 implementation (#2775) (d1a094a), closes #2767 #2700

Features

• Camoufox-based crawler template (#2842) (7f08de4), closes #2836
• improved cross platform metric collection (#2834) (e41b2f7), closes #2771
• playwright: add handleCloudflareChallenge helper (#2865) (9a1725f)
• remove old docker CI (#2831) (7f09d56)
• use native impit streaming (#2833) (af2fe23), closes #2756

3.12.2 (2025-01-27)

Bug Fixes

• core: type definition of Dataset.reduce (#2774) (59bc6d1), closes #2773
• destructure CrawlerRunOptions before passing them to addRequests (#2803) (02a598c), closes #2802
• graceful BasicCrawler tidy-up on CriticalError (#2817) (53331e8), closes #2807

Features

• impit-based HttpClient implementation (#2787) (61d7ffa)
• add support for parsing comma-separated list environment variables (#2765) (4e50c47)
• stopping the crawlers gracefully with BasicCrawler.stop() (#2792) (af2966f), closes #2777

3.12.1 (2024-12-04)

Bug Fixes

• log status message timeouts to debug level (55ee44a)
• social: support new URL formats for Facebook, YouTube and X (#2758) (4c95847), closes #525

Features

• tieredProxyUrls accept null for switching the proxy off (#2743) (82f4ea9), closes #2740

3.12.0 (2024-11-04)

Bug Fixes

• .trim() urls from pretty-printed sitemap.xml files (#2709) (802a6fe), closes #2698
• ensure correct column order in CSV export (#2734) (b66784f), closes #2718
• ignore errors from iframe content extraction (#2714) (627e5c2), closes #2708
• update fingerprintGeneratorOptions types (#2705) (fcb098d), closes #2703

Features

• allow using other HTTP clients (#2661) (568c655), closes #2659

3.11.5 (2024-10-04)

Bug Fixes

• forefront request fetching in RQv2 (#2689) (03951bd), closes #2669
• prolong- and deleteRequestLock forefront option (#2690) (cba8da3), closes #2681 #2689 #2669
• check .isFinished() before RequestList reads (#2695) (6fa170f)
• core: accept UInt8Array in KVS.setValue() (#2682) (8ef0e60)
• core: trigger errorHandler for session errors (#2683) (7d72bcb), closes #2678
• decode special characters in proxy username and password (#2696) (0f0fcc5)
• http-crawler: avoid crashing when gotOptions.cache is on (#2686) (1106d3a)
• puppeteer: rename ignoreHTTPSErrors to acceptInsecureCerts to support v23 (#2684) (f3927e6)
• respect forefront option in MemoryStorage's RequestQueue (#2681) (b0527f9), closes #2669

3.11.4 (2024-09-23)

Bug Fixes

• SitemapRequestList.teardown() doesn't break persistState calls (#2673) (fb2c5cd), closes /github.com/apify/crawlee/blob/f3eb99d9fa9a7aa0ec1dcb9773e666a9ac14fb76/packages/core/src/storages/sitemap_request_list.ts#L446 #2672

3.11.3 (2024-09-03)

Bug Fixes

• improve FACEBOOK_REGEX to match older style page URLs (#2650) (a005e69), closes #2216
• RequestQueueV2: reset recently handled cache too if the queue is pending for too long (#2656) (51a69bc)

3.11.2 (2024-08-28)

Bug Fixes

• RequestQueueV2: remove inProgress cache, rely solely on locked states (#2601) (57fcb08)
• use namespace imports for cheerio to be compatible with v1 (#2641) (f48296f)
• Use the correct mutex in memory storage RequestQueueClient (#2623) (2fa8a29)

Features

• globs & regexps for SitemapRequestList (#2631) (b5fd3a9)
• resilient sitemap loading (#2619) (1dd7660)

3.11.1 (2024-07-24)

Bug Fixes

• use getHTML in the shadow root expansion (#2587) (a244d62), closes #2583

3.11.0 (2024-07-09)

Features

• add iframe expansion to parseWithCheerio in browsers (#2542) (328d085), closes #2507
• add ignoreIframes opt-out from the Cheerio iframe expansion (#2562) (474a8dc)
• Sitemap-based request list implementation (#2498) (7bf8f0b)

3.10.5 (2024-06-12)

Bug Fixes

• allow creating new adaptive crawler instance without any parameters (9b7f595)
• declare missing peer dependencies in @crawlee/browser package (#2532) (3357c7f)
• fix detection of HTTP site when using the useState in adaptive crawler (#2530) (7e195c1)
• mark context.request.loadedUrl and id as required inside the request handler (#2531) (2b54660)

3.10.4 (2024-06-11)

Bug Fixes

• add waitForAllRequestsToBeAdded option to enqueueLinks helper (925546b), closes #2318
• add missing useState implementation into crawling context (eec4a71)
• make crawler.log publicly accessible (#2526) (3e9e665)
• playwright: allow passing new context options in launchOptions on type level (0519d40), closes #1849
• respect crawler.log when creating child logger for Statistics (0a0d75d), closes #2412

3.10.3 (2024-06-07)

Bug Fixes

• adaptive-crawler: log only once for the committed request handler execution (#2524) (533bd3f)
• increase timeout for retiring inactive browsers (#2523) (195f176)
• respect implicit router when no requestHandler is provided in AdaptiveCrawler (#2518) (31083aa)
• revert the scaling steps back to 5% (5bf32f8)

Features

• add waitForSelector context helper + parseWithCheerio in adaptive crawler (#2522) (6f88e73)
• log desired concurrency in the default status message (9f0b796)

3.10.2 (2024-06-03)

Bug Fixes

• Autodetect sitemap filetype from content (#2497) (62a9f40), closes #2461
• improve fix for double extension in KVS with HTML files (#2505) (157927d), closes #2419

Features

• Loading sitemaps from string (#2496) (38ed0d6), closes #2460

3.10.1 (2024-05-23)

Bug Fixes

• adjust URL_NO_COMMAS_REGEX regexp to allow single character hostnames (#2492) (ec802e8), closes #2487
• investigate and temp fix for possible 0-concurrency bug in RQv2 (#2494) (4ebe820)
• provide URLs to the error snapshot (#2482) (7f64145), closes /github.com/apify/apify-sdk-js/blob/master/packages/apify/src/key_value_store.ts#L25

3.10.0 (2024-05-16)