Change Log
All notable changes to this project will be documented in this file.
See Conventional Commits for commit guidelines.
3.18.1 (2026-08-12)
Bug Fixes
3.18.0 (2026-08-04)
Bug Fixes
- allow any
clickOptions in enqueueLinksByClickingElements (#3823) (abcbfb1)
- avoid duplicate final crawler persistence (#3866) (79dd6c4), closes #3167
- basic-crawler: declare
@apify/datastructures as a dependency (#3815) (4e9d227)
- core: avoid re-submitting already-enqueued requests in addRequestsBatched (#3843) (704c269), closes #2456 #2456 #2456 #3120
- core: cap retries for unprocessed requests in addRequestsBatched (#3765) (b3170a6), closes #3764
- core: clear
AutoscaledPool.pause() polling interval on timeout (#3847) (42b8914)
- core: ignore explicitly undefined baseUrl in enqueueLinks helpers (#3924) (a58a9c1)
- core: keep crawler enqueue limits when options carry explicit undefined (#3927) (5f1b0ab), closes #3924 #3924
- core: keep same-domain filtering after an off-domain redirect (#3923) (9b312ef), closes #3921
- core: remove
RequestList persist state listener on teardown (#3846) (63394ee)
- core: skip malformed cookie fragments in mergeCookies (#3822) (b3c3ef2)
- core: unblock backpressured sitemap load on persistState (#3863) (0b31a8b)
- declare missing dependencies (#3817) (5a9a661)
- filter sitemap-derived URLs by enqueue strategy (#3797) (9dc3160)
- impit-client: honor redirect handler for session cookie parity (#3938) (9371301)
- make crawler terminal status message reliably delivered (#3733) (7aed264)
- memory-storage: don't increment
handledRequestCount when updating a handled request (#3827) (4739d48)
- memory-storage: prevent storage names from escaping the storage directory (#3715) (a04c297)
- memory-storage: use platform basename in key-value store warning (#3864) (dcefe01)
- playwright,puppeteer: only intercept requests for non-GET methods or payloads (#3819) (1942f1e)
- playwright: silence transient errors in
enqueueLinksByClickingElements (#3732) (7b98cc3), closes #3216
- read properties directly from
GotScrapingHttpClient response (#3807) (c673c8d)
- respect falsy transformRequestFunction return in context enqueueLinks (#3925) (6cb4fc3), closes #3920
- sitemap: recognize sitemap_index as a valid input sitemap url (#3909) (eacab5e)
- stagehand: wait for Stagehand to register a page before using it (#3865) (238214f)
- update handleCloudflareChallenge for new Cloudflare challenge markup (#3717) (5baa8bd), closes #3629 apify/camoufox-js#283 #3629
- utils: advance the CPU tick baseline in systemInfoV2 getCurrentCpuTicks (#3824) (bb23800)
- utils: allow apostrophes and asterisks in extracted URLs (#3790) (e9baffd), closes #2755
- utils: prevent ReDoS in social email regex (#3845) (d6a2b39)
- utils: prevent sitemap metadata leak across url without loc (#3795) (895ed62)
- utils: use article:* selectors for article Open Graph properties (#3789) (b660506)
Features
Performance Improvements
- core: improve
RequestList initialization speed on large arrays (#3754) (c451c13), closes #3750
3.17.0 (2026-06-04)
Bug Fixes
- add network timeouts to discoverValidSitemaps to prevent indefinite hangs (#3429) (ee90835)
- bind
Browser.pages() correctly in PuppeteerPlugin (#3439) (c3a4b3b)
- browser-pool: allow unsetting launch context proxy URL (#3621) (d5ab219)
- browser-pool: preserve caller's AbortContext across p-limit queue (#3673) (c4d58d9), closes #3670
- BrowserCrawler closes ctx.page before errorHandler runs on navig… (#3655) (ab34886)
- change sequential approach to parallel for Iterator first page (#3402) (adf3dae), closes #3395
- core: apply object searchParams in HTTP request options (#3622) (d32946e)
- core: release storage open queue slot on failure (#3662) (18ebc61), closes #3661
- correct failed requests delta count in periodic log message (#3482) (a098f42)
- Correctly track the number of requests handled by a crawler (#3410) (b2296ce), closes #3083 #3330
- e2e: resolve flaky Camoufox tests due to GitHub rate limits (#3527) (08c98cf), closes #3524
- honor custom respectRobotsTxtFile userAgent in enqueueLinks (#3578) (1d2528b)
- log warning when Sitemap.load returns empty sitemap due to loading error (#3462) (2cf9c1c)
- parse proxy URLs correctly when using
useIncognitoPages (#3433) (db2bb68)
- Prevent accidental request dropping with
maxRequestsPerCrawl (#3531) (b23319b)
- prevent iframe expansion failure on pages with Trusted Types CSP (#3590) (c0b9b50), closes #3588
- prevent v3 sitemap discovery init hangs in discoverValidSitemaps (#3434) (6c04f92), closes #3412
- unref BrowserController.close() fallback timer to avoid keeping Node alive longer than needed (#3671) (eef94d4)
- using GET for candidate checks - sitemaps (#3464) (f82eb04), closes #3463
Features
- adaptive-crawler: add shouldPropagateError callback to prevent browser fallback (#3635) (d3a29d9)
- add
ignoreProxyCertificate option for the internal proxy-chain instance (#3418) (02eec66), closes #3369
- add abortDownload context helper to FileDownload (#3528) (3b3d8f6)
- Dynamic memory snapshots (#3471) (87c67dc), closes #3408
- expose Download objects on PlaywrightCrawlingContext (#3596) (5dc1627), closes #3583
- Modularize SystemStatus to allow custom backpressure mechanisms (#3529) (7159a13)
- templates: use ImpitHttpClient and install optional deps in Docker (#3603) (113dca6), closes #3601
- utils: add sitemapFilter option to parseSitemap (#3557) (1d4f6b9)
Performance Improvements
3.16.0 (2026-02-06)
Bug Fixes
- adaptive-crawler: Fix persistence of rendering type detection results (#3368) (4abca8b)
- certain redirect responses change request methods (#3296) (efac644), closes #2586
- clean turbo cache and tsbuildinfo files in yarn clean (#3348) (6cd9456)
- core: ensure
maxCrawlDepth warning is logged only once (#3337) (9d01334), closes #3336
- handle multiple
BasicCrawler.stop() calls correctly (#3324) (9c0580b), closes #3257
- impit-client: pause fromWeb stream to prevent early consumption (#3347) (72aacb4), closes #555
- more permissive accept (#3373) (d03af1b), closes #3242
- remove deprecation from
RequestQueueV1 (#3341) (89309bc)
- suppress info message for undefined maxRequestsPerCrawl (#3237) (f3d9a79)
Features
Performance Improvements
3.15.3 (2025-11-10)
Bug Fixes
Features
3.15.2 (2025-10-23)
Bug Fixes
Features
- export cheerio types in all crawler packages (#3204) (f05790b)
Performance Improvements
3.15.1 (2025-09-26)
Bug Fixes
- use correct config for storage classes to avoid memory leaks (#3144) (911a2eb)
Performance Improvements
- Improve glob performance by reusing minimatch objects (#3168) (e5632e2)
3.15.0 (2025-09-17)
Bug Fixes
ImpitHttpClient respects the internal Request timeout (#3103) (a35376d)proxyUrls list can contain null (#3142) (dc39cc2), closes #3136- don't fail
exportData calls on empty datasets (#3115) (298f170), closes #2734
- respect
maxCrawlDepth with a custom enqueueLinks transformRequestFunction (#3159) (e2ecb74)
Features
- add
collectAllKeys option for BasicCrawler.exportData (#3129) (2ddfc9c), closes #3007
- add
TandemRequestProvider for combined RequestList and RequestQueue usage (#2914) (4ca450f), closes #2499
3.14.1 (2025-08-05)
Note: Version bump only for package @crawlee/root
3.14.0 (2025-07-25)
Bug Fixes
Features
Bug Fixes
3.13.9 (2025-06-27)
Bug Fixes
- Do not log 'malformed sitemap content' on network errors in
Sitemap.tryCommonNames (#3015) (64a090f), closes #2884
- Fix link filtering in enqueueLinks in AdaptivePlaywrightCrawler (#3021) (8a3b6f8), closes #2525
Features
3.13.8 (2025-06-16)
Bug Fixes
- Do not enqueue more links than what the crawler is capable of processing (#2990) (ea094c8), closes #2728
- Persist rendering type detection results in
AdaptivePlaywrightCrawler (#2987) (76431ba), closes #2899
Features
3.13.7 (2025-06-06)
Bug Fixes
- use merged cookies correctly in
GotScrapingHttpClient (#3000) (a2985f2), closes #2991
3.13.6 (2025-06-05)
Bug Fixes
- enable full cookie support for
ImpitHttpClient (#2991) (120f0a7)
- ensure
PlaywrightGotoOptions won't result in unknown when playwright is not installed (#2995) (93eba38), closes #2994
- extract only
body from iframe elements (#2986) (c36166e), closes #2979
3.13.5 (2025-05-20)
Features
- add
MinimumSpeedStream and ByteCounterStream helpers (#2970) (921c4ee)
- Allow the AdaptivePlaywrightCrawler result comparator to signal an inconclusive result (#2975) (7ba8906)
3.13.4 (2025-05-14)
Bug Fixes
- core: respect
systemInfoV2 in snapshotter (#2961) (4100eab), closes #2958
- core: use short timeouts for periodic
KVS.setRecord calls (#2962) (d31d90e)
- core: optimize request unlocking to get rid of unnecessary unlock calls (#2963) (a433037)
- social: extract emails from each text node separately (#2952) (799afc1)
Features
3.13.3 (2025-05-05)
Bug Fixes
- await
_createPageForBrowser in browser pool (#2950) (27ba74b), closes #2789
- convert
[@apilink](https://github.com/apilink) to [@link](https://github.com/link) on build (#2949) (abe1dee), closes #2717
- disable default fingerprints in the Camoufox template (#2928) (6dffa00)
- Fix trailing slash removal in BrowserPool (#2921) (c1fc439), closes #2878
- Fix useState behavior in adaptive crawler (#2941) (5282381)
- Persist SitemapRequestList state periodically (#2923) (e6e7a9f), closes #2897
- respect
autoscaledPoolOptions.isTaskReadyFunction option (#2948) (fe2d206), closes #2922
- statistics: track actual request.retryCount in Statistics (#2940) (c9f7f54)
Features
- pass
response to FileDownload.streamHandler (#2930) (008c4c7)
3.13.2 (2025-04-08)
Features
3.13.1 (2025-04-07)
Bug Fixes
Features
3.13.0 (2025-03-04)
Bug Fixes