v3.13.1

3.13.1 (2025-04-07)

Bug Fixes

• don't double increment session usage count in BrowserCrawler (#2908) (3107e55), closes #2851
• rename RobotsFile to RobotsTxtFile (#2913) (3160f71), closes #2910
• treat 406 as other 4xx status codes in HttpCrawler (#2907) (b0e6f6d), closes #2892

Features

• add respectRobotsTxtFile crawler option (#2910) (0eabed1)