How I scraped 4M product pages in 36 hours without getting my IPs burned

[Cecilia]

[font=宋体]Last year I had to pull 4M product pages from 28 retailers in 36 hours for a price-intelligence client. On hour six I was already losing half my requests to bot protection.[/font]
[font=宋体]The classic stack didn't survive contact: Scrapy, datacenter proxies, max concurrency. TLS fingerprinting got us first, then Cloudflare challenges, then the IPs got nuked. I burned two days throwing more concurrency at it — made everything worse.[/font]
[font=宋体]What actually worked was boring. I dropped to 25 concurrent requests and ramped up slowly while watching error rates — the sensitive sites stayed slow, the easy ones got more threads. For the hardest sites I let a real Playwright browser solve the initial challenge once, then reused the session cookies with a fast HTTP client. And I stopped rotating IPs per request — that screams bot.[/font]
[font=宋体] I moved to Helodata's residential pool with sticky sessions, holding the same exit IP 10-30 minutes for the checkout-style flows, and the 195+ country coverage meant I didn't need a second provider for the APAC leg.[/font]
[font=宋体]End result: 4M pages in 41 hours, 99.2% success rate,. The client's pricing model paid for the whole project in their first deal.[/font]
[font=宋体]What's your rotation strategy for high-volume scrapes in 2026? Sticky, rotating, or hybrid?[/font]
[font=宋体] [/font]
Since a few people asked — the proxy provider is  https://helodata.com?ref=14sb6s [font=宋体]    [/font]. Sticky sessions are the key feature for me