And those leaks compound. A crawler that looks cheap at 50,000 pages a day can become the biggest line item in an engineering budget at 5 million.
The fixes below target the places where spend actually piles up: the proxy layer, the bytes on the wire, the crawl schedule and the data sitting in storage long after anyone has looked at it.
Price Proxies by Successful Request
Proxy pricing pages advertise the wrong number. Cost per IP or per gigabyte says very little; the figure that matters is cost per successful request, which absorbs every block, CAPTCHA and retry along the way. A pool that only succeeds half the time doubles the real price of every record it returns.
Datacenter proxies look cheapest on paper. But many retail and travel sites flag hosting-company IP ranges on sight.
Residential IPs pass more checks, although per-GB billing punishes heavy pages. ISP proxies sit in the middle: they're hosted on data center hardware yet registered to consumer internet providers, and they're often sold per IP with unmetered traffic.
That pricing model changes the math for jobs that hit the same sites every day. Finding an isp proxy cheap enough to run around the clock usually beats paying bandwidth rates on residential traffic, and static IPs keep sessions stable on sources that require a login.
Stop Paying for Bytes Nobody Parses
The median mobile home page weighed about 2.6 MB in 2025, according to HTTP Archive's Web Almanac, with images accounting for roughly 900 KB. A price monitor needs a few kilobytes of that. Why pay to move the rest?
Headless browsers download everything unless told otherwise. Playwright's network interception API can abort image, font and media requests before they ever hit the proxy, which trims transfer volume on every page load. Blocking third-party analytics and ad scripts helps too, since they add requests without adding a single useful field.
Better still, skip the browser entirely. Plenty of sites load product data from a JSON endpoint that a plain HTTP client (Python's httpx, for example) can call directly, at a fraction of the CPU and bandwidth.
Some anti-bot systems notice when a "browser" never loads images, so test block lists against each target before rolling them out.
Recrawl Only What Changed
Not every page deserves a daily visit. Prices on a large retailer can move several times a day while specification pages sit untouched for months, so recrawl frequency should follow how often each page type actually changes.
Conditional requests help as well, although server-side support is thin. Google's crawling team reported in a post on HTTP caching that only 0.017% of its fetches were cacheable, down from 0.026% a decade earlier. Sending ETag and If-Modified-Since headers still costs nothing, and every 304 response is a page that didn't need downloading again.
Speed is its own trap. Hammering a domain triggers rate limits, and every ban burns IPs that cost money to acquire. Exponential backoff after a 429 response is slower on paper and cheaper in practice.
Treat Bad Data as a Line Item
Collection is only half the bill. In a Harvard Business Review article, data quality consultant Thomas Redman cited IBM's estimate that bad data cost the US $3.1 trillion in 2016, largely through people correcting errors by hand.
Scraped data is especially prone to this. Validate records at ingestion (a price of $0.00 or a missing currency code should fail loudly) and deduplicate before anything reaches the warehouse. A duplicate row costs almost nothing to drop at the door and far more to untangle in a quarterly report.
Storage format matters, too. Compressed Apache Parquet files are typically several times smaller than raw JSON, and older snapshots can move to Amazon S3 Glacier storage classes that cost a small fraction of S3 Standard.
Where Web Data Costs Are Heading
Clean access to public web data will keep getting pricier. Cloudflare began blocking AI crawlers by default for new domains in 2025, and anti-bot vendors keep refining browser fingerprinting, so a successful request in 2027 will probably cost more than one today.
Teams that track cost per usable record will spot those increases early and adjust their stack. The ones still buying capacity by the gigabyte and hoping for the best won't notice until the invoice arrives.