Scraping at Scale: Lessons From Processing 10M Pages

By the JIRO Engineering Team at Blackvault Technology

Indore, MP, India — December 2024

---

We have been scraping the web at scale for over a year now. In that time, we have processed 10 million+ pages, handled millions of requests per month, and learned some hard lessons about web scraping.

This is our honest account of what works, what does not, and what we wish we knew when we started.

Our Infrastructure at Scale

Before diving into lessons, it helps to understand our infrastructure. We process over 2 million requests per month across 9 search engines and 12 social platforms.

Architecture Overview

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Client    │───▶│   FastAPI   │───▶│    Redis    │
│  Requests   │    │   Gateway   │    │   Cache +   │
└─────────────┘    └──────┬──────┘    │  Rate Limit │
                         │           └──────┬──────┘
              ┌──────────▼──────────┐       │
              │   Engine Workers    │       │
              │  (async aiohttp)    │       │
              └──────────┬──────────┘       │
                         │                  │
              ┌──────────▼──────────┐       │
              │    Parsers          │       │
              │  (Beautiful Soup)   │       │
              └──────────┬──────────┘       │
                         │                  │
              ┌──────────▼──────────┐       │
              │   LLM Service       │       │
              │ (OpenAI/Claude)     │       │
              └─────────────────────┘       │
                                           │
                              ┌────────────▼────────────┐
                              │     PostgreSQL /        │
                              │   SQLite (fallback)     │
                              └─────────────────────────┘

Infrastructure Costs

Component	Specification	Monthly Cost
VPS	8 CPU, 32GB RAM, 500GB SSD	$40
Redis	In-memory cache	Included
Database	PostgreSQL managed	$15
Bandwidth	5TB transfer	$10
Proxies	Rotating residential	$50
LLM API	GPT-4o (pay-per-use)	$30
Total		$145/month

For comparison, an equivalent SerpAPI setup at our volume would cost $2,500+/month.

Lesson 1: Rate Limiting is Non-Negotiable

The #1 mistake new scrapers make: not rate limiting.

Early on, we would fire off 100 concurrent requests to a site, get blocked, and wonder why. We learned the hard way that websites have rate limits, and they are not happy when you exceed them.

What We Learned

• Respect robots.txt: It is not just polite, it is smart. Sites that block you in robots.txt will actively hunt you.
• Throttle your requests: 1-2 requests per second per domain is usually safe. More than that, and you will get blocked.
• Use exponential backoff: If you get a 429 (Too Many Requests), wait 5 seconds, then 10, then 20. Do not hammer the server.
• Rotate user agents: Do not use the same user agent for every request. Use a pool of realistic browser user agents.

How JIRO Does It

JIRO implements 4-tier rate limiting:

1. Per-IP: 100 requests/minute
2. Per-user: 60 requests/minute (FREE), 120 (MICRO), 300 (DEVELOPER), 1000 (BUSINESS)
3. Per-engine: 320 requests/minute
4. Global: 10,000 requests/minute

All rate limits are enforced with atomic Lua scripts in Redis, so they are race-condition-proof even under heavy concurrency.

Rate Limiting Code Example

import redis
import time

r = redis.Redis(host='localhost', port=6379, decode_responses=True)

def rate_limit(key, limit, window):
    """Sliding window rate limiter using Redis."""
    now = time.time()
    pipe = r.pipeline()
    pipe.zadd(key, {now: now})
    pipe.zremrangebyscore(key, 0, now - window)
    pipe.zcard(key)
    pipe.expire(key, window)
    results = pipe.execute()
    return results[2] <= limit

# Usage
if rate_limit("user:123:requests", 60, 60):
    process_request()
else:
    raise HTTPException(status_code=429, detail="Rate limit exceeded")

Lesson 2: Circuit Breakers Save You

What happens when a search engine goes down? If you do not have circuit breakers, your entire system grinds to a halt.

We learned this when Google started returning 503 errors for 2 hours. Our system kept retrying, wasting resources and slowing down everything else.

What We Learned

• Circuit breakers are essential: If a service fails 5 times in a row, stop calling it. Give it a cooldown period (60 seconds), then try again.
• Fallback engines: Always have a backup. If Google fails, fall back to Bing or DuckDuckGo.
• Graceful degradation: If all engines fail, return cached results or a helpful error message.

How JIRO Does It

JIRO implements per-engine circuit breakers shared across all instances:

• Threshold: 5 consecutive failures trips the circuit
• Cooldown: 60 seconds before retrying
• State: CLOSED (normal), OPEN (blocked), HALF_OPEN (probing)
• Shared: All instances see the same state (via Redis)

This means if one instance detects that Google is down, all instances stop calling Google. No wasted resources, no cascading failures.

Circuit Breaker Implementation

class CircuitBreaker:
    def __init__(self, failure_threshold=5, cooldown=60):
        self.failure_threshold = failure_threshold
        self.cooldown = cooldown
        self.failures = 0
        self.state = "CLOSED"  # CLOSED, OPEN, HALF_OPEN
        self.last_failure_time = 0
    
    def call(self, func, *args, **kwargs):
        if self.state == "OPEN":
            if time.time() - self.last_failure_time > self.cooldown:
                self.state = "HALF_OPEN"
            else:
                raise Exception("Circuit breaker is OPEN")
        
        try:
            result = func(*args, **kwargs)
            self.on_success()
            return result
        except Exception as e:
            self.on_failure()
            raise e
    
    def on_success(self):
        self.failures = 0
        self.state = "CLOSED"
    
    def on_failure(self):
        self.failures += 1
        self.last_failure_time = time.time()
        if self.failures >= self.failure_threshold:
            self.state = "OPEN"

Lesson 3: Caching is Your Best Friend

Every request that hits the origin server costs money (time, bandwidth, reputation). Caching is the cheapest way to scale.

What We Learned

• Cache aggressively: If a query has been searched in the last hour, return the cached result.
• Cache with TTL: Set a time-to-live (1 hour for search, 24 hours for social).
• Cache invalidation: When a user explicitly requests fresh data, bypass the cache.
• Cache compression: Compress cached data to save Redis memory.

How JIRO Does It

JIRO caches:

• Search results: 1-hour TTL, 1 credit (vs 3 credits for live)
• Scraped pages: 24-hour TTL
• AI responses: 1-hour TTL
• Social posts: 1-hour TTL

Cached results are 10x cheaper than live results, so users naturally prefer caching. It is a win-win: cheaper for users, less load for us.

Cache Strategy

Request arrives
    │
    ▼
Check cache (Redis)
    │
    ├── HIT → Return cached result (1 credit)
    │
    └── MISS → Fetch from source (3 credits)
                 │
                 ▼
           Store in cache (1-hour TTL)
                 │
                 ▼
           Return result

Lesson 4: Proxies Are a Necessary Evil

If you are scraping at scale, you need proxies. Period.

What We Learned

• Residential proxies: Best for social media scraping. Look like real users.
• Datacenter proxies: Faster, cheaper, but easier to detect. Good for search engines.
• Rotate proxies: Do not use the same proxy for every request. Rotate every 10-50 requests.
• Monitor proxy health: Dead proxies waste your money and slow you down.

How JIRO Does It

JIRO supports:

• Rotating proxies: Automatically rotate through a pool of proxies
• Proxy authentication: Username/password or IP whitelist
• Proxy health checks: Automatically remove dead proxies
• Fallback: If no proxy is available, fall back to direct connection (with caution)

Proxy Configuration

# Rotating residential proxies
proxies = {
    "http": "http://user:pass@residential.provider.com:8080",
    "https": "http://user:pass@residential.provider.com:8080",
}

# JIRO proxy pool configuration
{
    "proxy_pool": [
        {"host": "residential1.provider.com", "port": 8080, "auth": "user:pass", "type": "residential"},
        {"host": "residential2.provider.com", "port": 8080, "auth": "user:pass", "type": "residential"},
        {"host": "datacenter1.provider.com", "port": 8080, "auth": "user:pass", "type": "datacenter"},
    ],
    "rotation_strategy": "round_robin",  # or "random", "least_used"
    "health_check_interval": 300,  # 5 minutes
    "max_consecutive_failures": 3,
}

Lesson 5: Error Handling is Everything

Things will fail. Servers go down, networks hiccup, parsers break. Your system needs to handle this gracefully.

What We Learned

• Always have a fallback: If the primary engine fails, try the secondary. If that fails, return cached results.
• Do not crash on malformed HTML: Use robust parsers that can handle broken markup.
• Log everything: You cannot fix what you cannot see. Log failures, timeouts, and unexpected responses.
• Alert on anomalies: If error rates spike, get notified immediately.

How JIRO Does It

JIRO implements:

• Fallback engines: If primary fails, try built-in fallback
• Error classification: Distinguish between transient (retry) and permanent (skip) errors
• Audit logging: Every request is logged with user ID, endpoint, cost, and status
• Metrics: Track success rates, latency, and error rates per engine

Error Classification Table

Error	Type	Action	Retry Count
429 Too Many Requests	Transient	Retry with backoff	3
503 Service Unavailable	Transient	Retry with backoff	3
403 Forbidden	Permanent	Skip, alert	0
404 Not Found	Permanent	Skip	0
Timeout	Transient	Retry with backoff	2
Connection Error	Transient	Retry with backoff	2
DNS Error	Permanent	Skip, alert	0
SSRF Detected	Permanent	Block, alert, log	0

Lesson 6: Credits > Subscriptions

When we were building JIRO, we chose a credit-based pricing model instead of subscriptions. It was the best decision we made.

Why Credits Work Better

• No surprise bills: Users only pay for what they use
• Natural scaling: Free tier for testing, credits for production
• No churn: Users do not cancel subscriptions; they just stop buying credits
• Fair pricing: Heavy users pay more, light users pay less

How JIRO Does It

JIRO's credit costs:

• Cached search: 1 credit
• Live search: 3 credits
• Google search: 10 credits
• Scrape: 2 credits
• AI search: 8 credits
• AI synthesis: 12 credits
• Agentic task: 20 credits

Free tier: 1,000 credits/month
Starter: 10,000 credits/month ($9)
Pro: 100,000 credits/month ($49)
Business: 1,000,000 credits/month ($199)

Lesson 7: Open Source is a Force Multiplier

We open-sourced JIRO after 6 months of development. It was the best marketing decision we ever made.

What Happened

• GitHub stars: 500+ in the first month
• Contributions: 15+ pull requests from the community
• Bug reports: Users found bugs we never would have caught
• Documentation: Community members wrote tutorials and guides
• Trust: People trust open-source software more than closed-source SaaS

Why It Works

Open source is not just about code. It is about:

• Transparency: Users can audit what you are doing
• Community: People want to contribute to projects they use
• Trust: No hidden backdoors, no surprise changes
• Longevity: Even if we shut down, the project lives on

Lesson 8: Security Cannot Be an Afterthought

We spent an entire month on security. It was worth it.

What We Fixed

• SSRF protection: Blocking private IPs, localhost, metadata endpoints
• Encryption at rest: AES-256-GCM for API keys
• JWT verification: No insecure fallbacks
• Atomic operations: Lua scripts for rate limiting, circuit breakers, credit deductions
• WAF: SQL injection + XSS pattern blocking
• Audit logging: Every action is logged

The Cost of Security

Security is expensive:

• Time: 1 month of full-time development
• Complexity: More code, more tests, more maintenance
• Performance: Encryption, hashing, and validation add overhead

But the alternative is worse. A security breach destroys trust, invites lawsuits, and kills products.

The Numbers

After 6 months of production use:

Metric	Value
Pages processed	10M+
Monthly requests	2M+
Uptime	99.9%
Average latency	250ms
Cache hit rate	65%
Cost per request	$0.001
Error rate	<0.1%
P95 latency	450ms
P99 latency	800ms

Team Recommendations

Small Team (1-3 engineers)

• Use managed services where possible (cloud databases, managed Redis)
• Automate everything (deploy scripts, monitoring, alerts)
• Focus on core features, not infrastructure

Medium Team (4-10 engineers)

• Dedicate 1 engineer to infrastructure/DevOps
• Implement comprehensive monitoring and alerting
• Invest in CI/CD and automated testing

Large Team (10+ engineers)

• Separate teams for platform, engines, and AI
• Implement distributed tracing (OpenTelemetry)
• Use Kubernetes for orchestration
• Implement SLOs and error budgets

Try JIRO Today

JIRO is 100% free and open-source. Deploy in 2 minutes:

curl -fsSL https://raw.githubusercontent.com/DevAnimecx/jiro-cloud/master/deploy_vps.sh | bash -s -- your-domain.com

Free tier: 1,000 credits/month
Cloud: https://searchjiro.vercel.app

About Blackvault Technology

JIRO is developed by the jirosearch team at Blackvault Technology, led by Adarsh Kushwah (CEO & Founder).

• Location: Indore, MP, India
• GitHub: @blackvault-technology
• LinkedIn: Blackvault Technology
• CEO: Adarsh Kushwah

---

This article was written by the JIRO Engineering Team at Blackvault Technology. JIRO is an open-source search and scraping platform developed in Indore, MP, India.