NEWOfficial SDKs Now Available — Python, JavaScript & Go SDKs with full API coverage
Back to blog
scrapingai

Web Scraping Best Practices: Lessons From Processing 10M Pages

By the JIRO Engineering Team at Blackvault Technology

the JIRO Engineering Team at Blackvault Technology11 min read

Web Scraping Best Practices: Lessons From Processing 10M Pages

By the JIRO Engineering Team at Blackvault Technology

Indore, MP, India — December 2024

Web scraping is part art, part science. After processing 10 million+ pages and handling millions of requests per month, we have learned what works and what does not.

This is our comprehensive guide to web scraping best practices, based on real-world experience.

1. Respect robots.txt

robots.txt is not just a suggestion — it is a contract. Sites that block you in robots.txt will actively hunt you if you ignore it.

What robots.txt Tells You

The robots.txt file is located at the root of every website (e.g., https://example.com/robots.txt). It tells you:

  • Which paths the site allows or disallows
  • Which user agents the rules apply to
  • Optional crawl-delay directives
  • Sitemap locations

What to Do

  • Always check robots.txt before scraping
  • Respect Disallow rules — if a site says "do not scrape /admin", do not scrape /admin
  • Set a reasonable crawl delay — 1-2 seconds between requests is usually safe
  • Identify your bot — use a descriptive user agent with contact info

What NOT to Do

  • Ignore robots.txt
  • Scrape at high speed
  • Hide your identity
  • Scrape copyrighted content

robots.txt Parsing Code

import urllib.robotparser

def can_scrape(url, user_agent="JIRO-Bot"):
    """Check if we can scrape a URL based on robots.txt."""
    parsed = urllib.parse.urlparse(url)
    robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
    
    rp = urllib.robotparser.RobotFileParser()
    rp.set_url(robots_url)
    rp.read()
    
    return rp.can_fetch(user_agent, url)

# Usage
if can_scrape("https://example.com/page"):
    scrape_page("https://example.com/page")
else:
    print("Skipping: disallowed by robots.txt")

2. Rate Limiting

The #1 cause of being blocked is not rate limiting.

Best Practices

  • 1-2 requests/second per domain is usually safe
  • Exponential backoff on 429 (Too Many Requests) responses
  • Respect Retry-After headers — if a site says "wait 60 seconds", wait 60 seconds
  • Throttle during peak hours — sites are more sensitive during business hours

Rate Limiting by Site Type

Site Type Recommended Rate Notes
News sites 1 req/2s High sensitivity
E-commerce 1 req/s Moderate sensitivity
Social media 1 req/3s Very high sensitivity
Government 1 req/5s Legal risk
Blogs/Wikis 2 req/s Low sensitivity
APIs (if available) Per API docs Use official API when possible

JIRO's Approach

JIRO implements 4-tier rate limiting:

  1. Per-IP: 100 requests/minute
  2. Per-user: 60-1000 requests/minute (based on plan)
  3. Per-engine: 320 requests/minute
  4. Global: 10,000 requests/minute

All enforced with atomic Lua scripts for race-condition-proof concurrency.

Rate Limiter Implementation

import redis
import time

class TokenBucketRateLimiter:
    def __init__(self, redis_client, key, rate, capacity):
        self.redis = redis_client
        self.key = key
        self.rate = rate  # tokens per second
        self.capacity = capacity  # max tokens
    
    def consume(self, tokens=1):
        """Try to consume tokens. Returns True if allowed, False if rate limited."""
        now = time.time()
        pipe = self.redis.pipeline()
        
        # Get current token count and last refill time
        pipe.hgetnx(self.key, "tokens")
        pipe.hgetnx(self.key, "last_refill")
        tokens_str, last_refill_str = pipe.execute()
        
        tokens = float(tokens_str) if tokens_str else self.capacity
        last_refill = float(last_refill_str) if last_refill_str else now
        
        # Refill tokens based on elapsed time
        elapsed = now - last_refill
        new_tokens = min(self.capacity, tokens + elapsed * self.rate)
        
        if new_tokens >= tokens:
            # Consume tokens
            new_tokens -= tokens
            pipe.hset(self.key, mapping={
                "tokens": new_tokens,
                "last_refill": now
            })
            pipe.expire(self.key, 60)
            pipe.execute()
            return True
        return False

3. User Agent Rotation

Using the same user agent for every request is a red flag. Rotate through a pool of realistic browser user agents.

Best Practices

  • Use real browser user agents — Chrome, Firefox, Safari
  • Rotate regularly — every 10-50 requests
  • Match user agent to request — if you are scraping a mobile site, use a mobile user agent
  • Do not use headless browser signatures — they are easily detected

User Agent Pool

USER_AGENTS = [
    # Chrome on Windows
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    # Chrome on Mac
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    # Chrome on Linux
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    # Firefox on Windows
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
    # Safari on Mac
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15",
    # Edge on Windows
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0",
]

import random

def get_random_user_agent():
    return random.choice(USER_AGENTS)

4. Proxy Rotation

If you are scraping at scale, you need proxies. Datacenter proxies are faster but easier to detect. Residential proxies are slower but look like real users.

Best Practices

  • Rotate proxies every 10-50 requests
  • Use geographically appropriate proxies — scrape Amazon.de from German IPs
  • Monitor proxy health — remove dead proxies automatically
  • Have a fallback — if no proxy is available, fall back to direct connection (with caution)

Proxy Types Comparison

Type Speed Cost Detection Risk Best For
Datacenter Fast Low High Search engines
Residential Slow High Low Social media
Mobile Medium Very High Very Low Mobile sites
Rotating Varies Medium Low General purpose

JIRO's Approach

JIRO supports:

  • Rotating proxies: Automatically rotate through a pool
  • Proxy authentication: Username/password or IP whitelist
  • Health checks: Automatically remove dead proxies
  • Fallback: Direct connection if proxies fail

5. Handle Failures Gracefully

Things will fail. Servers go down, networks hiccup, parsers break. Your scraper needs to handle this.

Best Practices

  • Retry with exponential backoff: 3-5 retries, doubling the wait time each time
  • Classify errors: Transient (retry) vs permanent (skip)
  • Return cached results: If all engines fail, return what you have
  • Log everything: You cannot fix what you cannot see

Error Classification Table

Error Type Action
429 Too Many Requests Transient Retry with backoff
503 Service Unavailable Transient Retry with backoff
403 Forbidden Permanent Skip, alert
404 Not Found Permanent Skip
Timeout Transient Retry with backoff
Connection Error Transient Retry with backoff
DNS Error Permanent Skip, alert

Retry Implementation

import asyncio
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception_type

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=4, max=10),
    retry=retry_if_exception_type((TimeoutError, ConnectionError))
)
async def fetch_with_retry(url, **kwargs):
    async with aiohttp.ClientSession() as session:
        async with session.get(url, **kwargs) as response:
            response.raise_for_status()
            return await response.text()

6. Parse Robustly

HTML is often malformed. Your parser needs to handle broken markup gracefully.

Best Practices

  • Use tolerant parsers: Beautiful Soup, lxml (not strict XML parsers)
  • Validate extracted data: Check that required fields are present
  • Handle missing elements: Use fallbacks or default values
  • Clean extracted text: Remove extra whitespace, decode HTML entities

Content Extraction Code

from bs4 import BeautifulSoup
import re

def extract_content(html):
    soup = BeautifulSoup(html, "lxml")
    
    # Remove unwanted elements
    for tag in soup(["script", "style", "nav", "header", "footer", "aside"]):
        tag.decompose()
    
    # Get text
    text = soup.get_text(separator="\n")
    
    # Clean up
    lines = [line.strip() for line in text.split("\n") if line.strip()]
    text = "\n".join(lines)
    
    # Remove extra whitespace
    text = re.sub(r"\n{3,}", "\n\n", text)
    
    return text

JIRO's Approach

JIRO uses:

  • Beautiful Soup for HTML parsing
  • lxml for speed-critical paths
  • Tolerant parsing: Gracefully handles malformed HTML
  • Data validation: Pydantic models ensure correct data types

7. Cache Aggressively

Every request that hits the origin server costs money (time, bandwidth, reputation). Caching is the cheapest way to scale.

Best Practices

  • Cache with TTL: 1 hour for search, 24 hours for social
  • Cache invalidation: Bypass cache for fresh data requests
  • Cache compression: Compress cached data to save memory
  • Cache deduplication: Do not cache duplicate queries

JIRO's Approach

JIRO caches:

  • Search results: 1-hour TTL
  • Scraped pages: 24-hour TTL
  • AI responses: 1-hour TTL
  • Social posts: 1-hour TTL

Cached results are 10x cheaper than live results, so users naturally prefer caching.

Cache Implementation

import redis
import json
import hashlib

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

def get_cache_key(url, params=None):
    """Generate a cache key from URL and parameters."""
    key = f"scrape:{url}"
    if params:
        key += f":{hashlib.md5(json.dumps(params, sort_keys=True).encode()).hexdigest()}"
    return key

def get_cached(url, ttl=3600, params=None):
    key = get_cache_key(url, params)
    cached = r.get(key)
    if cached:
        return json.loads(cached)
    return None

def set_cached(url, data, ttl=3600, params=None):
    key = get_cache_key(url, params)
    r.setex(key, ttl, json.dumps(data))

8. Circuit Breakers

If a service fails repeatedly, stop calling it. Give it a cooldown period, then try again.

Best Practices

  • Threshold: 5 consecutive failures trips the circuit
  • Cooldown: 60 seconds before retrying
  • State machine: CLOSED (normal), OPEN (blocked), HALF_OPEN (probing)
  • Shared state: All instances see the same circuit state

JIRO's Approach

JIRO implements per-engine circuit breakers shared across all instances:

  • If one instance detects Google is down, all instances stop calling Google
  • No wasted resources, no cascading failures

9. Monitor Everything

You cannot improve what you cannot measure. Monitor your scraper's performance.

Key Metrics

  • Success rate: % of requests that succeed
  • Latency: Average response time
  • Error rate: % of requests that fail
  • Cache hit rate: % of requests served from cache
  • Proxy health: % of proxies that are working

JIRO's Metrics Dashboard

Metric Target Actual
Success rate >99% 99.9%
P50 latency <200ms 180ms
P95 latency <500ms 320ms
P99 latency <1000ms 680ms
Cache hit rate >60% 65%
Error rate <0.5% 0.1%

Monitoring Code Example

from prometheus_client import Counter, Histogram, Gauge

# Metrics
requests_total = Counter("scrape_requests_total", "Total scrape requests", ["engine", "status"])
request_duration = Histogram("scrape_request_duration_seconds", "Request duration", ["engine"])
cache_hits = Counter("scrape_cache_hits_total", "Cache hits", ["engine"])
circuit_breaker_state = Gauge("circuit_breaker_state", "Circuit breaker state", ["engine"])

# Usage
@request_duration.labels(engine="google").time()
def scrape_google(query):
    try:
        result = do_scrape(query)
        requests_total.labels(engine="google", status="success").inc()
        return result
    except Exception as e:
        requests_total.labels(engine="google", status="error").inc()
        raise

Web scraping sits in a legal gray area. Here is what you need to know:

Jurisdiction Key Law Scraping Status
United States CFAA Generally legal for public data
European Union GDPR + ePrivacy Regulated, consent may be required
United Kingdom Data Protection Act Similar to GDPR
Australia Privacy Act Regulated for personal data
Canada PIPEDA Regulated for personal data
India DPDP Act Emerging regulation
  • Scraping publicly available data
  • Scraping for research purposes
  • Scraping non-personal data
  • Scraping data that is not behind a login

What Is Generally Illegal

  • Scraping copyrighted content without permission
  • Scraping personal data without consent (GDPR)
  • Scraping paywalled content
  • Scraping at a rate that overwhelms the server (CFAA)
  • Scraping data in violation of ToS (enforceability varies)

JIRO's Approach

JIRO implements:

  • robots.txt compliance
  • Rate limiting to prevent server overload
  • Audit logging for accountability
  • User consent for sensitive data
  • Data minimization (only extract what is needed)

11. Data Quality

Scraped data is often messy. Here is how to ensure quality:

Data Quality Checklist

  • Validation: Check required fields are present
  • Type checking: Ensure correct data types (int, string, date)
  • Deduplication: Remove duplicate results
  • Normalization: Standardize dates, numbers, addresses
  • Enrichment: Add metadata (source, timestamp, confidence)
  • Cleaning: Remove HTML entities, extra whitespace, boilerplate

Data Quality Pipeline

from pydantic import BaseModel, validator
from datetime import datetime

class SearchResult(BaseModel):
    title: str
    url: str
    snippet: str
    source: str
    timestamp: datetime
    confidence: float
    
    @validator("title")
    def title_not_empty(cls, v):
        if not v or len(v.strip()) < 3:
            raise ValueError("Title must not be empty")
        return v.strip()
    
    @validator("url")
    def valid_url(cls, v):
        if not v.startswith(("http://", "https://")):
            raise ValueError("URL must be valid")
        return v
    
    @validator("confidence")
    def confidence_range(cls, v):
        if not 0 <= v <= 1:
            raise ValueError("Confidence must be between 0 and 1")
        return v

# Validate results
results = [SearchResult(**r) for r in raw_results]

Conclusion

Web scraping is powerful, but it comes with responsibilities. Follow these best practices, respect the sites you scrape, and build systems that are robust and reliable.

Try JIRO Today

JIRO implements all these best practices out of the box:

curl -fsSL https://raw.githubusercontent.com/DevAnimecx/jiro-cloud/master/deploy_vps.sh | bash -s -- your-domain.com

Free tier: 1,000 credits/month
Cloud: https://searchjiro.vercel.app

About Blackvault Technology

JIRO is developed by the jirosearch team at Blackvault Technology, led by Adarsh Kushwah (CEO & Founder).

This article was written by the JIRO Engineering Team at Blackvault Technology. JIRO is an open-source search and scraping platform developed in Indore, MP, India.

AK

Adarsh Kushwah

CEO & Founder of Blackvault Technology. Building open-source developer tools and AI infrastructure in Indore, MP, India.

GitHubLinkedIn

Stay ahead of the curve

Get weekly insights on search technology, web scraping, and open-source infrastructure. No spam, unsubscribe anytime.

Join 2,000+ developers. Free forever.