serpapiscrapingaiself-hosted

How to Scrape Twitter/X: Complete Guide to Twitter Data Extraction

By the JIRO Engineering Team at Blackvault Technology

the JIRO Engineering Team at Blackvault TechnologySeptember 11, 20267 min read

How to Scrape Twitter/X: Complete Guide to Twitter Data Extraction

By the JIRO Engineering Team at Blackvault Technology

Indore, MP, India — December 2024

Twitter (now X) is one of the most important sources of real-time public data. With 500M+ daily tweets and 250M+ daily active users, it is essential for brand monitoring, sentiment analysis, and trend tracking.

But scraping Twitter is hard. They have aggressive anti-scraping measures, require authentication, and frequently change their HTML structure.

In this guide, we will show you how to scrape Twitter effectively, what data you can extract, and how JIRO makes it possible.

What You Can Scrape from Twitter

Tweets

Text, media (images, videos, GIFs)
Author, username, verified status
Like count, retweet count, reply count, quote count
View count, bookmark count
Created timestamp
Hashtags, mentions, URLs
Poll options and votes
Language, place, sensitivity flags
Conversation ID, in_reply_to_status_id

Profiles

Display name, username, bio
Avatar, banner image
Follower count, following count
Tweet count, listed count
Verified, protected, suspended status
Joined date, location, website
Description, pinned tweet

Search Results

Tweets matching a query
Recent or top tweets
Filters (verified, media, etc.)

Trends

Trending topics
Tweet volume
Location-specific trends

Twitter's Anti-Scraping Measures

Twitter is one of the hardest sites to scrape. Here is what you will encounter:

1. Authentication Required

Twitter requires a logged-in account to view most content. This means:

OAuth or cookies needed for access
Login detection — frequent logins trigger security checks
Account bans — scraping can result in permanent bans

2. Rate Limiting

Twitter limits requests based on:

Authentication status: 300 requests/15 min (unauthenticated), 900/15 min (authenticated)
Endpoint: Search has different limits than profiles
Account age: New accounts have stricter limits

3. CAPTCHA

Twitter uses CAPTCHA challenges (including reCAPTCHA) to detect bots. Solving CAPTCHAs requires a third-party service.

4. IP Bans

Repeated violations result in IP bans. Once banned, you need a new IP (proxy) to continue.

5. Dynamic Content

Twitter uses heavy JavaScript rendering. Simple HTTP requests will not work — you need a headless browser or API reverse-engineering.

6. GraphQL API Complexity

Twitter's current frontend uses GraphQL with obfuscated query IDs. These IDs change frequently, requiring constant updates to the scraper.

Legal Considerations

Twitter's Terms of Service

Twitter's terms of service explicitly prohibit automated access without permission. Scraping Twitter data can result in:

Account suspension or ban
Legal action (CFAA in the US, similar laws elsewhere)
IP blocks

Is Scraping Twitter Legal?

The legality of scraping Twitter depends on jurisdiction:

Jurisdiction	Status	Notes
United States	Generally legal for public data	CFAA applies to unauthorized access
European Union	Regulated	GDPR requires consent for personal data
United Kingdom	Regulated	Data Protection Act applies
India	Emerging regulation	DPDP Act may apply

Recommendations

Use the official API whenever possible
Respect rate limits — do not hammer the servers
Cache results — reduce requests
Do not redistribute data — keep scraped data private
Use proxies — rotate IPs to avoid bans

Alternatives to Twitter Scraping

1. Official API (Recommended)

Twitter's official API (X API) is the best option:

Legal and supported
Rich data access
Stable API
Free tier available

Limitations:

Rate limits (500K tweets/month on free tier)
Requires application approval for elevated access
Costs can add up at scale

2. Third-Party Services

Services like Apify, Bright Data, and ScrapingBee offer Twitter scraping:

Legal (they handle compliance)
Reliable (they maintain the scrapers)
Expensive ($50-500/month)

3. Self-Hosted (JIRO)

JIRO offers self-hosted Twitter scraping:

Free and open-source
No vendor lock-in
Requires maintenance (you need to keep up with Twitter's changes)

Using JIRO to Scrape Twitter

JIRO handles the complexity of Twitter scraping. No need to manage authentication, proxies, or parsing — JIRO does it all.

Python SDK

from jiro import Jiro

client = Jiro(api_key="YOUR_API_KEY")

# Search tweets
tweets = client.search("OpenAI GPT-4", engine="twitter", limit=10)
for tweet in tweets:
    print(f"@{tweet['username']}: {tweet['text'][:100]}...")
    print(f"  Likes: {tweet['likes']}, Retweets: {tweet['retweets']}")
    print()

# Get user tweets
tweets = client.twitter_user("elonmusk", limit=10)

# Get user profile
profile = client.twitter_profile("elonmusk")
print(f"Name: {profile['name']}")
print(f"Followers: {profile['followers']}")
print(f"Bio: {profile['description']}")

REST API

# Search tweets
curl "http://localhost:8000/v1/twitter/search?q=OpenAI+GPT-4&limit=10" \
  -H "Authorization: Bearer YOUR_API_KEY"

# Get user tweets
curl "http://localhost:8000/v1/twitter/user?username=elonmusk&limit=10" \
  -H "Authorization: Bearer YOUR_API_KEY"

# Get user profile
curl "http://localhost:8000/v1/twitter/profile?username=elonmusk" \
  -H "Authorization: Bearer YOUR_API_KEY"

MCP Server (Claude Desktop)

{
  "mcpServers": {
    "jiro": {
      "command": "python",
      "args": ["-m", "jiro.mcp_http"],
      "env": {
        "JIRO_API_KEY": "jsk_live_..."
      }
    }
  }
}

Now Claude can search Twitter autonomously:

User: "What are people saying about the new iPhone?"

Claude: [Uses JIRO to search Twitter, reads top tweets, synthesizes answer]

Twitter Data Extraction Examples

Extracting Tweet Data

def extract_tweet_data(tweet):
    return {
        "id": tweet.get("id_str") or tweet.get("id"),
        "text": tweet.get("full_text") or tweet.get("text", ""),
        "author_id": tweet.get("user", {}).get("id_str"),
        "username": tweet.get("user", {}).get("screen_name"),
        "display_name": tweet.get("user", {}).get("name"),
        "created_at": tweet.get("created_at"),
        "likes": tweet.get("favorite_count"),
        "retweets": tweet.get("retweet_count"),
        "replies": tweet.get("reply_count"),
        "quotes": tweet.get("quote_count"),
        "views": tweet.get("view_count"),
        "hashtags": [h["text"] for h in tweet.get("entities", {}).get("hashtags", [])],
        "mentions": [m["screen_name"] for m in tweet.get("entities", {}).get("user_mentions", [])],
        "urls": [u["expanded_url"] for u in tweet.get("entities", {}).get("urls", [])],
        "is_retweet": "retweeted_status" in tweet,
        "is_pinned": tweet.get("pinned", False),
    }

Extracting User Profile Data

def extract_user_data(user):
    return {
        "id": user.get("id_str") or user.get("id"),
        "username": user.get("screen_name"),
        "display_name": user.get("name"),
        "bio": user.get("description"),
        "location": user.get("location"),
        "website": user.get("url"),
        "followers": user.get("followers_count"),
        "following": user.get("friends_count"),
        "tweets": user.get("statuses_count"),
        "listed": user.get("listed_count"),
        "verified": user.get("verified"),
        "protected": user.get("protected"),
        "created_at": user.get("created_at"),
        "avatar": user.get("profile_image_url_https"),
        "banner": user.get("profile_banner_url"),
    }

Use Cases

1. Brand Monitoring

Monitor mentions of your brand:

tweets = client.search("JIRO search", engine="twitter", limit=50)
for tweet in tweets:
    print(f"@{tweet['username']}: {tweet['text'][:100]}...")
    print(f"  Likes: {tweet['likes']}, Date: {tweet['created_at']}")
    # Analyze sentiment, identify influencers

2. Trend Tracking

Track trending topics:

tweets = client.search("AI agents", engine="twitter", limit=100)
for tweet in tweets:
    print(f"@{tweet['username']}: {tweet['text'][:100]}...")
    print(f"  Likes: {tweet['likes']}, Retweets: {tweet['retweets']}")
    # Identify key influencers and discussions

3. Customer Support

Monitor customer complaints:

tweets = client.search("JIRO broken", engine="twitter", limit=50)
for tweet in tweets:
    print(f"@{tweet['username']}: {tweet['text'][:100]}...")
    print(f"  Likes: {tweet['likes']}, Date: {tweet['created_at']}")
    # Identify issues and respond

4. Competitive Intelligence

Monitor competitor mentions:

tweets = client.search("SerpAPI", engine="twitter", limit=50)
for tweet in tweets:
    print(f"@{tweet['username']}: {tweet['text'][:100]}...")
    print(f"  Likes: {tweet['likes']}, Date: {tweet['created_at']}")
    # Identify pain points and opportunities

5. Sentiment Analysis

Analyze sentiment around a topic:

from textblob import TextBlob

def analyze_twitter_sentiment(tweets):
    sentiments = []
    for tweet in tweets:
        blob = TextBlob(tweet["text"])
        polarity = blob.sentiment.polarity
        
        if polarity > 0.1:
            sentiment = "positive"
        elif polarity < -0.1:
            sentiment = "negative"
        else:
            sentiment = "neutral"
        
        sentiments.append({
            "text": tweet["text"][:100],
            "sentiment": sentiment,
            "polarity": polarity,
            "likes": tweet["likes"],
        })
    
    return sentiments

sentiments = analyze_twitter_sentiment(tweets)
for s in sentiments[:10]:
    print(f"{s['sentiment'].upper()}: {s['text']}")
    print(f"  Polarity: {s['polarity']:.2f}, Likes: {s['likes']}")

6. Influencer Identification

Find influential users in a niche:

tweets = client.search("machine learning", engine="twitter", limit=100)
user_engagement = {}
for tweet in tweets:
    username = tweet["username"]
    engagement = tweet["likes"] + tweet["retweets"] * 2 + tweet["replies"]
    user_engagement[username] = user_engagement.get(username, 0) + engagement

top_influencers = sorted(user_engagement.items(), key=lambda x: x[1], reverse=True)[:10]
for user, engagement in top_influencers:
    print(f"@{user}: {engagement} total engagement")

Twitter Scraping Ethics

What Is Acceptable

Scraping public tweets for research
Rate-limiting your requests
Caching results to reduce load
Identifying your bot in the user agent
Not redistributing data commercially

What Is Unacceptable

Scraping private tweets or DMs
Scraping at a rate that overwhelms servers
Redistributing data commercially without permission
Impersonating a human user
Ignoring rate limits and CAPTCHAs

Try JIRO Today

JIRO makes Twitter scraping easier:

curl -fsSL https://raw.githubusercontent.com/DevAnimecx/jiro-cloud/master/deploy_vps.sh | bash -s -- your-domain.com

Free tier: 1,000 credits/month
Cloud: https://searchjiro.vercel.app

About Blackvault Technology

JIRO is developed by the jirosearch team at Blackvault Technology, led by Adarsh Kushwah (CEO & Founder).

Location: Indore, MP, India
GitHub: @blackvault-technology
LinkedIn: Blackvault Technology

This article was written by the JIRO Engineering Team at Blackvault Technology. JIRO is an open-source search and scraping platform developed in Indore, MP, India.

Adarsh Kushwah

CEO & Founder of Blackvault Technology. Building open-source developer tools and AI infrastructure in Indore, MP, India.

GitHub LinkedIn

Stay ahead of the curve

Get weekly insights on search technology, web scraping, and open-source infrastructure. No spam, unsubscribe anytime.

Join 2,000+ developers. Free forever.