How to Scrape Twitter (X) With Playwright (2026)

The question I get from people who already picked their tool is narrower than how to scrape X in general: it is how to scrape Twitter with Playwright without the capture coming back empty. Playwright fits X because X renders its timeline in JavaScript and fetches the real tweets from an internal GraphQL API. Playwright runs that JavaScript in a genuine browser and lets you read the JSON the moment it lands, which is the part that survives X’s constant markup changes.

I can show why that matters in one request. A logged-out fetch of https://x.com/nasa returns HTTP 200 and about 607 KB of HTML, and that markup contains zero data-testid="tweet" elements. The tweets do not exist until a browser executes the scripts.

import requests

ua = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36"
r = requests.get("https://x.com/nasa", headers={"User-Agent": ua}, timeout=25)
print(r.status_code)                        # -> 200
print(len(r.text))                          # -> ~607000 characters
print(r.text.count('data-testid="tweet"'))  # -> 0

This guide is the Playwright pattern I run against live X pages: capture the GraphQL responses, page through profiles and search, hold a logged-in session, and dodge the blocks. I tested the code in July 2026 and marked where each step breaks, then I show the single API call I reach for when I do not want to babysit a browser.

Can you scrape Twitter (X) with Playwright?

You can scrape Twitter (X) with Playwright by opening the page in a real browser and intercepting the GraphQL responses X fires for its timeline, which hands you structured JSON without parsing a line of HTML. Playwright drives Chromium, Firefox, or WebKit, executes X’s scripts the way a human browser does, and exposes every network response through a response event you can hook.

Two things make Playwright the cleaner choice over a bare HTTP client for X. It runs the JavaScript that fetches the tweets, so the data actually loads, and its network layer lets you grab X’s own UserTweets and SearchTimeline payloads instead of scraping DOM nodes that X renames on a whim.

The tradeoff is weight and upkeep. A browser is slower and heavier than a plain request, and X changes the operation names and query identifiers on its own schedule, so a Playwright scraper needs occasional patching. The rest of this guide is the setup that survives longest, starting with what you install.

What do you need before you start?

You need Python 3.8+, the Playwright package, and a browser binary before you start scraping. Playwright ships its own browsers, so you install the library and then pull Chromium in a second command.

pip install playwright
playwright install chromium

You also need to decide which resource you are collecting, because the target URL and the GraphQL operation that carries the data differ per object:

• A profile’s tweets load through the UserTweets operation at x.com/<username>.
• A single tweet loads through TweetResultByRestId (also seen as TweetDetail) at x.com/<username>/status/<id>.
• Search and hashtag results load through SearchTimeline at x.com/search?q=<terms>.

Knowing which operation carries your data is the whole game with network interception, because you filter the response stream on that name. The next section captures it.

How do you scrape Twitter (X) with Playwright step by step?

You scrape Twitter (X) with Playwright by opening the target page, listening on the response event for X’s GraphQL operation, and reading response.json() when the matching call arrives. The official Playwright network guide documents the page.on("response", ...) event and the page.expect_response() helper, so you capture the body the instant X returns it.

Here is the pattern I run. It opens a profile, waits for the network to settle, and keeps every UserTweets and UserByScreenName payload:

# pip install playwright && playwright install chromium
from playwright.sync_api import sync_playwright
import json

captured = []

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    def on_response(response):
        if "UserTweets" in response.url or "UserByScreenName" in response.url:
            try:
                captured.append(response.json())
            except Exception:
                pass

    page.on("response", on_response)
    page.goto("https://x.com/nasa", wait_until="networkidle")
    page.wait_for_timeout(4000)
    browser.close()

print(f"captured {len(captured)} GraphQL payloads")
print(json.dumps(captured[:1], indent=2)[:400] if captured else "no payloads")

The win is that UserTweets and UserByScreenName are X’s own operations, and their JSON carries the full tweet objects: the tweet text, created_at, the public metrics, and the author fields. You skip CSS selectors entirely, which is what keeps this from breaking every time X reskins its interface.

To collect more than the first page, trigger X’s own pagination by scrolling and let the response hook keep firing as new UserTweets batches load:

for _ in range(3):
    page.mouse.wheel(0, 3000)
    page.wait_for_timeout(2500)

If you want a ready-made tool instead of your own loop, tweet-harvest is a Node CLI that drives Playwright to scrape X search results into a CSV. It still runs a browser under the hood, so the same session and maintenance caveats apply as to the script above.

Parsing the captured GraphQL JSON

The tweet objects sit a few levels down inside each payload, under the timeline instructions, so I inspect one payload once to find the path and then pull the fields with jmespath. The exact path shifts when X reshuffles its schema, and this reflects the July 2026 UserTweets shape:

import jmespath

expr = jmespath.compile(
    "data.user.result.timeline_v2.timeline.instructions[]."
    "entries[].content.itemContent.tweet_results.result.legacy"
)
for payload in captured:
    for t in expr.search(payload) or []:
        print(t["created_at"], "-", t["favorite_count"], "-", t["full_text"])

Flattening once with jmespath is steadier than chaining dictionary keys, because a missing branch returns None instead of raising, so a partial schema change degrades gracefully rather than crashing the run.

Reading the rendered DOM instead

You can read the rendered DOM instead of the JSON when you only need the visible text and want the simplest possible selector. After the timeline renders, Playwright’s locator pulls the tweet text nodes directly:

page.goto("https://x.com/nasa", wait_until="networkidle")
page.wait_for_timeout(4000)
for el in page.locator('article [data-testid="tweetText"]').all():
    print(el.inner_text())

The DOM route is more fragile than the JSON route, because the data-testid attributes change whenever X ships a markup update, and a headless session often meets a login wall before any article renders. I keep a Python-focused walkthrough of the wider toolset in how to scrape Twitter (X) with Python. Once profiles work, search is the next resource, and it uses a different operation.

How do you scrape X search results and hashtags with Playwright?

You scrape X search results and hashtags with Playwright by loading x.com/search?q=<terms> and filtering the response event for the SearchTimeline operation, which carries the matching tweets as JSON. A hashtag is just a query, so %23nasa in the URL returns the tag’s results the same way a keyword does.

from urllib.parse import quote

results = []

def on_search(response):
    if "SearchTimeline" in response.url:
        try:
            results.append(response.json())
        except Exception:
            pass

page.on("response", on_search)
page.goto(f"https://x.com/search?q={quote('nasa artemis')}&f=live",
          wait_until="networkidle")
page.wait_for_timeout(4000)
for _ in range(3):        # each scroll fires another SearchTimeline call
    page.mouse.wheel(0, 3000)
    page.wait_for_timeout(2500)

Search is walled harder than a public profile, because X gates logged-out queries aggressively, so a SearchTimeline capture on a guest session usually returns little or nothing. This is the resource where a logged-in session and a residential IP stop being optional, which is the session setup covered next.

How do you scrape a logged-in X session with Playwright?

You scrape a logged-in X session with Playwright by saving your session cookies once with storage_state and reusing them on every later run, because logged-out X returns almost no timeline data. A guest visit to a profile or a search hits a login gate, so full timelines, search, and follower lists need an authenticated context.

Playwright’s authentication guide covers saving and loading storage_state as JSON. You log in once by hand, write the state to a file, then launch new contexts from that file:

# One-time: log in manually in the opened window, then save the session.
ctx = browser.new_context()
page = ctx.new_page()
page.goto("https://x.com/login")
page.wait_for_timeout(30000)   # complete the login by hand
ctx.storage_state(path="x_session.json")

# Later runs: reuse the saved cookies, no password step.
ctx = browser.new_context(storage_state="x_session.json")

Reusing a saved state avoids typing a password on every run, which is exactly the behavior that trips X’s login defenses. Every call in that context is tied to a real X account, so a dedicated throwaway account rather than your personal one is standard practice, because the account carries the suspension risk. Even with a valid session, a datacenter IP and a headless fingerprint can still get the session challenged, which is the next thing to handle.

How do you scrape Twitter with Playwright without getting blocked?

You scrape Twitter with Playwright without getting blocked by hiding the automation fingerprint, routing through residential proxies, and pacing requests like a human. X scores the connection and the browser before it serves data, so a default headless Chromium on a datacenter IP is the profile it refuses fastest.

Three changes move the result more than anything else:

• Mask the headless fingerprint. A stock Playwright launch exposes navigator.webdriver and other automation tells. A stealth patch such as playwright-stealth, or a CDP-only tool like nodriver, removes the most common signals.
• Use residential proxies. Playwright takes a proxy per browser or per context through the proxy= argument. Datacenter ranges are pre-flagged by X, so residential or mobile IPs survive far longer.
• Pace and vary behavior. Bursty, identical requests trip per-IP rate limits. Randomized scroll distances, short waits, and a realistic viewport read as human.

browser = p.chromium.launch(
    headless=True,
    proxy={"server": "http://residential-host:port",
           "username": "user", "password": "pass"},
)

You can also make a run lighter and harder to fingerprint by blocking images, fonts, and media with page.route(), which cuts bandwidth and trims the surface X can measure:

page.route("**/*", lambda route: route.abort()
    if route.request.resource_type in {"image", "font", "media"}
    else route.continue_())

One boundary is worth respecting. X’s robots.txt sets a catch-all Disallow: / for generic bots and names paths like /*/followers and /search/realtime explicitly, so public, logged-out profile and tweet content is the defensible zone rather than gated lists. I go deeper on the exact block signals in how to scrape Twitter without getting blocked. Even a well-disguised session, though, runs into the one problem no stealth patch fixes: X keeps moving the target.

Why do Playwright Twitter scrapers keep breaking?

Playwright Twitter scrapers keep breaking because X rotates the GraphQL doc_id query hashes and renames operations on its own schedule, so a capture that filtered on yesterday’s operation name suddenly returns nothing. Nothing in your Playwright code is wrong when this happens. The URL you were matching just stopped appearing in the response stream, which is why a working scraper needs re-checking every few weeks.

The guest-token path that unauthenticated scrapers lean on is just as unstable. X’s guest tokens expire within hours and are bound to the requesting IP, and the activation endpoint refuses requests without the current web bearer, so a scraper built on it needs constant re-activation.

The sanctioned alternative is not cheap either. X’s API pricing lists post reads at $0.005 per resource on the pay-per-use plan, capped at 2 million reads a month, and the standalone free read tier is gone. A job that reads even a few hundred thousand posts a month runs into real per-read billing, which is the backdrop for every build-or-buy decision here.

There is a legal line worth knowing before you scale up. US courts have treated scraping public, logged-out data favorably, most directly in X Corp. v. Bright Data, where Judge William Alsup dismissed X’s claims in 2024 against a company that scraped and resold public posts. X’s Terms of Service separately restrict automated access without permission and set liquidated damages of $15,000 for any party that accesses more than 1,000,000 posts in 24 hours by automated means. For a feed you need to keep alive within those limits, moving the doc_id chase off your plate is usually the cheaper trade.

How do you scrape Twitter without managing a browser or tokens?

You scrape Twitter without managing a browser or tokens by calling a scraper API that takes a username, tweet ID, or search query and returns parsed JSON, while the provider handles the GraphQL operations, doc_id rotation, proxies, and login. You send one HTTP request from any language and get clean fields back, with no Chromium process and nothing to patch when X renames an operation.

This is the request shape I use. It is a single GET with the target and an api_key:

curl "https://chocodata.com/api/v1/twitter/profile?username=nasa&api_key=$CHOCO_API_KEY"

The Python version is the same shape and drops straight into your parsing code, returning the fields you would otherwise dig out of a nested GraphQL payload:

import requests

resp = requests.get(
    "https://chocodata.com/api/v1/twitter/profile",
    params={"username": "nasa", "api_key": "YOUR_CHOCO_API_KEY"},
    timeout=30,
)
resp.raise_for_status()
profile = resp.json()["data"]
print(profile["name"], "-", profile["followers_count"], "followers")

When I probed the ChocoData endpoint in July 2026, a request with an invalid key was rejected before it returned any data, which matches its rule that unauthorized requests do not consume credits. To collect a keyword or hashtag feed instead of a profile, you swap the resource to search and read the list the same way, with no SearchTimeline operation name to track:

curl "https://chocodata.com/api/v1/twitter/search?query=nasa&api_key=$CHOCO_API_KEY"

For a one-off pull of a few hundred records, a Playwright script from this guide is fine. For a feed you need to keep alive, moving the token refresh and doc_id chase to a provider is the cheaper trade once you price in the patching. If you want to compare the managed options against the DIY routes head to head, I rank them in best Twitter scrapers in 2026.