Twitter / X Terms of Service & Scraping Policy Explained

I sat down and read the X (Twitter) terms of service on scraping the way an engineer reads an error log: looking for the exact line that decides the outcome. The line is short and it is not ambiguous. X says crawling or scraping the Services in any form, for any purpose without prior written consent is expressly prohibited. That sentence is the whole policy in one breath, and everything else (the developer agreement, the automation rules, robots.txt) builds on it.

This guide sticks to the policy itself. I quote the documents directly, link each one, and separate the contract question (what the terms say) from the legal question (what courts have actually done). Those two get blurred together constantly, and the difference is the entire point.

What do the X (Twitter) terms of service say about scraping?

The X Twitter terms of service prohibit scraping in plain language: crawling or scraping the Services in any form, for any purpose without prior written consent is expressly prohibited. This clause entered the X Terms of Service in the September 2023 rewrite and remains in the current version, which carries an effective date of May 8, 2025.

The prohibition is deliberately broad. “In any form, for any purpose” covers the logged-out web page, the logged-in timeline, the mobile app surface, and the API responses. There is no carve-out for public data, no carve-out for research, and no carve-out for low volume. The only stated exception is “prior written consent,” which in practice means a license or the official API under its own terms.

The same Terms of Service section also restricts the rate and the method of access. X reserves the right to limit how you use the Services and prohibits attempts to work around technical limitations or to disrupt the operation of the platform. That language matters for scrapers because bypassing a login wall, a rate limit, or a CAPTCHA is itself called out as a violation, separate from the scraping clause.

Here is the consumer Terms of Service language that scrapers run into, mapped to what each clause governs.

The takeaway from the consumer terms alone is that scraping is prohibited by contract before you write a line of code. The next document, the developer agreement, governs the sanctioned path X points you toward instead.

What does the X Developer Agreement say about scraping and the API?

The X Developer Agreement governs the official X API and prohibits reverse-engineering it, which closes the unofficial-client route that many scrapers historically used. It states you shall not reverse engineer, decompile, disassemble, or translate the X API or otherwise attempt to derive source code, and it bars reselling, sublicensing, or redistributing access to the API or its data. The companion Developer Policy reinforces this, instructing developers to use the API “as intended and documented” and barring any attempt to “interfere with, intercept, disrupt, or disable” its features.

This is the document that turns “use the API instead” into a set of binding conditions. The developer agreement does not repeat the consumer scraping clause word for word. It assumes you reached it because you chose the sanctioned door, and it then tells you what you may and may not do once inside: register an app, stay inside the rate limits, do not derive the private endpoints, do not redistribute the raw data.

For anyone weighing “scrape the site” against “use the API,” the agreement reframes the choice. The API is the only access X authorizes in writing, and it is metered and priced. The tiers on X’s developer platform make that concrete. As of 2026 the structure is Free for testing, Basic at $200 per month, Pro at $5,000 per month, and Enterprise in the tens of thousands, alongside a newer pay-per-use model that charges roughly $0.005 per post read against a monthly cap near 2,000,000 reads.

Those numbers are why the “scraping prohibited” clause has teeth. The sanctioned alternative carries a real and rising cost, so the policy is also a commercial gate. The automation rules sit on top of both documents and govern what your code is allowed to do once it has access.

What do the X automation rules say about scraping and bots?

The X automation rules prohibit programmatic actions that scraping projects often bolt on, and they route all automated access through the official API. X’s official automation development rules state that you may not “use non-API-based forms of automation, such as scripting the X website, which may result in the permanent suspension of your account.” The same page bars unsolicited bulk messages and lists automated following, liking, and reposting among the behaviors it treats as spam.

The automation rules and the scraping clause cover different verbs. Scraping is about reading and collecting data. Automation is about writing and acting: following, liking, posting, messaging. A read-only scraper does not trip the follow/unfollow ban, but it still trips the scraping clause, so neither document gives a read-only project a clean pass.

Where the two overlap is the method. Both the scraping clause and the automation rules treat browser automation and unofficial clients as outside the sanctioned path. The automation rules name the official X API as the route for any programmatic access, which lines up with the developer agreement and the consumer terms. The policy is consistent across all three documents: the API is the one approved door, and everything else is prohibited by contract.

A summary of the three policy documents and what each one prohibits:

With the written policy mapped, the next question is what the machine-readable signal says, because robots.txt is the file scrapers check first and the one most often misread.

What does X’s robots.txt disallow for scraping?

X’s robots.txt gives the wildcard User-agent: * a Disallow: /, which asks every bot that is not explicitly named to crawl nothing on the site. When I fetched the live file, the default rule was a full disallow, and the only agents granted selective access were a short allow-list led by Googlebot and Bingbot.

The named search bots do not get the whole site either. Googlebot and Bingbot are allowed to reach public surfaces like hashtag pages and search queries, while follower and following lists (/followers, /following), likes (/likes), analytics (/analytics), media (/media), and photo endpoints (/photo) are disallowed even for them. So the file is not a simple open/closed switch. It is an allow-list where almost everyone lands in the “crawl nothing” bucket.

Two facts about robots.txt decide how much weight to give it:

• It is a voluntary request. The Robots Exclusion Protocol relies on the crawler to comply. A server does not enforce it the way a login wall blocks access, so a scraper can technically ignore it. The official protocol is explicit that compliance is up to the crawler.
• Ignoring it removes a defense. Because X publishes a Disallow: /, you cannot later argue the file permitted your bot. Combined with the terms of service, the disallow turns “the file let me in” into a non-starter.

X’s robots.txt policy in 2025 and 2026 reflects the same posture as the 2023 terms rewrite: the prior version of the terms had pointed to robots.txt as the guideline for permitted crawling, and the rewrite deleted that provision. Now the file disallows nearly everything and the terms require written consent regardless of what the file says. The machine-readable signal and the legal text point the same way. What they do not settle is whether any of this is actually enforceable in court, which is the separate legal question.

Is scraping Twitter (X) illegal under the law?

Scraping Twitter is prohibited by the terms of service, and that is a contract question separate from whether it is illegal under statute. A breach of the terms exposes you to the remedies X wrote into the contract. Illegality would require violating a law such as the Computer Fraud and Abuse Act (CFAA), copyright, or a data-protection regime. US courts have recently pulled those two questions apart, and the results have not gone X’s way for public data.

In X Corp. v. Bright Data, the U.S. District Court for the Northern District of California dismissed X’s breach-of-contract and tort claims in May 2024. Judge William Alsup held that the claims, as applied to scraping and selling publicly available data, were preempted by the Copyright Act, and he wrote that letting platforms fully control public data “risks the possible creation of information monopolies that would disserve the public interest.” The parties later moved toward a settlement, so the dispute did not produce a binding appellate rule, and the reasoning still signals how a major court views public-data scraping.

That ruling rhymes with hiQ Labs v. LinkedIn, where the Ninth Circuit found in 2022 that scraping publicly available data likely does not violate the CFAA, because public pages are not access “without authorization” in the statute’s sense. Read together, the cases describe a US landscape where scraping public data is hard to cast as a federal crime, even when a site’s terms forbid it.

The risks that remain are real and worth stating plainly:

The honest summary is that scraping public X data is unlikely to be a CFAA crime in the US after hiQ, the terms of service still prohibit it as a contract matter, and copyright and privacy law set independent limits. None of that is legal advice, and the rules differ by jurisdiction. I keep a fuller treatment in is scraping Twitter (X) legal. The practical question for most teams is how to get the data they need while staying inside these lines, which is where compliant access comes in.

How can you collect X data while respecting the policy?

You stay inside the X policy by getting data through a route X authorizes or that does not require you to breach the terms yourself: the official X API under its license, a written data agreement, or a managed provider that takes on the access decision. Each route trades cost, control, and risk differently.

• Official X API. The only access X authorizes in writing. It is metered and, since the 2026 pricing change, billed pay-per-use at roughly $0.005 per read with a hard cap near 2M reads per month. It is the cleanest path on policy and the most expensive at volume.
• Licensed or enterprise data. A negotiated agreement or X’s enterprise tier (reported around $42,000 per month) gives broad access with contractual cover. This fits large, funded data operations.
• A managed scraping API. A third-party service that returns structured X data and handles the collection layer, so you are not personally bypassing X’s gates in your own infrastructure.

For the third route, the request shape is what you would expect from a normal REST call. Here is the ChocoData pattern for a profile lookup, the documented form for its X endpoints:

curl "https://chocodata.com/api/v1/twitter/profile?username=nasa&api_key=$CHOCO_API_KEY"

The same chocodata.com/api/v1/twitter/... shape covers the other surfaces a project usually needs: the profile and account endpoint, the tweet and post endpoint, the search, hashtag and trends endpoint, and the follower and following endpoint. You can start one on the ChocoData sign-up and call it with a single API key.

Choosing among these is a judgment call about volume, budget, and how much policy and legal risk you are willing to hold yourself. The terms of service draw the line in the same place no matter which you pick: scraping without consent is prohibited, the API is the sanctioned door, and public-data case law has softened the legal edge without erasing the contract. If you want the engineering side of getting data out without tripping blocks, I walk through it in how to scrape Twitter (X) without getting blocked and the broader complete guide to scraping Twitter (X).