The Question Behind Every Social Listening Project

Brand teams, market researchers, trend analysts: everyone wants social data, because it's where customers say what they actually think instead of what a survey nudges them into saying. The trouble is that "scrape social media" means a completely different project depending on which platform you point at, and most articles on the subject either promise you everything or quietly skip the parts that don't work.

So this is the honest map. Platform by platform: what public data you can realistically collect, how people actually collect it, and where the hard limits are. If you're scoping a social listening project, read this before you commit budget. The budget conversation changes a lot depending on whether "social" turns out to mean Reddit or Instagram.

One rule up front, because everything else hangs off it: login-walled content is off the table. If you need an account to see something, collecting it at scale means breaking the terms you agreed to at signup, babysitting authenticated sessions that platforms fingerprint and ban within days, and often touching data people shared with a limited audience in mind. We don't do it, and we'd tell you not to either. What's left, the genuinely public surface, is still a huge amount of signal.

What "Public" Actually Means

The test we use: open the URL in a clean incognito window, no cookies, and see if the content renders. If it does, a search engine crawler can see it too, and it's public in the practical sense. That covers public posts and video pages, comments visible to anonymous visitors, profile pages with bios and follower counts, and whatever engagement numbers those pages display.

It does not cover private or followers-only accounts, DMs, closed groups, anything behind a login prompt even when "everyone" has an account, or data pulled through an authenticated API session with borrowed credentials.

The distinction matters legally as well. US case law (the hiQ v. LinkedIn litigation most notably) has generally treated automated collection of publicly accessible data more favorably than collection behind authentication, though terms of service, copyright, and data protection law all still apply. We wrote up the broader picture in our guide on whether web scraping is legal.

Reddit: Start Here

If a client asks us where to begin a social listening program, the answer is almost always Reddit. It's the closest thing to a fully public forum among the major platforms: posts, full comment trees, scores, and subreddit metadata all render without login, and there's an official API on top. The catch arrived in 2023, when API pricing pushed heavy commercial users toward paid enterprise tiers. The free tier's rate limits (on the order of 100 queries per minute for OAuth clients) are fine for monitoring, painful for large historical pulls.

What makes Reddit unusually good for trend research is structure. Conversations are threaded, searchable, and organized by topic, so you can watch sentiment about a product category shift week by week in the exact communities that care about it. A defect complaint in a niche subreddit is often the earliest written record of a problem that later floods support queues.

YouTube: Take the Free API and Say Thank You

YouTube is the rare case where the official route is genuinely good enough, and we'll say so even though we sell scraping. The Data API exposes video metadata, view counts, likes, comments, channel statistics, and search, free within a daily quota of 10,000 units (a search costs 100 units, most reads cost 1). For a typical brand-monitoring workload, quota management beats scraping on every axis.

Scraping earns its place at the edges. The API only returns current view counts, so building a time series means snapshotting on a schedule. Transcript extraction at scale is another gap, and so is monitoring past the quota ceiling. Public video pages render without login, so none of this is technically hard.

X: A Budget Decision, Not a Technical One

X is the cautionary tale. Before 2023 the Twitter API was cheap and entire research fields were built on it. Then free access all but disappeared, the basic paid tier (a couple hundred dollars a month) capped reads at levels too low for serious monitoring, and enterprise access reportedly starts north of $40K a month. The predictable result was a wave of organizations moving from API access to scraping.

The complication is that X login-walled much of the site at the same time. Anonymous visitors see individual tweet pages inconsistently; search and full timelines generally want a login. What's reliably public is fragmentary: individual post URLs, some profile pages. Teams that need broad X coverage today face a genuinely unpleasant choice between enterprise API rates, working within that fragmentary public surface, or buying from resellers of varying legitimacy. Be straight with your stakeholders about this one. X data stopped being cheap for everyone, not just you.

TikTok: Public, But Defended

TikTok's video pages, creator profiles, and hashtag pages are technically public. They render for logged-out visitors, which is why TikTok links preview in your browser. Captions, view/like/share/comment counts, follower counts, and hashtag aggregate views are all collectible fields.

But TikTok spends real money on defense: request signing on its internal endpoints, device fingerprinting, aggressive rate limiting, and terms that prohibit automated collection. There's a Research API, but access is restricted, mostly academic and regionally limited. Our practical read: monitoring a curated set of public creators and hashtags is feasible with serious anti-detection engineering. Platform-wide collection is an arms race, and unless scraping infrastructure is your actual business, you will lose it. The first time we ran TikTok collection at any real volume, the thing that broke wasn't the parser, it was the signing scheme, which changed underneath us mid-project. Plan for that kind of churn or don't start.

Instagram: The Hardest Mainstream Target

Instagram has spent years shrinking its logged-out surface. Public profiles and individual posts sometimes render for anonymous visitors and sometimes bounce to a login wall, and which one you get depends on IP reputation, request history, and whatever experiment Meta is running that week. Meta's terms prohibit scraping and Meta litigates.

What's realistically collectible: public profile metadata (bio, follower count, post count) and public post data at modest scale, with sophisticated infrastructure, and with the expectation that your access pattern breaks regularly. The official Graph API covers accounts you own or manage plus limited hashtag search for business accounts, so for competitive monitoring of accounts you don't control there is no sanctioned high-volume path. Track a curated list of public accounts. "All of Instagram" is not a project, it's a wish.

Facebook and LinkedIn

Short section because the honest answer is short: mostly off the table. Facebook's public surface is small and heavily defended. LinkedIn walls nearly everything behind login and has been litigating against scrapers for a decade; the public profile pages it feeds to search engines are the narrow exception, relevant for recruiting intelligence but high-risk at volume. Unless you have a data partnership, leave both out of scope.

Collectibility at a Glance

Platform	Public without login?	Official API viable?	Scraping difficulty	ToS posture
Reddit	Yes, nearly everything	Yes (paid at scale)	Low	Permits API; restricts bulk scraping
YouTube	Yes	Yes (generous quota)	Low–medium	API preferred; scraping restricted
X (Twitter)	Partial, shrinking	Yes but very expensive	Medium–high	Prohibits scraping
TikTok	Yes, key surfaces	Research-only access	High	Prohibits scraping
Instagram	Inconsistent	Own-account only	Very high	Prohibits scraping; litigious
Facebook / LinkedIn	Minimal	Own-account / partner only	Very high	Prohibits; litigious

If you take one thing from the table: build the deep, reliable layers of your program on Reddit and YouTube, treat TikTok and Instagram as targeted curated collection, treat X as a line item to argue about, and leave the login-walled platforms alone.

The GDPR Question You Can't Skip

Social media data is personal data. A username, a profile photo, a post with an opinion in it: under GDPR, and increasingly under US state privacy laws, these are personal data even when publicly posted. "It was public" is not a legal basis for processing. You need one, usually legitimate interest, and legitimate interest requires an actual balancing assessment, not a sentence in a slide deck.

The guardrails that keep programs defensible are mostly common sense. Aggregate where you can; sentiment percentages and trend lines rarely need individual identities attached, so strip or hash usernames unless the identity is the point (tracking a public figure, say). Collect purposefully: "public posts mentioning our product keywords, retained 12 months, pseudonymized" is a program, while "everything about everyone mentioning our brand" is a liability. Stay away from special-category data like health and politics even in public posts. And handle deletion: if a post disappears from the platform, your pipeline should be able to reflect that, because both regulators and platforms care.

None of this is a reason to avoid social data. It's a reason to design the pipeline like adults.

What People Actually Do With This Data

Two use cases account for most of the real demand we see.

The first is brand monitoring. Mentions of your brand and competitors across Reddit threads, YouTube reviews and comments, and public TikTok content amount to an early-warning system that surveys can't match; a product defect surfaces in a subreddit days before it shows in ticket volumes. We've covered that pipeline in depth in our guide to brand monitoring and sentiment analysis.

The second is trend and market research. Hashtag view velocity on TikTok, subreddit growth rates, YouTube search interest: leading indicators for consumer categories like beauty, food, and personal finance, which CPG and investment teams use to spot category shifts quarters before sales data confirms them. This work overlaps heavily with media monitoring, where social sources sit alongside news and review sites in one collection pipeline.

Beyond those two: influencer vetting from public engagement metrics, competitive content analysis, and academic research through the sanctioned research APIs.

How We Run Social Projects

Social platforms are the most adversarial scraping environment there is. Rotating login walls, request signing, fingerprinting, terms that shift quarterly. Running that in-house is a permanent engineering tax, which is the main reason clients hand it to us.

Our approach is unglamorous. We scope honestly, which sometimes means telling you the dataset you asked for doesn't exist at a defensible price and designing the curated version that does. We route each platform to the right tool, official APIs for YouTube and Reddit, managed scraping of public surfaces elsewhere, with proxy rotation, fingerprint management, and request pacing on our side of the fence. Compliance (public-only collection, pseudonymization options, configurable retention) is part of the pipeline design rather than a bolt-on. And what you receive is analysis-ready: deduplicated, normalized across platforms, delivered as CSV, JSON, API, or straight into your database.

Get the Signal Without the Headaches

Public social data is one of the highest-signal datasets a consumer business can collect, provided you're realistic about what each platform allows. Tell us which platforms, keywords, and accounts matter to you, and we'll come back with a blunt feasibility read and a working data sample within days.