Job Postings Are a Leading Indicator, If You Collect Them

Companies announce layoffs in press releases. They announce growth in job postings, usually months before it shows up anywhere else. A company that opens 40 engineering roles in Austin is telling you something concrete about its roadmap and its budget. A company that quietly pulls half its open requisitions is telling you something too, and you won't read about it in a filing until long after the decision was made.

This is why postings became a staple of alternative data. Hedge funds track hiring velocity as a revenue proxy. Recruiters watch where demand is heating up before their competitors notice. Strategy teams reverse-engineer a competitor's priorities from the roles it's hiring for. Workforce analytics firms have built entire products on aggregated posting data.

The raw material is public. Every posting exists because the company wants people to see it. The hard part is collection: postings are spread across boards, aggregators, and tens of thousands of individual career pages, they change daily, and turning them into a clean time series takes more engineering than anyone expects going in. This guide covers where the data lives, what to extract, and the traps that quietly ruin naive collection efforts. We've stepped in most of them at some point.

What Hiring Data Tells You

Start with velocity. The number of open roles at a company, tracked weekly, is a directional growth proxy. Investors watch for inflection points: a sustained ramp usually precedes expansion, and a wave of pulled postings often precedes cost-cutting, sometimes weeks before any public statement. This is a standard input for funds building alternative data pipelines, and one of the more accessible datasets hedge funds collect at scale.

Role composition is often more interesting than the count. A retailer suddenly hiring machine learning engineers is building something. A SaaS company opening its first sales roles in Frankfurt and Tokyo is going international. A startup that swaps its "Growth" postings for "Finance Operations" postings has just been told by its board to behave. Titles, departments, and locations are strategy leaks, published in plain text, on a schedule.

Recruiters and staffing agencies read the same data differently: which companies are actively hiring for the skills they place, how long roles sit open in a given market, where demand outstrips supply. And labor economists aggregate across thousands of employers to see wage trends and remote-work adoption months before official statistics catch up. Government labor data lags by a quarter or more; postings lag by a day.

Where the Data Lives

Job posting data has an unusual source structure, and understanding it is the difference between a clean dataset and a duplicated mess.

Boards and aggregators

Indeed is the biggest aggregator in most Western markets. Enormous volume, but a lot of it is syndicated or scraped from elsewhere, so the same role appears under several listings with slightly different text and dates. Indeed also sits behind Cloudflare, and its dates are relative ("Posted 3 days ago"), which hurts time-series precision more than you'd think.

LinkedIn gets its own warning. The data is rich, but LinkedIn is the most litigious platform in this business. The hiQ case ran for years, and while it settled that scraping publicly accessible pages isn't a federal crime under the CFAA, LinkedIn keeps going after scrapers on contract and other theories, especially anything that touches a logged-in session. Our rule is simple: never scrape behind a LinkedIn login, and treat even public LinkedIn pages as a last resort. The same data almost always exists somewhere less hostile. See our legal overview of web scraping for the wider picture.

One underrated angle: many career pages embed schema.org JobPosting markup so Google Jobs can index them. That's machine-readable JSON-LD sitting right in the page source, and it's frequently the cleanest extraction target on the page. Check for it before you write a single CSS selector.

Career pages and ATS platforms

Here's the thing that separates professional posting datasets from amateur ones: most companies don't host their own job listings. They use an applicant tracking system, and a handful of ATS vendors cover a huge share of the market:

ATS	Typical customer	Data access pattern
Greenhouse	Tech companies, startups to mid-size	Public JSON API per company (boards-api.greenhouse.io)
Lever	Startups, tech	Public JSON endpoint per company (api.lever.co/v0/postings/...)
Workday	Large enterprises	JSON via the career-site's internal API; more complex pagination
Ashby	Newer startups	Public posting API
iCIMS, SmartRecruiters, Jobvite	Mixed enterprise	Varies; mostly structured endpoints

Greenhouse and Lever expose clean, publicly documented JSON. No HTML parsing at all. Once you know which ATS a company uses (the career page URL or a glance at network requests gives it away), one request pulls every open role with title, department, location, and full description:

import requests

# Greenhouse: every open role at a company, as clean JSON
r = requests.get(
    "https://boards-api.greenhouse.io/v1/boards/stripe/jobs",
    params={"content": "true"},
)
for job in r.json()["jobs"]:
    print(job["title"], "|", job["location"]["name"])

For a tracked universe of companies, say 2,000 public tech companies or every portfolio company in a sector, the ATS-first approach gives you primary-source data with none of the aggregator duplication. The real work is the discovery layer: mapping each company to its ATS and board token, then keeping that mapping current as companies switch vendors, which they do more often than you'd like.

And then there's Workday, which most large enterprises use. Workday career sites want session tokens and POST-based pagination, and every deployment is configured a little differently. Budget more time for Workday than feels reasonable. It still won't be enough.

The Fields That Matter

A useful record goes well past title and company: a normalized role family (raw titles are chaos — "Ninja", "Rockstar", "SDE II" — so you need a mapping layer), department, locations and remote status, salary range where disclosed, the full description text, seniority, and the requisition ID for dedup.

The fields that deserve the most care are first-seen and last-seen dates, and they come from your own scrape history, never from the platform's "posted X days ago". Nearly every metric anyone pays for (time-to-fill, hiring velocity, freeze detection) is computed off your observation timeline. Which means daily collection isn't optional. A weekly scrape can't tell you whether a role was filled in three days or six, and the platform won't tell you either.

Mining Tech Stacks from Descriptions

Job descriptions are the most honest public record of a company's technology choices. The marketing page says "AI-powered"; the posting says "experience with PyTorch, Ray, and Kubernetes required."

Extract technology mentions across a company's postings systematically and you get adoption signals (Snowflake showing up in a Fortune 500's data-engineering roles is a real procurement event, interesting both to Snowflake investors and to competitors selling against it), migration signals (postings that mention "migrating from Oracle to Postgres" literally announce the project), and target lists for B2B sellers, an angle that overlaps with scraping for lead generation.

The extraction itself is a dictionary-plus-context problem. Maintain a vocabulary of a few thousand technology terms, match against description text, and handle the collisions. "Go", "React", and "Ray" are all ordinary English words, and a naive matcher will happily conclude that every lifeguard posting in America requires React experience. Capitalization rules and context windows fix most of it. LLM extraction has made the tail cases easier, but at millions of postings a tuned dictionary pass with an LLM fallback costs a fraction of running a model over everything.

Salary Transparency Changed the Dataset

Until recently, compensation was the one field postings almost never disclosed. Pay transparency laws changed that: Colorado in 2021, New York City in late 2022, California and Washington in 2023, Illinois and others since. Employers must publish ranges on postings for roles in, or performable from, those jurisdictions. And because so many roles are remote-eligible, a lot of companies now publish ranges on all US postings rather than maintain two versions.

So a large and growing share of US postings carry employer-stated salary ranges, sitting in public HTML. That's a real compensation dataset, the kind that used to require an expensive survey subscription. Benchmark by role, level, and metro. Watch how a specific competitor's posted bands move over time. Spot the outlier paying a premium in your talent market.

Two caveats before you build a comp product on it. Posted ranges are often uselessly wide (a $120,000–$220,000 band says little about what's actually offered), and some employers post deliberately unhelpful ranges to technically comply. Treat floor, ceiling, and width as three separate signals and don't trust the midpoint.

Ghost Jobs, Reposts, and the Dedup Grind

This is where posting datasets earn or lose their credibility.

A meaningful fraction of listed roles, commonly estimated at 10–20% or more depending on industry, aren't actively being filled. Companies keep evergreen postings open to collect pipeline, to look like they're growing, or to satisfy visa-related posting requirements. A posting open for 200+ days is not a real, urgent vacancy. Flag long-lived postings instead of counting them as fresh demand.

Reposts are sneakier. Plenty of ATS configurations and boards re-list a role under a new ID every 30 days to refresh its posted date. The first velocity numbers we ever produced without requisition-level dedup looked fantastic, and were roughly double reality. Match on company plus normalized title plus location plus description similarity before you trust any trend line.

Then there's cross-source duplication: the same role on Indeed, ZipRecruiter, LinkedIn, and the company's own career page. If you must blend sources, dedup across them and treat the ATS version as canonical. And location normalization: "NYC", "New York, NY", "Remote (Eastern Time)", and "US Remote, HQ: New York" all need to resolve to the same geography before any regional analysis works.

Get these wrong and the dataset produces confident, wrong conclusions. The worst kind.

How ScrapeAny Handles Job Posting Data

The collection layer is deceptively large: ATS discovery across thousands of companies, Workday's session handling, Cloudflare and DataDome on the major boards, daily scheduling, and the dedup and normalization pipeline that makes the output trustworthy. It's the classic project that starts as "a quick scraper" and turns into a permanent engineering commitment.

We run this as a managed service. You define the universe (a list of companies, an industry, a geography, or specific boards) and we handle source discovery, anti-bot evasion, daily collection, requisition-level dedup, salary-range parsing, and delivery in whatever form your workflow wants: CSV drops, JSON, a queryable API, or direct writes to your database. You get a clean daily time series with first-seen and last-seen tracking. We absorb the breakage when Workday changes its markup again, which it will.

If you're weighing an internal build instead, our build vs. buy analysis walks through the actual cost math.

Start With a Sample

The fastest way to evaluate posting data is to look at real output for companies you already know well, where you can spot nonsense on sight. Tell us which companies or markets you care about — talk to our team and we'll put a working sample dataset in front of you within days.