Every quarter, the way humans and machines reach websites diverges further. AI agents now read more of the web than people do, the rails to charge them are live, and the open standards to identify them are converging. This is the data behind why AgentRail exists.
Cloudflare publishes the crawl-to-referral ratio for each major AI operator — how many requests their bots make to a site for every human visitor that AI sends back. The numbers are the clearest signal that the open web has become an unmetered raw-data feed for model training.
Cloudflare's traffic-by-purpose breakdown over the trailing twelve months: 80% of AI crawling is for model training, 18% for search-style answers, and just 2% on behalf of a logged-in user.2 Six months later: training rose to 82%, search dropped to 15%, user actions inched to 3%. The premise that "agents drive traffic back" doesn't survive contact with the data.
GPTBot went from 4.7% of monitored AI crawling in July 2024 to 11.7% in July 2025. ClaudeBot rose from 6% to nearly 10% over the same window.3 The chart isn't asymptoting — it's still slope-up. Every site that hasn't put an AI-facing layer in front of its content is, today, subsidising the next foundation model.
For two decades, "bot traffic" meant scrapers, spammers, and noise — a tax to be filtered out. In 2025 that mental model broke. Bots became the median visitor, and AI bots became the fastest-growing category of bot.
Inside the AI bot category, training crawlers fell from 90% of AI-driven traffic to 74% across 2025. Scraper bots rose from 10% to 24%, and a new category — agentic bots performing autonomous tasks — emerged at 1.7% of all AI traffic and is growing the fastest.4 That 1.7% is what the OpenAI Atlas / Claude for Chrome / Comet generation actually looks like on your access logs.
The IT and B2B SaaS sector already sees 2.8% of total site visits from AI sources — the highest of any vertical.6 Cohort data from Loamly shows individual B2B SaaS sites with 5–15% of total traffic from AI. If you're a structured-data API, you're not waiting for this — you're already in it.
For 25 years, the deal was simple: search engines indexed your content, sent you traffic, and you monetised that traffic with ads, subscriptions, or commerce. AI Overviews, AI Mode, ChatGPT search, and Perplexity each break the chain at the same point — they answer the user without the click.
In a single twelve-month stretch every major AI lab shipped a browser that can read pages, fill forms, and complete tasks on behalf of a user. Some live inside Chrome; some replace Chrome. All of them hit your site looking nothing like the visitor your analytics is built for.
The licensing market for AI training and answers has gone from theoretical to multi-billion-dollar in 18 months. Every deal that gets signed proves the price exists. The problem is who has the leverage to capture it.
Reddit, the NYT, the Washington Post, Reuters, the AP — every site with the leverage to be sued or to be irreplaceable has a deal. Every other site has nothing. A B2B catalog, a fintech data startup, a SaaS docs site, a vertical marketplace — none can negotiate bespoke licenses the way Reddit can, but their data is just as agent-readable. They need a platform that lets them charge by default, without a contract negotiation per AI vendor.
For years, "agents will pay merchants" was a slide. In a 12-month window it became a live network:
open standards, hyperscaler integrations, and real settled volume. The wallets exist. The
402 Payment Required spec is finally being used.
402 status code. APIs and agents settle in USDC on a single round-trip.21Three pieces of the stack are quietly being settled in the open: how agents discover tools (MCP), how sites describe themselves to LLMs (llms.txt), and how agents prove who they are (IETF identity drafts). Each one is becoming a default in real time.
Up from 1,200 a year ago — 7.8× year-over-year growth. 78% of enterprise AI teams report at least one MCP-backed agent in production by April 2026. Anthropic's SDK alone hit 97M monthly downloads — matching React's growth curve in a fifth of the time.27
A Markdown manifest at the domain root that points AI systems at clean, token-efficient
summaries of your content. Already in production at Anthropic, Perplexity, Stripe, Cursor,
Cloudflare, Hugging Face, Zapier.28 An llms-full.txt variant adds
governance, update cadence, and structured links to ai-manifest.json.
Active drafts: Agentic JWT (cryptographic agent identity with workflow-bound tokens), Web Bot Authentication Architecture (HTTP message signatures from agents), AI Agent Authentication & Authorization, and Digital Identity Management for AI Agent Communication.29 Visa published its Trusted Agent Protocol spec on the same theme.
Each standard is solving one piece of the same puzzle: turning "an HTTP request" into a first-class, identified, billable agentic action. AgentRail's middleware sits exactly at the seam — it reads the agent's identity, applies the merchant's policy, returns the right price, and posts back a structured event. Once the standards solidify, the merchant-side layer is the natural place to enforce them.
Once agents can pay, what do they buy? Every major research house has run the numbers. The forecasts vary by trillion or two — they all agree on the shape: a new layer of commerce, mostly agent-to-API, growing 30–50% a year for the rest of the decade.
Every chapter above is one input into the same conclusion. The economic layer of the agentic web doesn't exist for 99% of sites today. The data says it has to — and that the buy-side is already ready to pay.
The data above is the why. The product is the how. See AgentRail in action, or get on the waitlist.
All figures are pulled from primary publications between July 2024 and May 2026. Where two sources report different numbers for the same metric, we use the most recent figure from the more methodologically transparent source (Cloudflare Radar, Similarweb, HUMAN Security, primary company announcements).