We Logged 30 Days of AI Bot Traffic. About 2,500 Fetches Reached a Live Answer

About 2,500 verified fetches were linked to a live answer across 30 days of ridgequarter.com logs. That is the number I would report. The raw logs looked far more impressive: /robots.txt alone drew 13,494 hits claiming an AI crawler user agent, and the homepage drew 6,506. On both paths, roughly 70% failed verification against the bot vendors’ published IP ranges.

I have spent enough late nights mining search terms to distrust any dashboard that gets bigger when you stop asking questions. So we split the traffic by path and bot, then asked three different things of it: Did a request claim to be an AI bot? Did its IP match that bot’s published range? Could we link the verified fetch to a live answer served to a user? Those are not interchangeable questions. As Canonry explains in its server-log walkthrough, bots do not run JavaScript, so Google Analytics misses their requests. The evidence starts in access logs, but a user agent string is only a claim until the IP checks out.

The headline: about 30% passed verification on two key paths

The first cut is from our edge logs for a single 30-day window. Total hits means requests claiming an AI crawler user agent. Verified means the source IP matched the published range for the bot it claimed to be. This table does not say whether any of those requests fed an answer.

Source: ridgequarter.com edge logs, 30-day window. The SVG figure is a verified share, not a verified-hit count.

A raw crawler chart makes /robots.txt look like a busy front door. The IP check says most of that apparent traffic cannot be attributed to the bots named in the requests. Decorative SVGs were starker: only about one or two requests in a hundred passed verification. A claimed bot hit is not verified bot traffic, much less evidence that someone saw your brand in an AI answer.

Across all logged pages, about 2,500 verified fetches were linked to live answers. That count comes from joining verified requests to answers served to users, rather than from a mention score or the raw crawler total. It is the useful third cut, with an important limit: the join shows that a fetch was associated with an answer, not that the answer cited the fetched page or copied a particular sentence from it.

A user agent is self-reported. Anyone can put ChatGPT-User in a request header. Verification means checking the source IP against the vendor JSON for that exact bot, not merely checking its network operator. This is not a fussy distinction invented for a cleaner spreadsheet. HUMAN found that 5.7% of traffic claiming an AI crawler user agent was spoofed, with ChatGPT-User abused at roughly one in five. DataDome reported a 45% increase in AI agent spoofing from February to July 2026. Our unverified share is much larger than HUMAN’s broader figure. That is a reason to examine this site’s traffic, not to promote our percentage into an internet average.

Finding 1: the bot name did most of the pretending

Start with /robots.txt. It drew 13,494 requests claiming an AI bot identity, but only 3,912 passed the IP check: a roughly 71% haircut. The homepage was nearly identical, with 1,930 verified out of 6,506 claimed, or a 70.3% haircut. I used to tell clients bot logs were directionally true. I was wrong. On this small site with open robots, the unfiltered direction was mostly costume.

Decorative SVGs gave the game away. Thousands of requests, a verified share near one or two percent. No answer engine needs a divider graphic to write a sentence about Google Ads pricing. An unverified request that claims a bot identity and walks an asset tree is not evidence of search visibility. It belongs outside the visibility count, however authoritative its user agent looks in a CDN panel.

The IP check needs to be specific. HUMAN reported that much of the spoofed ChatGPT-User traffic it observed came from the same ASN as real traffic, using adjacent IPs and rotation across more than 10,000 IPs. Checking the ASN alone would let those requests through. Checking the exact published CIDR for the claimed bot is the difference between verification and recognising a familiar-looking neighbourhood. If the report verifies only the ASN, do not label the result verified AI visibility.

Finding 2: 250 verified fetches can still mean zero answers

Our category pages show why IP verification is necessary but insufficient. Several drew roughly 250 verified fetches each during the window. These were real bot requests by our check, not merely familiar names in a header. Yet those pages fed zero live answers in the answer-linked count. A report built from verified crawls would make them look productive. A report built from what reached a live answer would not.

That gap is easier to understand when you separate bulk crawling from a user-triggered fetch. Canonry distinguishes bulk crawlers such as GPTBot and ClaudeBot from user-facing fetchers such as ChatGPT-User and Claude-User. A bulk crawl can establish that an engine has read a page. It cannot establish that the page was read while answering someone’s question. Our category pages received the former kind of attention without showing up in the latter count.

The content distinction matters too. These category pages were broad and thin on a direct answer; they were easy to revisit without supplying much an engine could use in a response. That is an interpretation of the pattern, not something an edge log can prove about a model’s reasoning. The measured result is narrower and still useful: verified fetches accumulated, while answer-linked fetches did not.

Citations are a further step beyond that. Across 22 categories, Semrush found that fewer than half of the top-100 mentioned brands were also top-100 cited domains. A brand can be present in an answer without its site being cited. A page can be fetched without appearing in the answer at all. Crawled, used in a live answer, mentioned and cited are four different labels. Put them in separate columns before deciding which page has won.

Finding 3: answer-linked activity clustered on a few paths

The roughly 2,500 answer-linked verified fetches were not spread evenly across the site. Our edge-log and answer join showed this shape:

Source: ridgequarter.com verified edge logs joined to live answers during the same 30-day window. These are answer-linked fetches, not citation counts.

/robots.txt was also rechecked within many live sessions. It matters to the activity trail, but it does not supply answer text, so I would not present it as a content win alongside the homepage and the updates post. The window is short enough that the 2026 post’s burst of attention can tilt the distribution. Still, the operating lesson is plain: a sitewide bot total hides which URLs were present when answers were served. That is the part a content owner can inspect and change.

The bot names hid another split. In our join, ChatGPT-User accounted for most live-answering fetches, including requests to the homepage and the updates post around the time a question was asked. Claude-User showed steadier reading and far fewer fetches tied to an answer served seconds later. I would not collapse those patterns into one AI visibility score. The platforms also produce different answer shapes: Semrush found differences among Gemini, ChatGPT and AI Overviews in brand mentions, source entries and overlap. One bot’s strong count tells you little about another platform’s answers.

Put prompt results beside logs, not in place of them

Most businesses I talk to start with one of two reports. One is a prompt dashboard: run a fixed set of buyer questions across AI search products and record mentions, citations and sentiment. Search Engine Land describes a Brand Visibility Score based on answers mentioning the brand divided by total answers; HubSpot frames measurement around visibility, engagement and revenue. That tells you what appeared in sampled answers. By itself, it does not tell you which page was fetched near the answer or whether your server was asked at all.

The other report is a firewall or CDN bot total. It tells you something knocked. Without IP verification, it cannot even confirm who knocked. With verification, it still cannot tell you whether a fetch was associated with a live answer. Our earlier postmortem on 3,800 fetches that produced zero citations dealt with the page-level gap. This 30-day log cut deals with the measurement gap. Neither a prompt chart nor an edge log closes both on its own.

Here is the practical join I would keep:

1. Hold the questions steady. Use a fixed set of 25 to 50 buyer prompts naming your category. Each week, record whether each answer mentioned you, cited you with a link and how it described you. Keep ChatGPT, Gemini and AI Overviews separate.
2. Split requests by path. Record claimed AI-bot hits, IP-verified hits and verified fetches linked to a live answer. Do not silently promote a request from one column to the next.
3. Compare the paths with the answers. Join by date, prompt, bot name and path where the records allow it. Inspect which URLs appear around cited answers, then read those pages as a buyer would.

The mismatches are the useful part. If claimed hits rise while verified hits do not, fix the filter before telling anyone visibility grew. If verified fetches rise but answer-linked fetches do not, more crawling is not the result you wanted. If answer-linked fetches rise while your fixed prompts still show no citations, inspect what the page says and what the answer actually used; do not call the fetch a citation. The join narrows the question. It does not give you a window into the model.

What I’d change on Monday

If I owned this report for a business starting Monday, I would keep three measures:

• Verified live-answer fetches by path. This is the crawl measure closest to an answer someone saw, rather than to a bot’s claim about itself.
• Citation rate on a fixed prompt set. Track cited prompts divided by total prompts each week, separately for ChatGPT, Gemini and AI Overviews.
• The URLs associated with cited answers. Join prompt results to verified requests by date and bot where possible. Use the pairing to decide what to inspect or rewrite, without pretending timing proves causation.

I would stop using raw AI-bot hits as a visibility headline. On our two busiest paths, they were about 3.4 times the verified count; on SVGs, the verified share was only roughly one to two percent. I would also stop treating mention-only scores as citation scores. The mention-versus-citation gap Semrush observed is precisely why those labels need their own lines in the report.

This will not work for everyone. If your site gets fewer than a few hundred verified AI fetches a month, the path-level log split may be too thin to read. I would keep the fixed prompt tracking and wait for more volume rather than manufacture a trend from a handful of requests.

The limit, and the decision

This is one small B2B site, 30 days of logs, open robots and a narrow topic cluster around paid search. Our roughly 70% unverified share is not an internet average. A larger site with different bot rules, or a publisher blocking AI crawlers, will see a different mix. The window is also short enough for one heavily read post, such as our 2026 updates post, to skew where answer-linked fetches appear.

There is a harder limit. We can match a verified fetch to a live answer using timing, path and bot name. We cannot see inside the model and prove that the fetch caused a citation. Training-data memory may also contribute to what an answer says. That is why I want the prompt record beside the logs, and why I will not relabel the 2,500 answer-linked fetches as 2,500 citations.

The decision the numbers support is boring, which is why I trust it. Stop reporting the big bot total. Report the about 2,500 verified fetches linked to live answers in 30 days, show which paths carried them, and compare those paths with a fixed prompt set every week. When a page gets read but does not appear in your sampled answers, inspect the page. When a buyer question gets answered without you, work on the answer your page fails to provide. Everything else in the AI visibility deck is scenery. Count what reached an answer, not what knocked.