Our analysts track the ranking and AI search changes that move agency traffic reports. Brands building search programs can partner with vetted SEO agencies for strategic implementation.
The week brought a shared AI opt-out proposal and new data on AI's toll on clicks.
- Google, Microsoft, OpenAI, and Anthropic plan to honor one robots.txt signal for AI training opt-outs, with Microsoft starting February 1, 2027.
- Tracking tools flagged a fourth unconfirmed ranking spike since October 16, with no word from Google.
- Cyrus Shepard estimated a 0.9% CTR for AI impressions, based on one site's opt-out test.
- News Box-only results fell 41 points, to 34% of U.S. Google results pages between October 2025 and August 2026, per Trisolute News Dashboard data.
- Google expanded its spam guidance to tools that publish shared pages, generated content, or web apps.
Before any site blocks an AI feature, agencies need to know who can still ignore the block.
Our Take: Can a Licensing Deal Override an AI Opt-Out?
The IETF draft behind the shared token says private agreements may override a site's AI training preference.
It adds that a crawler gathering only training data may skip fetching pages marked against training, leaving the choice to each crawler.
We'd read a signed licensing deal as the winner, which leaves the token as a request from sites that hold no contract with the provider.
Reddit shows the model, blocking most automated crawlers and licensing its posts to Google for AI training.
A brand that writes the token asks each provider to honor a preference.
Any licensing deal the brand signs with that provider can take precedence, so the training terms in that contract deserve a read before signing.
Until vendors confirm how they handle the token, agencies should tell clients chasing AI visibility that any provider can still override it.
Shared Training Token Could Pull Sites From Gemini Answers
Google, Microsoft and OpenAI announced plans to honor one robots.txt token that opts a site out of AI training at every participating provider.
Google targets early 2027, Microsoft starts on February 1, 2027, and OpenAI aims to implement it "in the coming months."
Anthropic's Kevin Kelley, who wrote the IETF draft behind the token, said the Claude maker has started implementing it.
A site would write one policy token into robots.txt to cover every participating AI crawler.
@rustybrick @gaganghotra_ fyi, Barry, i've been spending the last year deeply participating in this group. you'll notice my name at the bottom in the acknowledgments. if you need help understanding, dm (also, fyi, it's not actually done at the ietf level -- and some significant areas of dispute remain)
— Nate Hake (@natejhake) October 30, 2026
The draft has no formal standing in the standards process yet.
Working group participant Nate Hake wrote on X that the process is not done and significant areas of dispute remain.
Google says it will read the token as a Google-Extended disallow on any site that leaves Google-Extended unspecified.
A Google-Extended disallow already removes pages from Gemini Apps grounding, so one training line could cost a client its Gemini visibility.
Agencies editing a client's robots.txt should treat every training rule as an AI visibility decision.
Google Leaves October Ranking Volatility Unexplained
Tracking tools showed another Google ranking spike around October 27, amid chatter about an unconfirmed update, Search Engine Roundtable reported.
The site has now flagged four spikes since October 16, none of them confirmed by Google.
Google Search ranking volatility continues to get more heated by the day - is it poisoned or is Google brewing something - old story at https://t.co/ZipF0uAfoh https://t.co/LBMIIjLQmF
— Barry Schwartz (@rustybrick) October 29, 2026
Google's last confirmed update, the September 2026 spam update, ended on October 8 after running 13 days and 16 hours.
Four unconfirmed spikes in under two weeks echo January’s ranking swings, which also ran without a confirmed update.
An SEO report that blames an unconfirmed update for a traffic drop can send the agency chasing the wrong fix.
Search Console data and the site's own change log are the first places to check before any cause goes into a client report.
Shepard's AI Feature Test Cut Organic Clicks, Too
SEO advisor Cyrus Shepard pulled one site from AI Overviews and AI Mode through Search Console, which cut its AI impressions to zero.
Its regular organic clicks dropped too, the result that Shepard called most surprising.
He estimated a 0.9% click-through rate for AI impressions, which made up about 8% of the site's impressions.
AI Overviews kept appearing for these queries during the test, just without Shepard's site in them.
Sitewide average position also rose from new posts and brand search, a confound that he acknowledged.
Search Console's AI reports show impressions without clicks, so Shepard estimated CTR by subtracting known AI impressions from site totals.
The shared training token is a separate control, so a training opt-out's cost still needs its own test.
With a third of site owners in Barry Schwartz's January poll ready to block AI Overviews if given the option, agencies need a way to price that call.
AI Overviews Win the Top Slot Over Top Stories
U.S. Google results pages showing only a News Box dropped 41 points between October 2025 and August 2026, to 34%.
Pages showing only an AI Overview climbed to 33% over the same stretch.
News SEO strategist Joy Johnston reported the figures in Search Engine Land, using monthly data from her employer, Trisolute News Dashboard.
Across the study period, an AI Overview ranked above the News Box 96% of the time on desktop and 87% on mobile when both appeared.
Johnston wrote that Google could eventually retire the News Box and show Top Stories only inside AI Overviews.
Brands that count on earned media for brand visibility in search should track AI Overview citations next to Top Stories placements.
Google Holds Publishing Platforms Responsible for User Pages
Google updated its user-generated content (UGC) spam guidance to cover tools that let users publish shared pages, generated content, or web apps.
It tells platforms to keep pages meant for sharing out of the index by default, using noindex or a robots.txt block.
The company’s noindex page says that a robots.txt block means the crawler "will never see the noindex rule."
Its robots.txt guidance adds that a blocked page can still be indexed if other sites link to it.
Platforms that rely on the block alone may still see shared URLs in results.
Search Engine Journal read the update as a sign of tougher enforcement, while noting the guidance stops short of announcing a crackdown.
Google Search Advocate John Mueller said in January that link spam in comment sections doesn't hurt the host site's SEO.
The new guidance covers platforms where users publish whole pages, which leaves the platform itself open to a Google manual action.
Agencies running technical SEO audits should check whether any client pages published through these tools are indexed.
SEO Industry Insights
Technical SEO stays relevant for AI search, Mueller told David Iwanow in a video interview.
He expects AI systems to keep building on it, with crawling for model training and for fresh retrieval now part of the job.
Mueller also said that he hasn't seen long redirect chains cause problems, since Google can often learn the final URL and crawl it directly.
Peec AI GEO expert Tomek Rudzki found that about an eighth of a typical page (1,900 of 14,600 characters) reaches Claude's answer model.
Across 734 pages, BM25, a 1990s keyword formula, picked the passage that Claude quoted 41.6% of the time, against 21.9% for random picks.
Peec noted that Anthropic has confirmed none of it.
It also found that 89.1% of pages Claude cites rank in Brave's top 20, which puts Brave indexing on any AI visibility checklist.
This week's opt-out news calls for three checks before any block goes live:
- Price the block. Compare Search Console clicks before and after any block to see how much the control costs.
- Keep Google-Extended explicit. Write the rule out in robots.txt, so the shared token can't decide it by default once Google supports it.
- Set noindex on shared pages. Tag every shared or generated page that a tool publishes, and leave the URLs crawlable so Google can read the tag.
Our last SEO roundup covered Gary Illyes' point that robots.txt governs crawling.
The shared token now adds AI training preferences to the same file.
Which of your traffic sources would one AI opt-out line put at risk?
These leading SEO agencies in our directory help brands price AI opt-outs against search traffic and audit robots.txt rules.
