
AI Agents in Your Marketing: What to Automate and What Still Needs a Person
November 4, 2026
How Does AI Turn Google Ads Data Into Spend and Lead Forecasts?
November 4, 2026Google, OpenAI and Microsoft Back One robots.txt AI-Training Signal: Should Your Site Use It?

Short answer: Not yet. Google, OpenAI and Microsoft announced on October 30, 2026 that they will honor a shared robots.txt token, __usage_ai-training, but none reads it today. It only covers training future AI models, not grounding (being cited in AI answers), and for most businesses AI-answer visibility is worth keeping. Review your robots.txt now and decide when support goes live.
What is the new robots.txt AI-training signal?
It’s a shared “policy token” for robots.txt, __usage_ai-training, that lets a website state one preference about AI training for every AI company that honors it, instead of writing separate rules for each crawler.
It comes from an IETF Internet-Draft (draft-kelley-aipref-training-preference-signal-00) tied to the IETF’s AI Preferences working group. The draft’s own example looks like this:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
That says “no AI training” for the whole site except /press-kit/. The draft adds no new robots.txt grammar, and it is not an instruction about crawling. It is a work-in-progress draft, not a finished standard; as Search Engine Journal notes, it was submitted as an individual draft.
Video transcript
- Google, OpenAI and Microsoft now back one robots.txt signal for AI training. Should your site use it?
- Instead of separate rules for each AI company, one robots.txt group can state your training preference for all of them.
- It isn't live yet. Microsoft starts February 1, 2027, Google says 2027, and OpenAI says in the coming months.
- Training means teaching future AI models. Grounding means your page is used, and often cited, in an AI answer.
- Grounding is how sites get cited in AI answers. Microsoft warns that opting out of grounding reduces brand visibility.
- For most businesses, no robots.txt change is needed today. Just know what your file says now.
- When support goes live, decide training and grounding separately, based on your goals.
- Want help with AI search visibility? Visit netembark.com/contact-us, call 800-707-2697, or email info@netembark.com.
Which companies support it, and when?
Google, OpenAI and Microsoft each announced support on October 30, 2026, but none has turned it on yet. Microsoft starts February 1, 2027; Google says it aims to support it in 2027; OpenAI says “in the coming months.”
| Company | Announcement | What it said |
|---|---|---|
| Search Central blog, Oct 30, 2026 | Plans to support it in 2027; an opt-out via the token is read as a Google-Extended disallow unless Google-Extended is set | |
| OpenAI | OpenAI, Oct 30, 2026 | Will honor the signal; aims to implement “in the coming months”; existing crawler controls still apply |
| Microsoft | Bing blog, Oct 30, 2026 | Honors new AI Training and AI Grounding preferences from Feb 1, 2027; NoArchive/NoCache stop counting after Mar 1, 2027 |
Google says that when a site opts out with the token and doesn’t specify Google-Extended, “Google plans to interpret that as if the publisher had disallowed Google-Extended.” All three say their existing controls keep working.
What's the difference between AI training and grounding?
Training uses your content to teach future AI models. Grounding uses your content to answer a question right now, often with a link or citation back to your page. They are separate choices.

The draft’s training definition is “Using an asset to modify the learned parameters of a generative AI model.” Microsoft defines grounding as using content “as input to help inform AI-generated responses,” and lets sites set it per page with Allow, Disallow or a new DisallowExtended value. Google notes that sites which allow Google-Extended “are eligible to appear in responses and receive clicks via grounding in Gemini Apps.”
Microsoft is direct about the trade-off: “If you select DisallowExtended or Disallow for grounding, you will see reduced brand visibility in AI experiences… Neither choice guarantees inclusion, traffic, or payment.”
Should your business website use the signal now?
Not yet. No provider reads it today, so adding it changes nothing for now. For most small and local businesses, the better move is to understand your current robots.txt and decide later, keeping training and grounding separate.

A few details matter when the time comes. A User-Agent: * group does not count as a training preference under the draft; the token has to be named. If there’s no token group, no preference has been expressed. And per Microsoft, the training setting doesn’t change eligibility for Bing search results. If your site currently uses NoArchive or NoCache, note Bing’s March 1, 2027 transition deadline.
Talk to Netembark about your site’s visibility in AI search → · Call (800) 707-2697
How does this affect being cited in AI answers?
The training signal itself isn’t about citations, but grounding is. For a business that wants calls and booked consultations, being cited in AI answers is usually the goal, so any future opt-out should be weighed against that visibility.
This is the old-agency versus AI-era shift. The old model treated robots.txt as a crawl file and SEO as rankings. The AI-era model also asks where your pages get used and cited: Google AI Mode, Gemini, Copilot and ChatGPT. A fast, clear site helps either way (see our web design guide).
How does Netembark help with AI search visibility?
Netembark tracks changes like this one, reviews each client’s robots.txt and AI-crawler settings, and builds answer-first content designed to be cited in AI answers, then ties it to paid media measured in calls, forms and booked consultations.
- SEO and AEO: search and answer-engine optimization with answer-first pages, FAQ schema and clear H2 questions.
- Web design: websites built to load fast and turn visits into leads.
- Paid media: Google Ads including Performance Max and AI Max, Microsoft Ads (Microsoft Advertising Partner), Meta (paid social; Meta Business Partner) and ChatGPT ads.
- Real conversions: on average, Netembark’s paid campaigns generate a 12–16% conversion rate (12–18% for medical clients), counting calls, forms and booked consultations. That’s an average, not a guarantee. Campaigns go live 2–4 days after the insertion order is signed.
Netembark is a verified Google Partner (Google Premier Partner 2016–2023), headquartered at 42 E Main St, Suite 202, Freehold, NJ 07728, with offices in New York and Florida. Learn more about Netembark.
Related: What an AI-Era Ad Agency Actually Does Differently, ChatGPT Ads vs. Google Search Ads: What the First Revenue Study Found and Does Performance Max Still Need a Brand Search Campaign?.
Frequently asked questions
What is the __usage_ai-training token?
A shared robots.txt policy token from an IETF Internet-Draft that lets a site state one AI-training preference for every provider that honors it.
Do Google, OpenAI and Microsoft support it?
They announced support on October 30, 2026, but none has implemented it yet. Microsoft starts February 1, 2027, Google aims for 2027, and OpenAI says in the coming months.
Is it an official internet standard?
No. It's a work-in-progress IETF Internet-Draft, submitted as an individual draft and tied to the AI Preferences working group.
What is the difference between AI training and grounding?
Training uses content to teach future AI models. Grounding uses content to answer a question now, often with a link or citation to your page.
Should I add the token to my robots.txt now?
There's no need yet, because no provider reads it today. Review your current robots.txt and decide once support goes live, treating training and grounding separately.
Does User-agent: * cover AI training?
No. Under the draft, a wildcard group does not match the training token. It has to be named on its own.
Will opting out of AI training remove my site from search?
Google and Microsoft say their training controls don't change search results eligibility. Opting out of grounding is different and can reduce visibility in AI answers.
Can Netembark review my site's AI settings?
Yes. Netembark reviews robots.txt and AI-crawler settings as part of its SEO and AEO work.
Want your site cited in AI answers, not just crawled? Talk to Netembark, a verified Google Partner (Google Premier Partner 2016–2023). Contact Netembark, email info@netembark.com, or call (800) 707-2697.
Sources (read November 3, 2026): Google Search Central, AI training control announcement (October 30, 2026); OpenAI, Making it easier for websites to express AI training preferences (October 30, 2026); Microsoft Bing, Your content, your choice (October 30, 2026); IETF, draft-kelley-aipref-training-preference-signal-00; Search Engine Journal, coverage of the shared opt-out.
Netembark LLC · Verified Google Partner (Google Premier Partner 2016–2023) · 42 E Main St, Suite 202, Freehold, NJ 07728 · (800) 707-2697 · info@netembark.com · @netembark
Want help with your Google Ads?
Book a free strategy call with our verified Google Partner team. We’ll review your account and show you where the leads are.



