WorkspaceAI Crawler Shield
Early accessG
All tools
Crawler controlsFree tool

AI Crawler Shield

Set your preferences. Keep search and model training separate.

Your crawler preferences

https://

Used in your export. We won’t visit your website.

Search discovery

Restrict training & related use

Check to restrict

Generated locally. No account or API key.

Your rules

A clear policy starts here.

Enter your domain, choose your preferences,
and generate rules you can review and export.

Merge these rules with your existing robots.txt. Preserve private-path restrictions and verify the sitemap URL before publishing.

Search access and training are different choices.

This generator creates preferences for documented crawlers. Google-Extended also covers certain Gemini grounding uses, while Googlebot controls Google Search crawling. Robots.txt asks cooperating crawlers to follow your rules; it does not enforce privacy or guarantee a search citation.

Understand the crawler controls

Put your rules to work

  1. Back up your current robots.txt.
  2. Merge the generated crawler groups and review existing restrictions.
  3. Publish at your site’s root, then verify the file and crawl access.