Block GPTBot.
Keep ChatGPT search separate.
GPTBot and OAI-SearchBot have independent robots.txt controls. You can restrict the training crawler while leaving search crawling available. Allowing search does not guarantee inclusion or a citation. OpenAI crawler documentation.
Choose the control for your goal
| Agent | Purpose | What to review |
|---|---|---|
GPTBot | Content that may be used for foundation-model training | Disallow this crawler if you want to restrict that use. |
OAI-SearchBot | Discovery for ChatGPT search | Keep access to the public pages you want search to discover. |
ChatGPT-User | Some user-requested page visits | It is separate from automatic search crawling; robots.txt may not apply. |
These distinctions come from OpenAI’s published agent descriptions.
A minimal training-only restriction
If your existing rules already allow the search pages you intend to expose, this group adds a GPTBot restriction without changing those search rules:
User-agent: GPTBot Disallow: /
Review any GPTBot group already in your file. Merge deliberately instead of appending conflicting copies. This example does not undo a search block that already exists.
Build your crawler preferencesCheck the file before publishing
- Back up the current file and note the paths you already restrict.
- Review the groups applying to GPTBot and OAI-SearchBot. When adding a specific bot group, preserve relevant path restrictions inside it; a general wildcard group is not a substitute.
- Publish the reviewed file at your domain’s
/robots.txtand open that URL to check the actual response. Keep any sitemap line only if its URL exists. - Check your server or firewall rules too. A robots.txt preference cannot overcome a blocked network request. Use the provider’s published IP information when investigating requests.
Google documents how user-agent groups are selected and the limits of robots.txt. Authentication is the appropriate control for private content.
Then review the page itself
Crawler access is one decision. The next task is making your public page useful to its reader. Our free checklist reviews answer placement, headings and source links in pasted content. If you want help drafting changes, Content Studio provides an editable rewrite and FAQ with two trial reviews.
Review facts and claims before publishing. Neither tool measures actual search visibility.