Block ChatGPT with robots.txt
A robots.txt
file "tells search engine crawlers which URLs the crawler can access on your site". Generally one would use this to tell a search engine which directories should be crawled, which ones shouldn't, or to completely block from crawling your site. The file lives at the root of your site. You can see mine here.
ChatGPT has two user agents that might attempt to crawl your site: GPTBot
and ChatGPT-User
. To disallow crawling from both of these, we can set our robots.txt
file like so:
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
Adding a robots.txt file to Eleventy
Using Eleventy's passthrough file copy we can create a robots.txt
file and then add the following to our Eleventy config to make sure the file is copied to your built site:
eleventyConfig.addPassthroughCopy('robots.txt')
And there we go. ChatGPT, in theory, should be blocked from crawling our site.