Visually allow or disallow paths, set crawl-delay, declare your Sitemap, and apply proven presets. Copy or download a valid robots.txt in seconds.
robots.txt controls crawling, not indexing. To keep a page out of Google's index, use a noindex meta tag on a crawlable page — not a Disallow rule.
A robots.txt file is a small plain-text file that lives at the root of your website and tells search engine crawlers which parts of your site they are allowed to request. It is the first file most well-behaved bots — Googlebot, Bingbot, and others — fetch before crawling. This free robots.txt generator gives you a visual builder so you never have to memorize the syntax or risk a typo that could deindex your site. Add rules with dropdowns, declare your sitemap, set an optional crawl-delay, and watch a valid file build itself on the right in real time.
Each rule has three parts: a directive (Allow or Disallow), a user-agent (which crawler the rule applies to, with * meaning all crawlers), and a path. A Disallow: /admin/ rule asks crawlers not to request anything under /admin/. An empty Disallow: means "nothing is disallowed" — the standard way to allow everything. The generator groups rules by user-agent automatically and always emits a syntactically correct file.
Disallow: / for all bots. Perfect for staging, development, or pre-launch sites you do not want crawled. Remove it before going live./wp-admin/ but explicitly allows /wp-admin/admin-ajax.php, the configuration WordPress and most SEO plugins recommend./cart/, /checkout/, internal search results, and faceted filter parameters that create near-infinite duplicate URLs.The single most common and most expensive mistake in technical SEO is confusing crawling with indexing. robots.txt blocks crawling, not indexing. If you Disallow a URL that is linked from other pages, Google may still index that URL and show it in search results — just without a description, because it was not allowed to read the content. Worse, if you block a page with robots.txt, Google can never see a noindex tag on that page, so the page can get stuck in the index indefinitely. The correct way to remove a page from search results is to allow crawling and add a noindex directive via a meta tag or the X-Robots-Tag HTTP header.
The file must be reachable at exactly https://yourdomain.com/robots.txt — the protocol root. A robots.txt placed in a subfolder is ignored entirely, and each subdomain (like blog.yourdomain.com) needs its own file. After you upload it, validate it with the robots.txt report and URL Inspection tool in Google Search Console to confirm your important pages are crawlable and the right pages are blocked. Declaring your Sitemap: URL inside robots.txt — which this tool does for you — is also a quick win, because crawlers read that line and discover your XML sitemap automatically. Once crawling is dialed in, optimize the snippets those pages show with our meta description generator.
A plain text file at the root of your site that tells crawlers which URLs they may request, using User-agent, Allow, Disallow, Crawl-delay and Sitemap directives from the Robots Exclusion Protocol.
No. It controls crawling, not indexing. A disallowed URL can still be indexed if linked elsewhere. Use a noindex meta tag on a crawlable page to keep it out of the index.
At the root of your domain (yourdomain.com/robots.txt). Files in subdirectories are ignored, and every subdomain needs its own robots.txt.
It disallows /wp-admin/ while allowing admin-ajax.php — the recommended setup for most WordPress sites. Add your sitemap URL on top.