A robots.txt file tells search engines which parts of your website they can and cannot crawl
A robots.txt file is a plain text document that lives in your website's root directory — the top level where your domain points. It contains instructions for search engine crawlers (the automated programs that index your site) about which pages and files they should visit and which they should skip. Search engines like Google, Bing, and others check for this file first when they arrive at your domain.
The file does not prevent people from viewing your pages in a browser. It only tells automated crawlers what to do. If you want to truly hide content from the public, you need password protection or other security measures instead. Robots.txt is about managing crawler traffic and telling search engines where not to waste time.
You do not need a robots.txt file for your site to work or to be indexed. But if you have pages you want crawlers to skip — like admin areas, duplicate content, or pages that use a lot of server resources — this file gives you a straightforward way to say so.
Key Takeaways
- A robots.txt file sits in your root directory (example.com/robots.txt) and contains plain text rules with no special formatting.
- The file uses two main commands: User-agent (which crawler the rule applies to) and Disallow (which paths to block).
- You can block all crawlers from a path, block only certain crawlers, or allow crawlers to visit everything by leaving the file empty or minimal.
- Search engines are not required to follow robots.txt rules, but the major ones (Google, Bing) do respect them in practice.
- A robots.txt file does not hide pages from people — it only tells crawlers not to visit, so the pages remain publicly viewable.
The basic structure and syntax
A robots.txt file uses a straightforward format: each rule has a User-agent line (which crawler it applies to) and one or more Disallow lines (which paths to block). Here is what a minimal file looks like:
User-agent: * Disallow: /admin/ Disallow: /private/
The asterisk (*) after User-agent means "all crawlers." The Disallow lines list the paths you want blocked. Each path starts with a forward slash and matches anything that begins with that text. So Disallow: /admin/ blocks /admin/, /admin/users, /admin/settings, and anything else under that folder.
Blank lines separate rules and make the file easier to read, but they are not required. You can also add comments by starting a line with a hash mark (#). Comments help you remember why you blocked something:
# Block the admin area User-agent: * Disallow: /admin/ # Block temporary test pages User-agent: * Disallow: /test/
How to block specific crawlers instead of all crawlers
If you want to block only certain search engines or crawlers, replace the asterisk with the crawler's name. Google's crawler is called Googlebot, Bing's is Bingbot, and so on. This rule blocks only Googlebot from your /private/ folder while allowing other crawlers to visit:
User-agent: Googlebot Disallow: /private/
You can list multiple User-agent rules in the same file. Each rule applies only to the crawler named. If you want the same rule to explore to multiple crawlers, you can repeat the User-agent line:
User-agent: Googlebot User-agent: Bingbot Disallow: /temp/
Both Googlebot and Bingbot will now skip the /temp/ folder. Any crawler not named in a User-agent line is not affected by that rule. If you do not have a rule for a specific crawler, it can crawl everything (unless you have a rule for the asterisk, which applies to all crawlers).
Using Allow to override a Disallow rule
Sometimes you want to block a broad folder but allow specific pages within it. Use the Allow command to create exceptions. This blocks everything under /products/ except the /products/sale/ folder:
User-agent: * Disallow: /products/ Allow: /products/sale/
The order matters. Crawlers read the rules from top to bottom and stop at the first match. So if you put Allow before Disallow, the Allow rule will match first and the Disallow will never explore. Always put your more specific rules (like Allow) after your broader ones (like Disallow).
The Allow command is useful when you have a large section you want to block but a few important pages within it that you want indexed. Without Allow, you would have to list every single path you want blocked individually, which gets messy fast.
Adding a Sitemap location
You can also tell crawlers where to find your sitemap — a file that lists all your pages. Add a Sitemap line at the end of your robots.txt file:
User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
The Sitemap line is optional and does not affect crawling rules. It is just a convenience — crawlers can find your sitemap without it, but listing it here makes sure they know about it. You can include multiple Sitemap lines if you have more than one sitemap file.
Common mistakes and what not to do
A frequent mistake is forgetting the leading slash. Disallow: admin does not work the same way as Disallow: /admin/. The slash matters because it anchors the path to the root of your domain. Without it, the rule may not match what you expect.
Another mistake is trying to block file types instead of paths. You cannot write Disallow: *.pdf to block all PDF files. Robots.txt does not support wildcards in that way. If you need to block PDFs, you have to block the folders they live in or use a different method like a meta tag in your HTML.
Do not use robots.txt to hide sensitive information. If a page contains passwords, private data, or anything you do not want public, use actual security (password protection, authentication, encryption). Robots.txt is just a polite request — it does not prevent someone from typing the URL directly into their browser or from a crawler that ignores the rules.
Testing your robots.txt file
Google Search Console (the free tool Google provides for website owners) includes a robots.txt tester. Upload your file to your website, then go to Search Console, find the robots.txt tester in the tools menu, and paste in a URL from your site. The tool will tell you whether that URL is blocked or allowed according to your rules.
You can also check your file manually by visiting example.com/robots.txt in your browser. You should see the plain text of your file. If you see a 404 error, the file is not in the right place (it must be in the root directory, not in a subfolder).
After you make changes, crawlers do not check your robots.txt file constantly. It can take days or weeks for search engines to re-crawl your site and notice the changes. If you need to block something when ready, use Search Console to request removal or use a meta tag in your HTML instead.
When you do not need a robots.txt file
If your website is small, has no duplicate content, and does not have admin areas or private folders, you probably do not need a robots.txt file at all. Search engines will crawl your public pages just fine without one. A robots.txt file is most useful for large sites with many pages, sites that have a lot of duplicate content, or sites where crawlers might waste resources on pages you do not want indexed.
If you do create a file, keep it straightforward. A file with just a few rules is easier to maintain and less likely to have mistakes. You can always add more rules later if you find that crawlers are visiting pages you want them to skip.
Frequently Asked Questions
Does robots.txt prevent people from seeing my pages?
No. Robots.txt only tells crawlers not to visit. Anyone can still type the URL into their browser and view the page. If you need to hide content from people, use password protection or other security measures. Robots.txt is not a security tool.
What happens if I do not have a robots.txt file?
Search engines assume they can crawl everything on your site. This is fine for most websites. You only need a robots.txt file if you want to block certain paths or manage crawler traffic. The absence of the file does not hurt your site.
Can I block a single page instead of a whole folder?
Yes. Use the full path to the page: Disallow: /about/contact.html blocks only that one page. But if you have many individual pages to block, it is usually cleaner to put them in a folder and block the folder instead.
Do all search engines follow robots.txt rules?
Most major ones (Google, Bing, Yahoo) do. Smaller or less reputable crawlers may ignore the file. Robots.txt is a standard, but not a law. If you need to block a crawler that ignores robots.txt, you can block its IP address at the server level or use other methods.
Can I use robots.txt to improve my search ranking?
Indirectly, yes. By blocking duplicate pages or low-quality content, you focus crawler resources on the pages you want indexed, which can help. But robots.txt itself does not boost rankings. It is a tool for managing what gets crawled, not for improving how pages rank.