What a robots.txt file does and why you need one

A robots.txt file is a text file you place in your website's root directory that tells search engine crawlers which pages they can and cannot visit. Search engines like Google, Bing, and others send automated bots to crawl websites and index their content. Without a robots.txt file, these bots will crawl everything. With one, you control what gets crawled — which saves server resources, keeps sensitive pages out of search results, and prevents crawlers from wasting time on pages you don't want indexed.

The file is named exactly robots.txt (lowercase, no spaces) and must live at the root of your domain — for example, yoursite.com/robots.txt. Search engines check for this file first before crawling anything else on your site. If the file doesn't exist, crawlers assume they can visit every page. If it exists but is misconfigured, you might accidentally block crawlers from pages you want indexed, or fail to block pages you want hidden.

Key Takeaways

  • A robots.txt file must be named exactly robots.txt, saved as plain text (not .doc or .rtf), and placed in your website's root directory at the domain level.
  • The file uses straightforward syntax: User-agent: specifies which crawler the rule applies to, and Disallow: lists the paths crawlers cannot visit.
  • Use User-agent: * to write rules that explore to all crawlers, or name specific crawlers like Googlebot if you want different rules for different search engines.
  • Test your robots.txt file using Google Search Console's robots.txt tester before uploading it, because a broken file can hide your entire site from search results.
  • Common mistakes include blocking pages you want indexed, using absolute URLs instead of paths, and forgetting to upload the file to the root directory.

The basic syntax and structure of robots.txt

A robots.txt file uses a straightforward format with two main commands. User-agent: specifies which crawler the rule applies to. Disallow: lists the path or paths that crawler cannot visit. Each rule set starts with a User-agent line, followed by one or more Disallow lines.

Here is a basic example that blocks all crawlers from a folder called /admin:

User-agent: *Disallow: /admin/

The asterisk (*) means "all crawlers." If you wanted to block only Googlebot, you would write User-agent: Googlebot instead. You can have multiple User-agent blocks in one file, each with its own rules. Paths always start with a forward slash and are case-sensitive — /Admin/ is different from /admin/.

If you want to allow a crawler to visit everything, you can either leave the file blank or write a rule with no Disallow paths. An empty Disallow line means "nothing is blocked for this user-agent." You can also use Allow: to explicitly permit access to a subfolder within a blocked directory, though this is less common and not supported by all crawlers.

How to write rules for common scenarios

Most websites block crawlers from a few standard places: admin panels, private user accounts, duplicate content, and temporary pages. Here are the patterns you will use most often.

To block an entire folder, write the folder path with a trailing slash: Disallow: /private/ blocks everything inside /private and its subfolders. To block a single file, write the full path: Disallow: /contact-form.php blocks only that file. To block all files with a certain extension, use a wildcard: Disallow: /*.pdf blocks all PDF files on your site.

If you have a folder you want to block except for one subfolder inside it, use two rules: first block the parent folder, then use Allow: to open the exception. For example, to block /uploads/ but allow /uploads/public/:

User-agent: *Disallow: /uploads/Allow: /uploads/public/

To block all crawlers except one specific search engine, write a rule for all crawlers first, then a separate rule for the one you want to allow. For example, to block everyone except Googlebot:

User-agent: *Disallow: /User-agent: GooglebotDisallow:

Note the blank line between rule sets — this separates them clearly. The second block allows Googlebot to crawl everything because the Disallow line is empty.

Creating and uploading the robots.txt file

Open a plain text editor — Notepad on Windows, TextEdit on Mac, or any code editor like Visual Studio Code. Do not use Microsoft Word or Google Docs, because they add formatting that will break the file. Write your rules using the syntax described above, then save the file as robots.txt (not robots.txt.txt or robots.doc).

Upload the file to your web server's root directory. If you use a hosting provider with a file manager (like cPanel or Plesk), navigate to the public_html or www folder — this is your root directory. Upload robots.txt there. If you use an FTP client like FileZilla, connect to your server and upload the file to the same location. If your site runs on a platform like WordPress, Shopify, or Wix, check your platform's documentation — some have built-in tools to create robots.txt without manual uploading.

After uploading, verify the file is accessible by visiting yoursite.com/robots.txt in your browser. You should see the contents of your file displayed as plain text. If you see a 404 error, the file is not in the root directory or has the wrong name.

Testing your robots.txt file before going live

Before you upload your robots.txt file to your live website, test it using Google Search Console. Log into Search Console, select your property, go to the Tools menu, and click robots.txt Tester. Paste your robots.txt content into the editor, then type a URL path from your site (for example, /admin/ or /private/contact.php) and click Test. The tool will tell you whether that path is allowed or blocked according to your rules.

Test several paths — some you want blocked and some you want allowed — to make sure the rules work as intended. Pay special attention to paths you want indexed, because a mistake here could hide your content from search results. If the tester shows a path is blocked when you wanted it allowed, edit your rules, test again, and repeat until everything is correct.

Google Search Console also shows you if there are syntax errors in your file. If the tester reports an error, fix it before uploading. Common errors include misspelled commands (like "Disalow" instead of "Disallow"), paths without a leading slash, or extra spaces in the wrong places.

Common mistakes and how to avoid them

The most damaging mistake is blocking pages you actually want indexed. Many developers accidentally write Disallow: / (which blocks everything) and forget to remove it, or they block a folder that contains important content. Always test before uploading, and review your rules carefully to make sure you are only blocking what you intend to block.

Another common error is using absolute URLs instead of paths. Robots.txt only understands paths relative to your domain root — Disallow: /admin/ is correct, but Disallow: https://yoursite.com/admin/ will not work. Similarly, do not include your domain name in any path.

Some developers forget that robots.txt is case-sensitive. Disallow: /Admin/ will not block /admin/ — the crawler will still visit the lowercase version. If your server is case-insensitive (which is common on Windows servers), this may not matter, but it is safer to assume it does and match the exact case of your actual folder names.

Finally, remember that robots.txt is a request, not a law. Malicious crawlers and scrapers often ignore it. If you have truly sensitive data, do not rely on robots.txt alone — use password protection, authentication, or server-level restrictions instead.

Frequently Asked Questions

Will robots.txt prevent my pages from being indexed if they are already in Google's index?

No. If a page is already indexed, adding it to robots.txt will not remove it from search results when ready. Google will eventually re-crawl your site, see the robots.txt rule, and stop crawling that page — but the old index entry may persist for weeks or months. To remove a page faster, use Google Search Console's removal tool or add a noindex meta tag to the page itself.

Can I use robots.txt to block visitors from accessing my site?

No. Robots.txt only controls search engine crawlers, not human visitors. Anyone can still visit any page on your site by typing the URL directly into their browser. To block human visitors, use password protection, authentication, or server-level restrictions like .htaccess files or firewall rules.

What happens if I do not have a robots.txt file?

Search engines will crawl your entire site and index all publicly accessible pages. This is fine for most websites. You only need a robots.txt file if you want to prevent crawlers from visiting certain pages or folders, or if you want to manage crawl budget on a very large site.

Can I have multiple robots.txt files on my site?

No. Search engines only look for one robots.txt file at the root of your domain. You cannot have separate robots.txt files for subfolders or subdomains. However, subdomains are treated as separate sites, so you can have a different robots.txt for each subdomain if needed.

Do I need to submit my robots.txt file to Google?

No. Google automatically checks for robots.txt every time it crawls your site. You do not need to manually submit it. However, you can view how Google sees your robots.txt file in Google Search Console, which is useful for debugging.