Everything You Need to Know About Robots.txt
The robots.txt file is one of the most critical files for your website's technical SEO. It acts as the "receptionist" of your website, instructing search engine bots like Googlebot on where they are allowed to go and where they are forbidden.
- Save Crawl Budget: Prevent bots from wasting time on useless pages like admin panels, scripts, or duplicate tags.
- Block Bad Bots: Use our advanced features to block Ahrefs, Semrush, and other marketing crawlers to save server resources.
- Protect Privacy: Keep sensitive directories (e.g.,
/wp-admin/) out of public search results.
Understanding the Syntax
A standard robots.txt file consists of "groups" of directives. Here are the most common terms you will see:
Disallow: /admin/ (Do not visit the admin folder)
Allow: /admin/public-image.jpg (Exception: Visit this specific file)
Disallow: /*.pdf$ (Block all PDF files)
3 Common Mistakes to Avoid
Many webmasters accidentally de-index their own sites by making small mistakes in this file. Be careful of the following:
- Blocking CSS/JS: Do not use
Disallow: /wp-includes/if they contain styles. Google needs these to render your page properly. - The Trailing Slash:
Disallow: /fishblocks everything starting with "fish".Disallow: /fish/only blocks the folder "fish". - Blocking Everything: Accidentally leaving
Disallow: /during development will block your entire site.
How to Upload Your Robots.txt File
- Click the "Download .txt" button above.
- Log in to your hosting Control Panel (cPanel) or use an FTP client.
- Upload the file to the root directory (public_html).
- Verify it by visiting:
https://yourdomain.com/robots.txt
Frequently Asked Questions
Why should I block Ahrefs or Semrush?
These are SEO tools, not search engines. They crawl your site to gather data for their software. Blocking them saves resources and does not affect your Google Ranking.
What is the "Allow" rule used for?
The "Allow" rule is used to grant access to a specific file inside a folder that you have blocked. For example, blocking `/wp-admin/` but allowing `/wp-admin/admin-ajax.php`.
Can I edit the file later?
Absolutely. It is just a text file. You can edit it manually via FTP or come back to this tool to generate a new optimized version anytime.