Free Robots.txt Checker & Validator – Test Your Robots.txt File
Use this robots.txt checker to fetch and validate any site's robots.txt, find syntax errors, and test whether Googlebot, Bingbot and AI crawlers can access a URL — using the same matching rules Google uses.
Enter any page or domain — we check /robots.txt at the site root. Or paste a file below.
Robots.txt Validator
Edit or paste your robots.txt below. It is validated instantly as you type.
Validation Report
Sitemaps listed
Rules by crawler
What Is a Robots.txt Checker?
A robots.txt checker reads the robots.txt file of any website and shows you the rules it gives to search engine bots. It shows which bots the rules apply to, which folders and pages are allowed or blocked, and where the sitemap is listed. It also checks the file for mistakes. With the URL tester, you can enter any page path and instantly see whether Googlebot, Bingbot or another crawler is allowed to visit it. It’s the fastest way to make sure you’re not accidentally blocking pages you want to rank.
What Is a Robots.txt File?
A robots.txt file is a small text file at the root of your website, at yourdomain.com/robots.txt. It tells search engine bots which parts of your site they can and can’t crawl. It’s usually used to keep bots away from pages that don’t need to be in search, like admin areas, cart and checkout pages, internal search results or filtered URLs. That way, bots spend their time on the pages that matter.
A simple robots.txt file looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/sitemap.xml
Why Checking Your Robots.txt Matters
One wrong line in robots.txt can stop Google from crawling your entire website. It happens more often than you’d think. A developer forgets to remove Disallow: / after launching a site, or a plugin update blocks your CSS and JavaScript files. Your rankings drop and traffic disappears, and many site owners don’t notice for weeks. Regular checks catch these problems early, before they cost you customers.
- Avoid blocking important pages: Make sure your key service, product and blog pages are open to search engines.
- Let Google see your pages properly: Blocking CSS or JavaScript files can stop Google from rendering your pages correctly.
- Use your crawl budget wisely: On large sites, blocking low-value URLs helps bots focus on pages that bring traffic.
- Help Google find your sitemap: A sitemap line in robots.txt makes it easy for crawlers to discover all your pages.
How to Use the Robots.txt Checker
- Enter a website URL and click Fetch Robots.txt, or paste your own robots.txt rules into the box.
- Click Validate Robots.txt to check the file for errors and warnings.
- To test a specific page, choose a bot, such as Googlebot or Bingbot, and enter the page path.
- Click Test URL to see whether that page is allowed or blocked.
- Fix any problems in your robots.txt file and test again.
How to Find the Robots.txt File of Any Website
Every website keeps its robots.txt file in the same place, at the root of the domain:
https://www.example.com/robots.txt
Just replace example.com with any domain. Each subdomain has its own file, so blog.example.com/robots.txt is separate from www.example.com/robots.txt. If the page shows a “404 not found” error, the site has no robots.txt file. In that case, search engines assume they can crawl everything.
Robots.txt Rules Explained
- User-agent: The bot the rules apply to.
*means all bots, whileGooglebotmeans only Google’s crawler. - Disallow: A path the bot should not crawl.
Disallow: /blocks the whole site. - Allow: A path the bot can crawl, even inside a blocked folder.
- Sitemap: The full URL of your XML sitemap.
*(wildcard): Matches any sequence of characters, such asDisallow: /*?sort=.$(end of URL): Matches the end of a URL, such asDisallow: /*.pdf$.
Common Robots.txt Mistakes
- Blocking the whole site:
Disallow: /underUser-agent: *stops all bots from crawling every page. It’s often left behind after a site goes live. - Using robots.txt to hide pages from Google: Blocked pages can still appear in search results if other sites link to them. To remove a page from Google, use a noindex tag instead. You can add one with our Meta Tag Generator.
- Blocking a page that has a noindex tag: If robots.txt blocks the page, Google can’t see the noindex tag, so the page may stay indexed.
- Blocking CSS and JavaScript files: Google needs these files to see your pages the way visitors do.
- Adding noindex rules in robots.txt: Google stopped supporting noindex in robots.txt in 2019.
- Relying on crawl-delay for Google: Google ignores the crawl-delay rule, although Bing still follows it.
- Wrong letter case: Paths are case-sensitive, so
/Blog/and/blog/are treated as different paths. - Missing sitemap line: Add your sitemap URL so crawlers can find it easily. Check it’s working with our Sitemap Checker.
Robots.txt Best Practices
Do:
- Keep the file simple, and block only what truly doesn’t need crawling.
- Add your full sitemap URL at the bottom of the file.
- Test important pages with the URL tester after every change.
- Check your robots.txt after launching a new site, changing themes or migrating to a new domain.
- Decide whether you want AI crawlers like GPTBot or Google-Extended to access your content, and set rules for them if needed.
Don’t:
- Use robots.txt to hide private or sensitive pages. The file is public, and anyone can read it. Use password protection instead.
- Block pages you want to rank, or the resources those pages need to load.
- Use robots.txt to fix duplicate content. Use canonical tags instead, and check them with our Canonical Tag Checker.
More Free Technical SEO Tools
- Find and validate your XML sitemap with the Sitemap Checker.
- Fix duplicate URL issues with the Canonical URL Generator.
- Set up international pages with the Hreflang Generator.
- Explore our complete collection of free SEO tools.
Pages Not Showing Up on Google?
If your pages aren’t getting crawled or indexed, robots.txt is a good place to start, but it’s rarely the only issue. Crawl blocks, noindex tags, canonical errors and sitemap problems often work together to keep pages out of search. Jagdish Prajapat is a well-known AI SEO expert in India with 12+ years of experience and 1400+ clients worldwide. He finds and fixes the technical issues stopping your website from ranking, so you get steady organic traffic without spending a single rupee on paid ads. Book a free consultation →