If you see the "Blocked by robots.txt" issue in Google Search Console, it means Googlebot is unable to crawl certain pages on your website due to restrictions in your robots.txt file. This can affect your SEO performance, especially if important pages are blocked.
In this guide, you will learn:
- What causes this issue
- How to check which pages are blocked
- Step-by-step method to fix it
What Causes the "Blocked by robots.txt" Issue?
The robots.txt file is used to control how search engines crawl your site. If a page is blocked by robots.txt, Google won’t index it.
Common reasons for this issue include:
- Blocking search pages: Blogger often blocks label pages (/search/label/) to avoid duplicate content.
- Blocking unnecessary pages: Pages like /feeds/, /admin/, or login pages are often blocked for security and SEO purposes.
- Incorrect robots.txt settings: If important pages are mistakenly blocked, they won’t appear in Google search results.
Example of a restricted URL:
https://www.creatoryt.in/search/label/Blogging
How to Check if Your Pages Are Blocked?
Method 1: Check in Google Search Console
- Open Google Search Console
- Go to Indexing > Pages
- Find the section "Why pages aren’t indexed"
- Look for "Blocked by robots.txt"
- Click on the affected pages to see the exact URLs
Method 2: Use Google’s Robots.txt Tester
- Go to Google Search Console > Settings > Robots.txt Tester
- Enter a blocked URL and click Test
- If you see a red warning, it means the page is blocked.
How to Fix the "Blocked by robots.txt" Issue?
Step 1: Update Your Robots.txt File in Blogger
- Go to Blogger Dashboard
- Click on Settings
- Scroll to Crawlers and Indexing
- Enable Custom robots.txt
- Copy and paste the new robots.txt file
Updated Robots.txt Code (To Allow Label Pages to Be Indexed)
User-agent: * Allow: /search Allow: / Sitemap: https://www.creatoryt.in/sitemap.xml
Old Robots.txt Code (Blocking Label Pages)
User-agent: * Disallow: /search Allow: / Sitemap: https://www.creatoryt.in/sitemap.xml
Change "Disallow: /search" to "Allow: /search" if you want label pages indexed.
Step 2: Validate Fix in Google Search Console
After updating the robots.txt file, you need to tell Google to recheck your site.
- Open Google Search Console
- Go to Indexing > Pages
- Click on the "Blocked by robots.txt" error
- Click Validate Fix
- Wait for Google to recrawl your site (may take 1-2 weeks)
Once Google updates the indexing, your issue should be resolved!
Should You Fix This Issue?
Not all "Blocked by robots.txt" issues need to be fixed.
- If your blog posts or important pages are blocked → Fix it immediately.
- If label pages (/search/label/) or feeds (/feeds/) are blocked → You can ignore it.
If important content is blocked, update robots.txt. Otherwise, this warning can be ignored for label or search pages.
Frequently Asked Questions (FAQs)
Q1. What happens if I don’t fix the "Blocked by robots.txt" issue?
If important pages are blocked, they won’t appear in Google search results, reducing your organic traffic.
Q2. How long does it take for Google to update the changes?
Google usually updates indexing within 1-2 weeks after you submit a fix in Search Console.
Q3. Should I allow label pages to be indexed?
Allowing label pages can increase traffic, but it may also create duplicate content issues.
Final Thoughts
The "Blocked by robots.txt" issue in Google Search Console is common, but it doesn’t always require fixing. If essential pages are blocked, update the robots.txt file. Otherwise, you can safely ignore warnings for label and feed pages.
Need more help? Drop your questions in the comments! Did this guide help you? Share it with others!
😊😊😊