Jump to content

Recommended Posts

This is a bit of a strange one, I have recently been using Google Chrome Lighthouse v3 and noticed that under the SEO check it reports that the ‘Page is blocked from indexing’The report also shows that the Blocking Directive Source is /robots.txt
robots.txt

User-agent: *
Disallow: 
Sitemap: http://www.mywebsite.co.uk/sitemap.xml

 

I have removed the following from the webpages, as after a bit of reading I found that these settings are not really needed, unless a webpage is to be not indexed or not followed.

<meta name="robots" content="index, follow">

 

Looking through Google Search Console, it shows that the webpages are being indexed, and all the pages can be viewed using site:www.mywebsite.co.uk

What’s going on, any ideas what is causing the Lighthouse report to show the ‘Page is blocked from indexing’.

Share this post


Link to post
Share on other sites

Is that the entire contents of your robots.txt?

If so remove the first two lines (they aren't needed)

If not, move the sitemap line to the very end of your robots.txt

Share this post


Link to post
Share on other sites

At the moment that is all I have in the robots.txt file, I was planning on excluding a couple of pages once I have learnt a bit more on the best ways of doing that.

But in the meantime I am just confused why Google Chrome Lighthouse throws an ‘Page is blocked from indexing’ error, when clearly the pages are being indexed. I am wondering if it is a Lighthouse Report issue.

Share this post


Link to post
Share on other sites

It could be a bug. Because you have no filters in the 'Disallow' line it might be messing with their algo. So just delete the top two lines and see what happens.

However, I'd take the advice given by lighthouse with a big pinch of salt. I've just run a test on one of my pages and got a whole load of errors on a page that ranks #1 for 3 different keywords.

Share this post


Link to post
Share on other sites

I removed the top two lines, waited a while and then ran the Lighthouse report again, it still shows the ‘Page is blocked from indexing’ error. Being that removing the two lines did not make any difference, I have put them back in.

Out of interest, I ran the Lighthouse report on half a dozen big company websites, and even on the following URL from the Google Chrome Lighthouse page. Guess what… they all show the same error in the report.

Tools for Web Developers - Page is blocked from indexing https://developers.google.com/web/tools/lighthouse/audits/indexing

It does look like it could be some kind of bug in their report causing this error.

Share this post


Link to post
Share on other sites

Like I said, take the report with a big pinch of salt. It told me I needed to minimise a file that was already minimised.

Most technical SEO suggestions can be ignored.

Share this post


Link to post
Share on other sites

You don't need Disallow as it's set to nothing, as @fisicx says could easily be picked up as root "/" by mistake, as adomin.com and adomin.com/ both use root "/" as their uri. You could also test the robots with GSC and the robots.txt Tester.

Share this post


Link to post
Share on other sites

I have tested the robots.txt with Google Search Console, it passes the test with no Errors and no Warnings.

Share this post


Link to post
Share on other sites

SOLVED - The Google Chrome Lighthouse report has been updated and now works fine. ?

Edited by GrahamUK33

Share this post


Link to post
Share on other sites

You don't need Disallow as it's set to nothing, and you have add (Allow: / ) which instructs all the search engines to index and crawl your website regardless of the bot they are using to crawl.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • By BrowserBugs
      Hi Gang,
      After some consideration reading posts on here I figured a good basic list of resources would be good. Not sure if a mod will lock this so it can be updated at a later date, might be nice to stop the "what is this SEO" type posts but any other peeps using with GOOD tools feel free to chip in.
      The Beginners Guide to SEO - Moz
      Read this! And before asking anything, read this! Seriously, read it; covers everything from how search engines work to tracking success. 90% of your questions will be answered before asking anything.
      Search Engine Optimization (SEO) Starter Guide - Google
      PageSpeed Insights - Google
      Yes, site speed is a factor, no quantity of keywords will fix these, this is 'SEO'.
      Microdata - Schema
      Mixed debate, personal favourite of mine, if nothing else have cool breadcrumbs under your listing.
      Structured Data Testing Tool - Google
      See what Google picks up from your structured data and clarify errors.
      HTTP / HTTPS Header Check
      A great way to see if the supposed 301 is real or if you're giving 302s instead.
      Redirect Checker
      Similar but also checks the number of redirects.
      Learn SEO - Moz
      Freaking great set of resources. If you're still stuffing keywords into titles and descriptions then you need to start here!
      Canonical vs NoIndex - Moz
      Yup, another Moz post but before thinking Canonical is a silver bullet read this.
      XML Sitemaps: The Most Misunderstood Tool in the SEO's Toolbox - Michael Cottam, Moz
      Worth a read, especially the consistency section.
      News & Rumours
      Google Algorithm Change History - Moz
      SEMrush Sensor
      Google PageRank & Algorithm Updates - Search Engine Roundtable
      Paid Tools
      SEMrush
      Moz Pro
      Advice
      Link Spam In Forum Comments Work Very Badly (Thanks @Rich C)
    • By VWeb
      I subscribe to this guy's email list. According to him, Google will begin using the mobile version of a page for search engine results, even if the search is performed on a desktop.
      https://backlinko.com/mobile-seo-guide
      This could bring a lot of business my way, in terms of making sites mobile friendly. But I'm not a fan of Google dictating to the world how to create their websites. If this is true, then sites that are not mobile-friendly will disappear from search engine results pages.
      I use Bing myself, but obviously most people still use Google.
    • By Mario89
      I created my first website. Please tell me, what can I change and improve?
      http://mario89.smarthost.pl/
    • By Robert Peet
      Hi everyone,
       
      Apologies if this is in the wrong place of the forum / has been discussed 10 millions before - I'm a web design noob so not sure what to search for.
       
      A few days ago my site was struck off google for the search term 'Manchester Wedding Photographer'. A few days before that a new 'domain address'' appeared in search console (top address in the attached image). A coincidence? I have redirected this to my usual address however it has so far made little difference. I also recently added a few more pages to my site however the content and metatags etc are unique so I doubt this is what is penalising me. 
      I'm currently using Wix, which I know is universally despised, but I did at least have some presence on google until recently (varied between pages 8-10).  I plan to move to wordpress, it's just finding the time. If I could rectify this problem in the meantime, that would be great.
      I'd welcome all advice.
      Many thanks
      Rob
      www.robertpeetphotography.co.uk

    • By Mazahar Khan
      Hello mates, In my website meta title is not caching by google. Anybody know what is this. my whole website is crawled and indexed but meta title of one page is not crawled.

      Thanks in advance.
  • Recently Browsing

    No registered users viewing this page.

  • Member Statistics

    • Total Members
      58,416
    • Most Online
      4,970

    Newest Member
    angelsphone09
    Joined
  • Forum Statistics

    • Total Topics
      65,740
    • Total Posts
      452,901
×