This is a bit of a strange one, I have recently been using Google Chrome Lighthouse v3 and noticed that under the SEO check it reports that the ‘Page is blocked from indexing’The report also shows that the Blocking Directive Source is /robots.txt
robots.txt

User-agent: *
Disallow: 
Sitemap: http://www.mywebsite.co.uk/sitemap.xml

 

I have removed the following from the webpages, as after a bit of reading I found that these settings are not really needed, unless a webpage is to be not indexed or not followed.

<meta name="robots" content="index, follow">

 

Looking through Google Search Console, it shows that the webpages are being indexed, and all the pages can be viewed using site:www.mywebsite.co.uk

What’s going on, any ideas what is causing the Lighthouse report to show the ‘Page is blocked from indexing’.

Is that the entire contents of your robots.txt?

If so remove the first two lines (they aren't needed)

If not, move the sitemap line to the very end of your robots.txt

At the moment that is all I have in the robots.txt file, I was planning on excluding a couple of pages once I have learnt a bit more on the best ways of doing that.

But in the meantime I am just confused why Google Chrome Lighthouse throws an ‘Page is blocked from indexing’ error, when clearly the pages are being indexed. I am wondering if it is a Lighthouse Report issue.

It could be a bug. Because you have no filters in the 'Disallow' line it might be messing with their algo. So just delete the top two lines and see what happens.

However, I'd take the advice given by lighthouse with a big pinch of salt. I've just run a test on one of my pages and got a whole load of errors on a page that ranks #1 for 3 different keywords.

I removed the top two lines, waited a while and then ran the Lighthouse report again, it still shows the ‘Page is blocked from indexing’ error. Being that removing the two lines did not make any difference, I have put them back in.

Out of interest, I ran the Lighthouse report on half a dozen big company websites, and even on the following URL from the Google Chrome Lighthouse page. Guess what… they all show the same error in the report.

Tools for Web Developers - Page is blocked from indexing https://developers.google.com/web/tools/lighthouse/audits/indexing

It does look like it could be some kind of bug in their report causing this error.

Like I said, take the report with a big pinch of salt. It told me I needed to minimise a file that was already minimised.

Most technical SEO suggestions can be ignored.

You don't need Disallow as it's set to nothing, as @fisicx says could easily be picked up as root "/" by mistake, as adomin.com and adomin.com/ both use root "/" as their uri. You could also test the robots with GSC and the robots.txt Tester.

I have tested the robots.txt with Google Search Console, it passes the test with no Errors and no Warnings.

There does look to be an issue with Lightroom v3 currently.

 

SEO blocked from indexing fails when robots.txt specifies user-agents #5329

https://github.com/GoogleChrome/lighthouse/issues/5329

SOLVED - The Google Chrome Lighthouse report has been updated and now works fine. ?

Edited May 25, 20188 yr by GrahamUK33

You don't need Disallow as it's set to nothing, and you have add (Allow: / ) which instructs all the search engines to index and crawl your website regardless of the bot they are using to crawl.