[roothost]

I always thought that this..

User-agent: *
Disallow: /blog/

..would block the search engines from indexing the page/folder.

[SniderDK]

roothost, on 31 August 2011 - 09:26 AM, said:

I always thought that this..

User-agent: *
Disallow: /blog/

..would block the search engines from indexing the page/folder.

it would request that spiders dont go there... its not a "force" as a crawler can decide to not check and or ignore.... although thats not what happens on the polite internet... theres not a technical limitation stopping them from ignoring the file


http://www.robotstxt.../robotstxt.html

[roothost]

SniderDK, on 31 August 2011 - 09:36 AM, said:

it would request that spiders dont go there... its not a "force" as a crawler can decide to not check and or ignore.... although thats not what happens on the polite internet... theres not a technical limitation stopping them from ignoring the file


http://www.robotstxt.../robotstxt.html

Ok, just wondered as I had blocked the /blog/ folder months (before I even added content to it) and noticed this morning that the big G had indexed /blog/ and /blog/news/.

[SniderDK]

Well that's abit rude of the gman...

They do have a tool in google webmaster tools to drop pages... and you could add meta tags... or check for googlebot in the useragent header and 404 the page... will get them to drop it fairly quick... obv don't do that if there's page history that could be saved

Thinking about it webmaster tools from google will tell you exactly what they think they should be doing in the diagnostics section....

[rallport]

I personally always use the meta noindex tag - I feel much more confident than using robots.txt

[roothost]

SniderDK, on 31 August 2011 - 10:11 AM, said:

Well that's abit rude of the gman...

They do have a tool in google webmaster tools to drop pages... and you could add meta tags... or check for googlebot in the useragent header and 404 the page... will get them to drop it fairly quick... obv don't do that if there's page history that could be saved

Thinking about it webmaster tools from google will tell you exactly what they think they should be doing in the diagnostics section....

Yh, I had to get some pages for another site dropped and the webmaster tools was pretty handy. It's not a major issue really, more of an annoyance.

rallport, on 01 September 2011 - 09:12 AM, said:

I personally always use the meta noindex tag - I feel much more confident than using robots.txt

I will certainly be doing that from now on.

[handatravel]

yes, it tells serach engine crawler not to crawl the blog folder

[roothost]

handatravel, on 13 September 2011 - 07:51 AM, said:

yes, it tells serach engine crawler not to crawl the blog folder

?

[oliviasmith]

roothost, on 31 August 2011 - 09:26 AM, said:

I always thought that this..

User-agent: *
Disallow: /blog/

..would block the search engines from indexing the page/folder.

Yes your blog folder will not index.

[roothost]

oliviasmith, on 14 September 2011 - 11:56 AM, said:

Yes your blog folder will not index.

That's not strictly correct as some people have said on here, and after some research it appears the Search engines dont always play nicely and this will not 100% block them from crawling and indexing the pahe/folder.

[zed]

thread locked in agreement with roothost