Hello.

 

I've recently updated a site in my Google webmaster tools. My problem that I am having is that instead of indexing my site's index file it shows the robot.txt file in the search results.

 

User-agent: * Disallow: /cgi-bin/ Disallow: /_notes/ Disallow: /agrana ...

 

This is an odd issue that I'm not finding any good solutions online for. Has anyone had a similiar issue and what was the steps necessary to resolve the matter?

 

Thank you.

Edited August 10, 201214 yr by ryans213

You should try disallowing robox.txt by using this following,

Disallow: /robots.txt

this would sort out your problem.

You should try disallowing robox.txt by using this following,

Disallow: /robots.txt

this would sort out your problem.

 

Strange. using robots.txt to disallow robots.txt. I think not.

 

I can only suggest checking to ensure that all the basics are correct. The robots.txt has to be in the root directory as does your index.html

Also check that the sintax is correct in that you have used the uppercase / lowercase correctly.

User-agent: *

Disallow: Allows all robots unrestricted access to your site

 

User-agent: *

Disallow: / Bars all robots from entire site

 

User-agent: *

Disallow: /this-directory/ Allows all robots access except to "this directory"

 

As you can see the placement and sintax is very important, and so has to be the starting point in trying to correct this. Also notice that you used the term robot.txt which could in itself be part of the problem since it must be named robots.txt (plural) Hope this points you in the right direction. Jim

Thank you both for your feedback.

 

Jim your attention to details showcases your experience here as the solution is almost always in the details. Unfortunately the robot.txt was my error and the file is actually robots.txt.

 

My file allows robots. After reading your your post I checked my file. The one thing I just tried to alter was spacing as the rest of the details seem to be inline with your feedback. I'm not sure how important things are here with this but I had spaces after the :'s.

 

User-agent: *

Disallow: /cgi-bin/

 

Changed now to.

 

User-agent:*

Disallow:/cgi-bin/

Sitemap:http://www.domain.net/sitemap.xml.gz

 

I'm not sure if this is enough of a change but I'm giving it a try.

 

Thanks again.

Hi, afraid you where correct in the fiirst instance. there should be a space between the colon and slash. Disallow: /cgi-bin/

Also you require a blank line before your reference to the site map:

User-agent:*

Disallow: /cgi-bin/

 

Sitemap:http://www.domain.net/sitemap.xml.gz

 

There are many robots.txt generators online. It may be worth running one of those and then simply comparing the sintax. I have never used a compressed site map myself but I cannot see that causing any problem. Good luck with finding the cause and solution. Jim

Thanks again JIm.

 

I changed things back to the original version. Its on odd situation as I did compare notes with a robots.txt generator when I created this file. I also have my sitemap verified in Google's webmaster tools. Looks like its back to the drawing board as to what is causing this kind of problem. I appreciate you taking the time to look into it.

No problems, let us know if you ever find out what caused this strange incident. Jim