April 3, 200818 yr Hi all, I'm posting this thread because I cant find any answers anywhere to this question. I was wondering what you all thought?? Basiclally, im wondering what the best way of preventing your incomplete site from being spidered by the search engines? I know you can include robots.txt to prevent your site, or pages being crawled; and I know you could use APACHE on your own machine to take away this problem - but is there any other way? What do you all tend to do? Suppose im worried that if I leave the noindex instructions in my robot.txt file for months while I get a site running - google et all my not bother to check to see it's changed very regularly. Valid concern or not? Thanks. Paul.
April 3, 200818 yr If a site in incomplete in my book it's not ready to go live anyway so it's shouldn't be publically accessible. 1. If you need certain pages to be crawled don't link to the unfinished sections/pages, and in fact don't even upload unfinished stuff. 2. You could password protect folders/pages until they are ready to go live 3. You could use rel="nofollow" and robots.txt as you say
Create an account or sign in to comment