Before I purchased my hosting i created a test area on my girlfriends site on her FTP /pbweb with all html pages within this folder. About a month ago I purchased my hosting and removed ALL files folders from my girlfriends website.

 

However when searching for "pb web design warrington" Google is still picking up her website.com/pbweb folder and files under that directory even though they are not there any more??

 

It is also picking up old html files from my hosting for my website as well. See attached images.

 

Why is it doing this and how can i get /pbweb etc removed from the search results for my girlfriends website and to pick up the NEW files from my website?

Hey,

 

Google has it's own cache and will keep certain pages listed for a while after they have been removed. I've seen dead pages linger in results for up to 5months before.

 

The best way to combat this would be to delete all pages (which you have done) and then block search engines from crawling the directory. It is then still a waiting game until your site is removed.

 

Alternatively, you could take advantage of the situation and 301 to your new website if it is in the same vein.

 

I'd recommend submitting your new website's sitemap to Google Webmaster Tools and knocking the crawler speed upto faster for the first few months. As long as you're using a different domain name then you should start seeing your new site listed in no time. Obviously this wont work if you have blocked crawler access.

Cheers,

 

I have set the crawler rate to Fastest (see attached image)

 

My girlfriends website as yet does not have a sitemap.xml or robots.txt

 

My site does have robots.txt for pbwebdesign-warrington.co.uk does this look ok? I also have a sitemap submitted. Do i need to disallow anything in my robots.txt could this be preventing anything crawling?

 

# robots.txt

User-agent: *

Disallow:

Disallow: /cgi-bin/

Disallow: /formmail/

Sitemap: http://www.pbwebdesign-warrington.co.uk/sitemap.xml

 

Would you recommend doing the same for my girlfriends website as well??

Ok, sorry, I may of misunderstood the situation a little.

 

Your website is already listed well, and as you do not have a lot of fresh updated content on your website there's no need to set the crawl rate so high.

 

On the other hand, it would be best to do this on your girlfriend's website as it may increase crawl rates through her site, therefore possibly decreasing the amount of time it takes google to recognise that your test pages are no longer there. But don't forget to turn this back down afterwards, as it can cause a high amount of traffic from crawlers.

 

I would recommend adding a robots.txt to your girlfriend's site and disallowing crawler access to /pbweb/. Also, if you are going to be doing further test-runs of sites in the future makes sure you always block crawler access to that specific test folder before uploading the files.

 

You could add a sitemap for her, it only takes a few minutes and it's not going to hurt.

Oh, and while I'm at it... you should probably fix some of the errors in your site markup: http://validator.w3.org/check?uri=http://www.pbwebdesign-warrington.co.uk/&charset=(detect+automatically)&doctype=Inline&group=0

Cheers for your help mate. Yeah the validation is on my 'to do' list... going to take forever lol.

 

Ive done the robots.txt and sitemap for my girlfriends website so ill upload them when i get home.

 

So should i leave the crawl rate set to fastest for my website pbweb for a week or so then lower it to default? Or just set it back to default now?

 

But set the crawl rate on my girlfriends website to fastest?? And how long do you leave it at this for?

You should be fine just setting yours back to default now, google had already indexed your site relatively well already. You'd just be increasing crawler traffic for little to no reason.

 

I'd suggest leaving your girlfriends site crawl rate on one of the faster settings for around two weeks then switching it back. Although if your test pages are removed before then you can just switch it back then & there.

 

Good luck buddy & hope it sorts things out for you.

COuld try 301 redirecting the /pbweb folder to your main website.

 

Do you think that would be the better option? I restricted access to /pbweb folder on my girlfriend site with robots.txt