June 29, 201115 yr Ok, my first wordpress based website is live, still being updated, but it's live. I am very concerned with the amount of crawl errors it has had, and continues to have. I have asked on the google help forums but basically wp was originally installed in a subfolder /blog/, however, I (stupidly) uninstalled it and reinstalled it to the root as we decided to do the whole site in wp. Now, I backed up all the posts, tags and categories and reinstalled at the new location (which created new permalinks). Simple enough, but google had already crawled, and indexed, the site which has kicked off a huge crawl error and I am stumped at what to do. There are no links on the site pointing to the old urls but google still keeps trying to crawl them and they keep appearing in my crawl error log in webmaster tools Is it possible that the robots.txt is causing it? http://www.lewesbonfirecelebrations.com/robots.txt The crawl errors are increasing on a daily basis and its starting to really worry me now.
June 29, 201115 yr If you haven't done it already, re-submit your sitemap to reflect the changes and then start the oh-so-fun task of 301 redirecting all the 404 errors. Webmaster tools can be a bit slow to respond to any changes you make, you'll probably see those error reports for a good 2 weeks after fixing them.
June 29, 201115 yr Author Is the robots.txt restricting access to anything? http://www.lewesbonfirecelebrations.com/robots.txt
June 29, 201115 yr Is the robots.txt restricting access to anything? http://www.lewesbonfirecelebrations.com/robots.txt LOL! I love the name of some of the bots there. That robots.txt looks good to me. Blocks a lot of bad bots and stops all bots visiting the core php files - pretty standard for a Wordpress site.
June 30, 201115 yr Author LOL! I love the name of some of the bots there. That robots.txt looks good to me. Blocks a lot of bad bots and stops all bots visiting the core php files - pretty standard for a Wordpress site. I only mentioned it as in the webmaster tools area, under crawl errors it does come up with errors that are 'restricted by robots.txt'. Also, with the 301, how would I go about redirecting a url where there is no 'new' url to point to? EG the old url http://www.lewesbonfirecelebrations.com/tag/heart/, there is now no tag 'heart'.
June 30, 201115 yr I only mentioned it as in the webmaster tools area, under crawl errors it does come up with errors that are 'restricted by robots.txt'. Also, with the 301, how would I go about redirecting a url where there is no 'new' url to point to? EG the old url http://www.lewesbonf....com/tag/heart/, there is now no tag 'heart'. There's a 301 plugin for WordPress. pretty simple stuff, a list of text fields looking for old url and the new url. Basically take your list of error urls from Webmaster tools add them in the old url one by one and point them to the new one. If the old url / content is gone completely send a permanent redirect to the home page or somewhere you want the juice to go.
June 30, 201115 yr Author There's a 301 plugin for WordPress. pretty simple stuff, a list of text fields looking for old url and the new url. Basically take your list of error urls from Webmaster tools add them in the old url one by one and point them to the new one. If the old url / content is gone completely send a permanent redirect to the home page or somewhere you want the juice to go. Any ideas which one? There are a fair few plugins.
June 30, 201115 yr Any ideas which one? There are a fair few plugins. Anyone that does the job for you. They all do the same thing more than likely. As in.... set directives in your .htaccess file
June 30, 201115 yr Author Anyone that does the job for you. They all do the same thing more than likely. As in.... set directives in your .htaccess file That's what I thought, just the one I installed didnt make any changes to the .htaccess file?
June 30, 201115 yr Anyone that does the job for you. They all do the same thing more than likely. As in.... set directives in your .htaccess file Simple 301 redirects is the one i've used with success.
June 30, 201115 yr Author Is what I'm using http://wordpress.org/extend/plugins/simple-301-redirects/screenshots/ The 2 images below are the details i entered to redirect one of the pages and the contents of my .htaccess in the root after I did the redirect. See it just shows my non www redirect. What am I doing wrong? lol
June 30, 201115 yr Is what I'm using http://wordpress.org...ts/screenshots/ The 2 images below are the details i entered to redirect one of the pages and the contents of my .htaccess in the root after I did the redirect. See it just shows my non www redirect. What am I doing wrong? lol You should have something along the lines of redirect 301 http://yoursite.com/oldurl.html http://yoursite.com/newurl.html Edited June 30, 201115 yr by a.g.r.c
June 30, 201115 yr Author You should have something along the lines of redirect 301 http://yoursite.com/oldurl.html http://yoursite.com/newurl.html Nothing like that in the .htaccess file mate, would the plugin have installed it's own one somewhere? On the plus side, the redirect is active, just tested it. So, in theory if google tries to follow that old link now it will end up at the new location meaning the error should eventually be removed from the webmaster tools area?
June 30, 201115 yr Nothing like that in the .htaccess file mate, would the plugin have installed it's own one somewhere? On the plus side, the redirect is active, just tested it. So, in theory if google tries to follow that old link now it will end up at the new location meaning the error should eventually be removed from the webmaster tools area? Yeh sorry not going to troubleshoot a plugin. But in regards to Google webmaster tools, i've not done enough there to see the 301s disapear with the directives. However if you believe you have followed the technical aspects asked of you by Google, ask them to re-index your site. Would take a few days, but it seems as if you went through an overhaul and naturally have these things to contend with. Make sure you have the 301s in place and i'd contact Google.
June 30, 201115 yr Author Yeh sorry not going to troubleshoot a plugin. But in regards to Google webmaster tools, i've not done enough there to see the 301s disapear with the directives. However if you believe you have followed the technical aspects asked of you by Google, ask them to re-index your site. Would take a few days, but it seems as if you went through an overhaul and naturally have these things to contend with. Make sure you have the 301s in place and i'd contact Google. Cheers buddy
July 8, 201115 yr If you have don all thing for you blog then submit sitemap of you blog again and wait till crawling of your website
July 8, 201115 yr Author If you have don all thing for you blog then submit sitemap of you blog again and wait till crawling of your website The website is being crawled pretty much everyday and new sitemaps have been submitted. Along with redirecting all dead urls and asking them to be removed from the crawl error log within google webmaster tools.
Create an account or sign in to comment