Looking in Google Analytics, I see a 404 error for this page

https://www.kirkbymicrowave.co.uk/sales/85054/Agilent+8510C+boot+upWe+TalerangA

I have no idea why Google should try to search such a page at all.

Now the first bit https://www.kirkbymicrowave.co.uk/sales/85054/ seems perfectly reasonable - such a page exists.

Although the manufacture (Agilent) and a particular network analyzer (8510C) are both related to my site, there's never been a page with the name Agilent+8510C. But at least I can understand where those terms come from.

Now where the hell does boot, upWe, and TalerrangA come from I have no idea. At least I know what the first word (boot) means, but I don't think its one the website, but the last two are certainty not on the website, and never have been. It seems as though Google has used a random letter generator to generate a URL, then find a 404 error

 

 

Edited July 19, 20188 yr by drkirkby
add a bit extra information.

If you check search console it should have a 'linked from' if it's known. I normally find junk like this is links from scrape directories as they often end up appending your urls with parameters, like links to yourdomain.com?url=yourdomain.com. Good to know your site correctly returns 404, I've seen many a case where a site allows this sort of thing, ends up with a hell of a lot of junk pages.

Even I have seen such activities with our Kb section. It's really frustrating to see so many 404 error warnings.

Anyone knows the exact steps to avoid this?

Edited July 24, 20188 yr by webhostuk

You should really be looking into site architecture, you can stop a load of scraped links and junk from appearing in the first place, and it will allow legitimate links (e.g. ones not appended) through fine, you can even write exceptions to the rules for things like Google Ads with with utm source etc, side note I had to write a rule for this as I managed to 404 Google Ads in my experiments lol.

The best way to do it is to build the canonical out of legitimate variables, then when you  $_SERVER['REQUEST_URI'] and compare you can tell if they don't match, and if they don't match you can determine why before adding a 404 header, simplified example;

<?php
$cur_uri =  $_SERVER['REQUEST_URI']; // Say comes back as /BLog/FREE-Chicken-AT-kfC
$canonical = "/blog/free-chicken-at-kfc";
if($cur_uri!=$canonical) { // Somethings up
	if(strtolower($cur_uri)==strtolower($canonical)) { // Case issue
		// 301 to $canonical
	} else {
		// 404
	}
}
?>

Edit: Always remember a url is a string of infinite possibilities, if expecting only 3 variables rule out the rest

Edited July 24, 20188 yr by BrowserBugs

17 hours ago, BrowserBugs said:

You should really be looking into site architecture, you can stop a load of scraped links and junk from appearing in the first place, and it will allow legitimate links (e.g. ones not appended) through fine, you can even write exceptions to the rules for things like Google Ads with with utm source etc, side note I had to write a rule for this as I managed to 404 Google Ads in my experiments lol.

The best way to do it is to build the canonical out of legitimate variables, then when you  $_SERVER['REQUEST_URI'] and compare you can tell if they don't match, and if they don't match you can determine why before adding a 404 header, simplified example;

<?php
$cur_uri =  $_SERVER['REQUEST_URI']; // Say comes back as /BLog/FREE-Chicken-AT-kfC
$canonical = "/blog/free-chicken-at-kfc";
if($cur_uri!=$canonical) { // Somethings up
	if(strtolower($cur_uri)==strtolower($canonical)) { // Case issue
		// 301 to $canonical
	} else {
		// 404
	}
}
?>

Edit: Always remember a url is a string of infinite possibilities, if expecting only 3 variables rule out the rest

I'd imagine you can use regex to catch a load of these.  One thing of note - querys on urls shouldn't return 404 if the base url is valid.  http://mydomain.com/blog?posts=4 the posts=4 being the query.  It should only return a 404 if /blog does not exist.

2 hours ago, rbrtsmith said:

I'd imagine you can use regex to catch a load of these.  One thing of note - querys on urls shouldn't return 404 if the base url is valid.  http://mydomain.com/blog?posts=4 the posts=4 being the query.  It should only return a 404 if /blog does not exist.

Queries is one area I do limit, but then you can build a url to test from so you'd include ?posts=4 because it's something you would expect, doesn't have to use only the canonical. An example would be pagination, ?page=2 etc but if you only had 4 pages and someone decides to add ?page=398 you need to work out if you want to allow it or forward them to page 1 etc, page 398 would 200 ok with no results to show. This method also helps reduce crawl on wasted urls.

In the op that weird url could easily have been /sales/85054/?Agilent+8510C+boot+upWe+TalerangA, or /sales/85054/?source=best+viagra+for+sale; canonical would say "hey, this is a duplicate of /sales/85054/" but returning a 200 ok for the scrub site linking to you.

I had to clean up a site for a client which had this very problem, ended up with over 900k pages in Googles index, they had used href="&page=" in the pagination links, ended up with loads of urls like ?typ=4&page=11&page=10&page=9 etc.

Edited July 25, 20188 yr by BrowserBugs

If the query fails I'd just default to the baseUrl,  Google should only index queries that result in different content/markup.  So invalid queries shouldn't get indexed given they just load the base page i.e. page=300 would just hit the initial page.

For existing sites that already have queries indexed say for page 300 and then those pages are removed.  Then that is a different story!   You'd likely need to set up a load of 301s

Edited July 26, 20188 yr by rbrtsmith

You would have thought so, it's a logical theory, sadly it's not as clean cut from my experience with analytical and console data. It all comes back to the 200 ok headers, by default this says the url is ok as it served a page on the given url. Canonical will sort itself out eventually, but in Googles example it does treat exact same content on separate urls as separate pages. To quote;

Quote

To specify which URL that you want people to see in search results. You might prefer people reach your green dresses product page via https://www.example.com/dresses/green/greendress.html rather than https://example.com/dresses/cocktail?gclid=ABCD.

Quote

To consolidate link signals for similar or duplicate pages. It helps search engines to be able to consolidate the information they have for the individual URLs (such as links to them) into a single, preferred URL. This means that links from other sites to http://example.com/dresses/cocktail?gclid=ABCD get consolidated with links to https://www.example.com/dresses/green/greendress.html.

So with this example if /dresses/cocktail?gclid=ABCD is identical to /dresses/green/greendress.html then so would /dresses/cocktail?gclid=ABCD&utm_source=somewhere, a third page, all give 200 ok with the exact same information.

The reason for testing where you are vs where you should have been for me is the same as validating information on a form submission. All to often I see queries being indexed in search engines, it takes a few sites to link to you in a variety of ways to start to blur the architectural lines. And if these sites linking to you were not asked for, and the links are trying to add variables then you need to ask why. Search engines don't add anything from organic results, AdWords add what you ask them to and you can adjust for this, but besthighheelshoes.net linking yourdomain.com/products/?url=yourdomain.com&referrer=big+heels ... do you want to accept the link or 404 and make their site have broken links all over the place?