Hi, so I'm kinda new to the robots.txt, but I learned a bit about it and i decided to give it a shot in order to optimize the crawl budget.

So I wanted my url with the end "/blog-home/" to not show up on the search page So I wrote this in the robots.txt file:

User-agent: *

Disallow: /wp-admin/

Disallow: /blog-home/

Allow: /wp-admin/admin-ajax.php

Then saved. Yet when I type site:mydomain.com the url with the /blog-home/ still shows up as a result. What did I do wrong? Maybe I just have to wait a little? (it's not even been an hour since I edited the .txt)

Edited August 23, 20223 yr by TrioWorks

You may have to wait until Google reindexes the site.

But even then, it will only work if there are no links to blog-home from anywhere else on the site.

You can’t disallow wp-admin and allow Ajax. That won’t work.

3 hours ago, fisicx said:

You may have to wait until Google reindexes the site.

But even then, it will only work if there are no links to blog-home from anywhere else on the site.

You can’t disallow wp-admin and allow Ajax. That won’t work.

Is there a way to check whether there are links that direct to blog-home or not?

About Ajax - The guy below said at 1:43 that it's a wordpress thing and that we have to allow this link idk:

 

It's your site - you would know if there are any links to blog-home

If you want ajax to work you need to allow access to wp-admin. But that's a different thing to robots.txt.

robots.txt is about stopping google from indexing parts of the site. Google isn't going to index wp-admin as there is no indexable content. So no point in blocking anything.

Read this to understand how it works:

https://developers.google.com/search/docs/advanced/robots/intro

This could be a possibility too:

This one is more common in websites that are more than a few years old. Google has stopped obeying noindex rules in robots.txt files as of September 1, 2019.

If your robots.txt file was created before that date, or contains noindex instructions, you’re likely to see those pages indexed in Google’s search results.

The solution to this problem is to implement an alternative ‘noindex’ method.

One option is the robots meta tag, which you can add to the head of any web page you want to prevent Google from indexing.

On 9/9/2022 at 5:04 PM, Allison Garner said:

This could be a possibility too:

This one is more common in websites that are more than a few years old. Google has stopped obeying noindex rules in robots.txt files as of September 1, 2019.

If your robots.txt file was created before that date, or contains noindex instructions, you’re likely to see those pages indexed in Google’s search results.

The solution to this problem is to implement an alternative ‘noindex’ method.

One option is the robots meta tag, which you can add to the head of any web page you want to prevent Google from indexing.

As I said, I'm new to robots.txt, so I didn't make the .txt prior september 2019. 

 

It's been over a month, and google still shows up /blog-home/. Does anyone know what should I do?

Are there any inbound links to any of the blog posts?

Have you disabled pings?

is there a reason why you don’t want the blog indexed?