Skip to content
View in the app

A better way to browse. Learn more.
Web Designer Forum

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.

Sitemap.xml and robots.txt issue. Please share your views

Barry530
in SEO (Search Engine Optimisation) & Search Engines

Featured Replies

Hi,


Please advice on the following:


I have entered now my website in to the google webmaster tools and also added the sitemap.xml file in to it.

Something with the robots.txt is a bit weird. The sitemap.xml file contains links to sites which the robots.txt disallow.


Cheers!

Barry

Most sitemap generates will follow all the links on a page whether the robots.txt file is blocking them from the SERPs bots.

Most sitemap generates will follow all the links on a page whether the robots.txt file is blocking them from the SERPs bots.

 

Bingo. Also I find it safer to auto generate an xml sitemap with say PHP rather than using a crawler method.

  • 3 weeks later...

The Sitemaps are used to tell the Google boot to crawl following links but the robots.txt disallows the robots to access them. Change your setting in robots.txt to make that crawlable.

Create an account or sign in to comment
Go to topic listing

Account

Navigation

Search

Search

Configure browser push notifications
Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.