If the site map has * every * url

Question

I have a website with a huge number (well, thousands or tens of thousands) of dynamic URLs, as well as several static URLs.

In theory, due to some tricky SEO binding on the main page, any spider can crawl the site and detect all dynamic URLs using a convenient search spider.

With that in mind, do I really have to worry about putting in the effort to create a dynamic sitemap index that includes all of these URLs, or do I just have to make sure all the main static URLs are there?

This actual way in which I will generate this is not a concern - I just questioned the need to do this.

In fact, the Google FAQ (and yes, I know that they are not the only search engine!) Recommends including URLs in the site map, which cannot be detected by crawling; based on this fact, if each URL of your site reaches a different one, probably the only URL that you really need as a baseline in your site map for a well-designed site is your home page?

Answer 1

If there are several ways to get to the page, you must select the main URL for each page containing the actual content, and place these URLs on the site map. That is, the site map should contain links to the actual content, and not all possible URLs to get the same content.

Also consider placing canonical meta tags on pages with this primary URL so that spiders can recognize the page even if it is reachable via different dynamic URLs.

Spiders spend limited time searching for each site, so you should easily find the actual content as soon as possible. A site map can be a great help, as you can use it to directly display the actual content so that the spider does not look for it.

We had good results using these methods, and now Google indexes 80-90% of our dynamic content. :)

Answer 2

In the SO podcast, they talked about restrictions on the number of links that you could include / send to the site map (about 500 per page with page limit based on pagerank?) And how you would need to split them into several pages.

Given this, do I really need to worry about wasting the effort to produce a dynamic sitemap index that includes all of these URLs, or should I just make sure all the main static URLs are there?

I got the impression that the site map is not necessarily connected with disabled pages, but rather with an increase in crawl of existing pages. In my experience, when a site contains a site map, secondary pages, even if they are visibly related to them, appear more often in Google results. Depending on page links / inbound links, etc. Your site may be a problem.