Robots.txt: Is this a wildcard rule?

Question

Simple question. I want to add:

Disallow */*details-print/ 

In principle, the blocking rules in the form /foo/bar/dynamic-details-print --- foo and bar in this example can also be completely dynamic.

I thought it would be simple, but then on www.robotstxt.org there is this message:

Note also that globularization and regular expression are not supported on either the User-agent lines or Disallow. "*" In the field "User-agent" is a special value meaning "any robot". In particular, you cannot have strings such as "User-agent: bot", "Disallow: / tmp / *" or "Disallow: * .gif".

So we can’t do this? Do search engines support? But here, the Quora.com robots.txt file:

 Disallow: /ajax/ Disallow: /*/log Disallow: /*/rss Disallow: /*_POST 

So who is right - Or do I not understand the text on robotstxt.org?

Thanks!

Answer 1

Answer: "It depends." "Robots.txt" robots.txt, as defined on robotstxt.org, is the minimum that bots are expected to support. Googlebot, MSNbot and Yahoo Slurp support some common extensions, and in fact there is no information that other bots support. Some say they support, while others do not.

In general, you can expect that the main search engine bots will support the wildcards that you wrote, and the one you have there looks like it will work. It would be best to run it for one or more of these robots.txt testers, or use Google Webmaster Tools to test it.