Robots.txt - does it work?

Question

I just came across a robots.txt file that looks like this:

User-agent: * Disallow: /foobar User-agent: badbot Disallow: * 

After only a few folders are allowed for everyone, is only the badbot rule badbot ?

Note. This question is only for understanding the above set of rules. I know that using robots.txt is not an appropriate security mechanism, and I do not use or protect it.

Answer 1

Each bot only ever observes a maximum of one record (block).

A block starts with one line or more of the User-agent , followed by Disallow lines ( at least one is required ). Blocks are separated by blank lines.

A bot called "badbot" will look for an entry with the User-agent: badblock (or similar, since the bot should be liberal in interpreting this field)). If such a line is not found, it will search for an entry with the User-agent: * . Even if this does not exist, the bot is allowed to do everything (= default).

So, in your example, a bot called "badbot" will only follow the second entry (you probably mean Disallow: / instead of Disallow: * ), while all other bots follow only the first entry.