I want to enable file scanning in:
/directory/
but not file scanning in:
/directory/subdirectory/
The correct robots.txt instruction:
User-agent: * Disallow: /subdirectory/
I am afraid that if I forbid / directory / subdirectory / that I would forbid scanning all files in / directory / that I do not want to do, so I will fix the use:
User-agent: * Disallow: /subdirectory/
user523521
source share