Robots + X-Robots checker

Check crawl/index directives across robots.txt, meta robots, and X-Robots-Tag.