robots.txt Checker

Read and display a site's robots.txt to verify crawler rules.