Package com.norconex.collector.http.robot.impl

Class Summary

Class	Description
StandardRobotsMetaProvider	Implementation of IRobotsMetaProvider as per X-Robots-Tag and ROBOTS standards.
StandardRobotsTxtProvider	Implementation of IRobotsTxtProvider as per the robots.txt standard described at http://www.robotstxt.org/robotstxt.html.