Package com.norconex.collector.http.robot

Interface Summary

Interface	Description
IRobotsMetaProvider	Responsible for extracting robot information from a page.
IRobotsTxtFilter	Holds a robots.txt rule.
IRobotsTxtProvider	Given a URL, extract any "robots.txt" rules.

Class Summary

Class	Description
RobotsMeta
RobotsTxt