Uses of Package
com.norconex.collector.http.robot

Packages that use com.norconex.collector.http.robot

Package	Description
com.norconex.collector.http.crawler
com.norconex.collector.http.delay
com.norconex.collector.http.delay.impl
com.norconex.collector.http.pipeline.importer
com.norconex.collector.http.robot
com.norconex.collector.http.robot.impl

Classes in com.norconex.collector.http.robot used by com.norconex.collector.http.crawler

Class	Description
IRobotsMetaProvider	Responsible for extracting robot information from a page.
IRobotsTxtProvider	Given a URL, extract any "robots.txt" rules.

Classes in com.norconex.collector.http.robot used by com.norconex.collector.http.delay

Class	Description
RobotsTxt

Classes in com.norconex.collector.http.robot used by com.norconex.collector.http.delay.impl

Class	Description
RobotsTxt

Classes in com.norconex.collector.http.robot used by com.norconex.collector.http.pipeline.importer

Class	Description
RobotsMeta

Classes in com.norconex.collector.http.robot used by com.norconex.collector.http.robot

Class	Description
IRobotsTxtFilter	Holds a robots.txt rule.
RobotsMeta
RobotsTxt

Classes in com.norconex.collector.http.robot used by com.norconex.collector.http.robot.impl

Class	Description
IRobotsMetaProvider	Responsible for extracting robot information from a page.
IRobotsTxtProvider	Given a URL, extract any "robots.txt" rules.
RobotsMeta
RobotsTxt