Uses of Package
com.norconex.collector.http.robot

Packages that use com.norconex.collector.http.robot

Package	Description
com.norconex.collector.http.crawler
com.norconex.collector.http.delay
com.norconex.collector.http.delay.impl
com.norconex.collector.http.pipeline.importer
com.norconex.collector.http.robot
com.norconex.collector.http.robot.impl

Classes in com.norconex.collector.http.robot used by com.norconex.collector.http.crawler

Class and Description
IRobotsMetaProvider Responsible for extracting robot information from a page.
IRobotsTxtProvider Given a URL, extract any "robots.txt" rules.

Classes in com.norconex.collector.http.robot used by com.norconex.collector.http.delay

Class and Description
RobotsTxt

Classes in com.norconex.collector.http.robot used by com.norconex.collector.http.delay.impl

Class and Description
RobotsTxt

Classes in com.norconex.collector.http.robot used by com.norconex.collector.http.pipeline.importer

Class and Description
RobotsMeta

Classes in com.norconex.collector.http.robot used by com.norconex.collector.http.robot

Class and Description
IRobotsTxtFilter Holds a robots.txt rule.
RobotsMeta
RobotsTxt

Classes in com.norconex.collector.http.robot used by com.norconex.collector.http.robot.impl

Class and Description
IRobotsMetaProvider Responsible for extracting robot information from a page.
IRobotsTxtProvider Given a URL, extract any "robots.txt" rules.
RobotsMeta
RobotsTxt