JavaScript is disabled on your browser.
Skip navigation links
Overview
Package
Class
Use
Tree
Deprecated
Index
Help
Package:
Description |
Related Packages
|
Classes and Interfaces
SEARCH:
Package com.norconex.collector.http.robot
package
com.norconex.collector.http.robot
Related Packages
Package
Description
com.norconex.collector.http
com.norconex.collector.http.robot.impl
All Classes and Interfaces
Interfaces
Classes
Class
Description
IRobotsMetaProvider
Responsible for extracting robot information from a page.
IRobotsTxtFilter
Holds a robots.txt rule.
IRobotsTxtProvider
Given a URL, extract any "robots.txt" rules.
RobotsMeta
RobotsTxt