Uses of Class
com.norconex.collector.http.robot.RobotsTxt

Packages that use RobotsTxt

Package	Description
com.norconex.collector.http.delay
com.norconex.collector.http.delay.impl
com.norconex.collector.http.robot
com.norconex.collector.http.robot.impl

Uses of RobotsTxt in com.norconex.collector.http.delay

Methods in com.norconex.collector.http.delay with parameters of type RobotsTxt

Modifier and Type	Method and Description
void	IDelayResolver.delay(RobotsTxt robotsTxt, String url) Delay crawling activities (if applicable).

Uses of RobotsTxt in com.norconex.collector.http.delay.impl

Methods in com.norconex.collector.http.delay.impl with parameters of type RobotsTxt

Modifier and Type	Method and Description
void	AbstractDelayResolver.delay(RobotsTxt robotsTxt, String url)

Uses of RobotsTxt in com.norconex.collector.http.robot

Methods in com.norconex.collector.http.robot that return RobotsTxt

Modifier and Type	Method and Description
RobotsTxt	IRobotsTxtProvider.getRobotsTxt(org.apache.http.client.HttpClient httpClient, String url, String userAgent) Gets robots.txt rules.

Uses of RobotsTxt in com.norconex.collector.http.robot.impl

Methods in com.norconex.collector.http.robot.impl that return RobotsTxt

Modifier and Type	Method and Description
RobotsTxt	StandardRobotsTxtProvider.getRobotsTxt(org.apache.http.client.HttpClient httpClient, String url, String userAgent)
protected RobotsTxt	StandardRobotsTxtProvider.parseRobotsTxt(InputStream is, String url, String userAgent)