Uses of Class
com.norconex.collector.http.robot.RobotsTxt

Packages that use RobotsTxt

Package	Description
com.norconex.collector.http.delay
com.norconex.collector.http.delay.impl
com.norconex.collector.http.robot
com.norconex.collector.http.robot.impl

Uses of RobotsTxt in com.norconex.collector.http.delay

Methods in com.norconex.collector.http.delay with parameters of type RobotsTxt

Modifier and Type	Method	Description
void	IDelayResolver.delay(RobotsTxt robotsTxt, String url)	Delay crawling activities (if applicable).

Uses of RobotsTxt in com.norconex.collector.http.delay.impl

Methods in com.norconex.collector.http.delay.impl with parameters of type RobotsTxt

Modifier and Type	Method	Description
void	AbstractDelayResolver.delay(RobotsTxt robotsTxt, String url)

Uses of RobotsTxt in com.norconex.collector.http.robot

Methods in com.norconex.collector.http.robot that return RobotsTxt

Modifier and Type	Method	Description
RobotsTxt	IRobotsTxtProvider.getRobotsTxt(HttpFetchClient fetchClient, String url)	Gets robots.txt rules.

Uses of RobotsTxt in com.norconex.collector.http.robot.impl

Methods in com.norconex.collector.http.robot.impl that return RobotsTxt

Modifier and Type	Method	Description
RobotsTxt	StandardRobotsTxtProvider.getRobotsTxt(HttpFetchClient fetcher, String url)
protected RobotsTxt	StandardRobotsTxtProvider.parseRobotsTxt(InputStream is, String url, String userAgent)