com.norconex.collector.http.robot

Interface IRobotsTxtProvider

All Known Implementing Classes:

StandardRobotsTxtProvider

public interface IRobotsTxtProvider

Given a URL, extract any "robots.txt" rules. Implementations are expected to cache existing robots.txt instances or, cache the fact none was found, for the duration of a crawl session so no attempt to re-download it is made.

Author:

Pascal Essiembre

Method Summary

Modifier and Type	Method and Description
RobotsTxt	getRobotsTxt(org.apache.http.client.HttpClient httpClient, String url, String userAgent) Gets robots.txt rules.

Method Detail

getRobotsTxt

RobotsTxt getRobotsTxt(org.apache.http.client.HttpClient httpClient,
                       String url,
                       String userAgent)

Gets robots.txt rules. This method signature changed in 1.3 to include the userAgent.

Parameters:

httpClient - the http client to grab robots.txt

url - the URL to derive the robots.txt from

userAgent - the User-Agent to match ourselves with the robot rules

Returns:

robots.txt rules