com.norconex.collector.http.robot.impl

Class StandardRobotsTxtProvider

java.lang.Object

com.norconex.collector.http.robot.impl.StandardRobotsTxtProvider

All Implemented Interfaces:

IRobotsTxtProvider

public class StandardRobotsTxtProvider
extends Object
implements IRobotsTxtProvider

Implementation of IRobotsTxtProvider as per the robots.txt standard described at http://www.robotstxt.org/robotstxt.html.

XML configuration usage:

  <robotsTxt ignore="false"
     class="com.norconex.collector.http.robot.impl.StandardRobotsTxtProvider"/>
 

Usage example:

The following ignores "robots.txt" files present on web sites.

  <robotsTxt ignore="true" />
 

Author:

Pascal Essiembre

Constructor Summary

Constructors

Constructor and Description
StandardRobotsTxtProvider()

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object other)
RobotsTxt	getRobotsTxt(org.apache.http.client.HttpClient httpClient, String url, String userAgent) Gets robots.txt rules.
int	hashCode()
protected RobotsTxt	parseRobotsTxt(InputStream is, String url, String userAgent)
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

StandardRobotsTxtProvider

public StandardRobotsTxtProvider()

Method Detail

getRobotsTxt

public RobotsTxt getRobotsTxt(org.apache.http.client.HttpClient httpClient,
                              String url,
                              String userAgent)

Description copied from interface: IRobotsTxtProvider

Gets robots.txt rules. This method signature changed in 1.3 to include the userAgent.

Specified by:

getRobotsTxt in interface IRobotsTxtProvider

Parameters:

httpClient - the http client to grab robots.txt

url - the URL to derive the robots.txt from

userAgent - the User-Agent to match ourselves with the robot rules

Returns:

robots.txt rules

parseRobotsTxt

protected RobotsTxt parseRobotsTxt(InputStream is,
                                   String url,
                                   String userAgent)
                            throws IOException

Throws:

IOException

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object