com.norconex.collector.http.robot.impl

Class StandardRobotsMetaProvider

java.lang.Object

com.norconex.collector.http.robot.impl.StandardRobotsMetaProvider

All Implemented Interfaces:

IRobotsMetaProvider, IXMLConfigurable

public class StandardRobotsMetaProvider
extends Object
implements IRobotsMetaProvider, IXMLConfigurable

Implementation of IRobotsMetaProvider as per X-Robots-Tag and ROBOTS standards. Extracts robots information from "ROBOTS" meta tag in an HTML page or "X-Robots-Tag" tag in the HTTP header (see https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag and http://www.robotstxt.org/meta.html).

If you specified a prefix for the HTTP headers, make sure to specify it again here or the robots meta tags will not be found.

If robots instructions are provided in both the HTML page and HTTP header, the ones in HTML page will take precedence, and the ones in HTTP header will be ignored.

XML configuration usage:

  <robotsMeta ignore="false" 
     class="com.norconex.collector.http.robot.impl.StandardRobotsMetaProvider">
     <headersPrefix>(string prefixing headers)</headersPrefix>
  </robotsMeta>
 

Usage example:

The following ignores robot meta information.

  <robotsMeta ignore="true" />
 

Author:

Pascal Essiembre

Constructor Summary

Constructors

Constructor and Description
StandardRobotsMetaProvider()

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object other)
String	getHeadersPrefix()
RobotsMeta	getRobotsMeta(Reader document, String documentUrl, ContentType contentType, Properties httpHeaders) Extracts Robots meta information for a page, if any.
int	hashCode()
void	loadFromXML(Reader in)
void	saveToXML(Writer out)
void	setHeadersPrefix(String headersPrefix)
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

StandardRobotsMetaProvider

public StandardRobotsMetaProvider()

Method Detail

getRobotsMeta

public RobotsMeta getRobotsMeta(Reader document,
                                String documentUrl,
                                ContentType contentType,
                                Properties httpHeaders)
                         throws IOException

Description copied from interface: IRobotsMetaProvider

Extracts Robots meta information for a page, if any.

Specified by:

getRobotsMeta in interface IRobotsMetaProvider

Parameters:

document - the document

documentUrl - document url

contentType - the document content type

httpHeaders - the document HTTP Headers

Returns:

robots meta instance

Throws:

IOException - problem reading the document

getHeadersPrefix

public String getHeadersPrefix()

setHeadersPrefix

public void setHeadersPrefix(String headersPrefix)

loadFromXML

public void loadFromXML(Reader in)
                 throws IOException

Specified by:

loadFromXML in interface IXMLConfigurable

Throws:

IOException

saveToXML

public void saveToXML(Writer out)
               throws IOException

Specified by:

saveToXML in interface IXMLConfigurable

Throws:

IOException

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object