com.norconex.collector.http.robot

Interface IRobotsMetaProvider

All Known Implementing Classes:

StandardRobotsMetaProvider

public interface IRobotsMetaProvider

Responsible for extracting robot information from a page.

Author:

Pascal Essiembre

Method Summary

Modifier and Type	Method and Description
RobotsMeta	getRobotsMeta(Reader document, String documentUrl, ContentType contentType, Properties httpHeaders) Extracts Robots meta information for a page, if any.

Method Detail

getRobotsMeta

RobotsMeta getRobotsMeta(Reader document,
                         String documentUrl,
                         ContentType contentType,
                         Properties httpHeaders)
                  throws IOException

Extracts Robots meta information for a page, if any.

Parameters:

document - the document

documentUrl - document url

contentType - the document content type

httpHeaders - the document HTTP Headers

Returns:

robots meta instance

Throws:

IOException - problem reading the document