Package com.norconex.collector.http.robot

Interface IRobotsMetaProvider

All Known Implementing Classes:

StandardRobotsMetaProvider

public interface IRobotsMetaProvider

Responsible for extracting robot information from a page.

Author:

Pascal Essiembre

Method Summary

Modifier and Type	Method	Description
RobotsMeta	getRobotsMeta(Reader document, String documentUrl, ContentType contentType, Properties httpHeaders)	Extracts Robots meta information for a page, if any.

Method Details

getRobotsMeta

RobotsMeta getRobotsMeta(Reader document,
 String documentUrl,
 ContentType contentType,
 Properties httpHeaders)
                  throws IOException

Extracts Robots meta information for a page, if any.

Parameters:

document - the document

documentUrl - document url

contentType - the document content type

httpHeaders - the document HTTP Headers

Returns:

robots meta instance

Throws:

IOException - problem reading the document