| Package | Description | 
|---|---|
| com.norconex.collector.http.pipeline.importer | |
| com.norconex.collector.http.robot | |
| com.norconex.collector.http.robot.impl | 
| Modifier and Type | Method and Description | 
|---|---|
| RobotsMeta | HttpImporterPipelineContext. getRobotsMeta() | 
| Modifier and Type | Method and Description | 
|---|---|
| void | HttpImporterPipelineContext. setRobotsMeta(RobotsMeta robotsMeta) | 
| Modifier and Type | Method and Description | 
|---|---|
| RobotsMeta | IRobotsMetaProvider. getRobotsMeta(Reader document,
             String documentUrl,
             ContentType contentType,
             Properties httpHeaders)Extracts Robots meta information for a page, if any. | 
| Modifier and Type | Method and Description | 
|---|---|
| RobotsMeta | StandardRobotsMetaProvider. getRobotsMeta(Reader document,
             String documentUrl,
             ContentType contentType,
             Properties httpHeaders) | 
Copyright © 2009–2021 Norconex Inc.. All rights reserved.