Package | Description |
---|---|
com.norconex.collector.http.robot |
Modifier and Type | Method and Description |
---|---|
List<IRobotsTxtFilter> |
RobotsTxt.getAllowFilters()
Gets "Allow" filters.
|
List<IRobotsTxtFilter> |
RobotsTxt.getDisallowFilters()
Gets "Disallow" filters.
|
List<IRobotsTxtFilter> |
RobotsTxt.getFilters()
Gets all filters.
|
Constructor and Description |
---|
RobotsTxt(IRobotsTxtFilter... filters)
Creates a new robot txt object with the supplied filters.
|
Constructor and Description |
---|
RobotsTxt(List<IRobotsTxtFilter> filters)
Creates a new robot txt object with the supplied filters.
|
RobotsTxt(List<IRobotsTxtFilter> filters,
float crawlDelay) |
RobotsTxt(List<IRobotsTxtFilter> filters,
List<String> sitemapLocations) |
RobotsTxt(List<IRobotsTxtFilter> filters,
List<String> sitemapLocations,
float crawlDelay) |
Copyright © 2009–2023 Norconex Inc.. All rights reserved.