public class RobotsTxt extends Object
| Modifier and Type | Field and Description |
|---|---|
static float |
UNSPECIFIED_CRAWL_DELAY |
| Constructor and Description |
|---|
RobotsTxt(IRobotsTxtFilter[] filters) |
RobotsTxt(IRobotsTxtFilter[] filters,
float crawlDelay) |
RobotsTxt(IRobotsTxtFilter[] filters,
String[] sitemapLocations) |
RobotsTxt(IRobotsTxtFilter[] filters,
String[] sitemapLocations,
float crawlDelay) |
| Modifier and Type | Method and Description |
|---|---|
boolean |
equals(Object obj) |
IRobotsTxtFilter[] |
getAllowFilters()
Gets "Allow" filters.
|
float |
getCrawlDelay() |
IRobotsTxtFilter[] |
getDisallowFilters()
Gets "Disallow" filters.
|
IRobotsTxtFilter[] |
getFilters()
Gets all filters.
|
String[] |
getSitemapLocations() |
int |
hashCode() |
String |
toString() |
public static final float UNSPECIFIED_CRAWL_DELAY
public RobotsTxt(IRobotsTxtFilter[] filters)
public RobotsTxt(IRobotsTxtFilter[] filters, float crawlDelay)
public RobotsTxt(IRobotsTxtFilter[] filters, String[] sitemapLocations)
public RobotsTxt(IRobotsTxtFilter[] filters, String[] sitemapLocations, float crawlDelay)
public IRobotsTxtFilter[] getFilters()
public IRobotsTxtFilter[] getDisallowFilters()
public IRobotsTxtFilter[] getAllowFilters()
public String[] getSitemapLocations()
public float getCrawlDelay()
Copyright © 2009–2021 Norconex Inc.. All rights reserved.