public class RobotsTxt extends Object
Modifier and Type | Field and Description |
---|---|
static float |
UNSPECIFIED_CRAWL_DELAY |
Constructor and Description |
---|
RobotsTxt(IRobotsTxtFilter... filters)
Creates a new robot txt object with the supplied filters.
|
RobotsTxt(List<IRobotsTxtFilter> filters)
Creates a new robot txt object with the supplied filters.
|
RobotsTxt(List<IRobotsTxtFilter> filters,
float crawlDelay) |
RobotsTxt(List<IRobotsTxtFilter> filters,
List<String> sitemapLocations) |
RobotsTxt(List<IRobotsTxtFilter> filters,
List<String> sitemapLocations,
float crawlDelay) |
Modifier and Type | Method and Description |
---|---|
boolean |
equals(Object other) |
List<IRobotsTxtFilter> |
getAllowFilters()
Gets "Allow" filters.
|
float |
getCrawlDelay() |
List<IRobotsTxtFilter> |
getDisallowFilters()
Gets "Disallow" filters.
|
List<IRobotsTxtFilter> |
getFilters()
Gets all filters.
|
List<String> |
getSitemapLocations() |
int |
hashCode() |
String |
toString() |
public static final float UNSPECIFIED_CRAWL_DELAY
public RobotsTxt(IRobotsTxtFilter... filters)
filters
- filterspublic RobotsTxt(List<IRobotsTxtFilter> filters)
filters
- filterspublic RobotsTxt(List<IRobotsTxtFilter> filters, float crawlDelay)
public RobotsTxt(List<IRobotsTxtFilter> filters, List<String> sitemapLocations)
public RobotsTxt(List<IRobotsTxtFilter> filters, List<String> sitemapLocations, float crawlDelay)
public List<IRobotsTxtFilter> getFilters()
null
)public List<IRobotsTxtFilter> getDisallowFilters()
null
)public List<IRobotsTxtFilter> getAllowFilters()
null
)public float getCrawlDelay()
Copyright © 2009–2023 Norconex Inc.. All rights reserved.