Package com.norconex.collector.http.robot

Class RobotsTxt

java.lang.Object

com.norconex.collector.http.robot.RobotsTxt

public class RobotsTxt
extends Object

Field Summary

Fields

Modifier and Type	Field	Description
static final float	UNSPECIFIED_CRAWL_DELAY

Constructor Summary

Constructors

Constructor	Description
RobotsTxt(IRobotsTxtFilter... filters)	Creates a new robot txt object with the supplied filters.
RobotsTxt(List<IRobotsTxtFilter> filters)	Creates a new robot txt object with the supplied filters.
RobotsTxt(List<IRobotsTxtFilter> filters, float crawlDelay)
RobotsTxt(List<IRobotsTxtFilter> filters, List<String> sitemapLocations)
RobotsTxt(List<IRobotsTxtFilter> filters, List<String> sitemapLocations, float crawlDelay)

Method Summary

Modifier and Type	Method	Description
boolean	equals(Object other)
List<IRobotsTxtFilter>	getAllowFilters()	Gets "Allow" filters.
float	getCrawlDelay()
List<IRobotsTxtFilter>	getDisallowFilters()	Gets "Disallow" filters.
List<IRobotsTxtFilter>	getFilters()	Gets all filters.
List<String>	getSitemapLocations()
int	hashCode()
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Details

UNSPECIFIED_CRAWL_DELAY

public static final float UNSPECIFIED_CRAWL_DELAY

See Also:

• Constant Field Values

Constructor Details

RobotsTxt

public RobotsTxt(IRobotsTxtFilter... filters)

Creates a new robot txt object with the supplied filters.

Parameters:

filters - filters

RobotsTxt

public RobotsTxt(List<IRobotsTxtFilter> filters)

Creates a new robot txt object with the supplied filters.

Parameters:

filters - filters

Since:

3.0.0

RobotsTxt

public RobotsTxt(List<IRobotsTxtFilter> filters,
 float crawlDelay)

RobotsTxt

public RobotsTxt(List<IRobotsTxtFilter> filters,
 List<String> sitemapLocations)

RobotsTxt

public RobotsTxt(List<IRobotsTxtFilter> filters,
 List<String> sitemapLocations,
 float crawlDelay)

Method Details

getFilters

public List<IRobotsTxtFilter> getFilters()

Gets all filters.

Returns:

filters (never null)

getDisallowFilters

public List<IRobotsTxtFilter> getDisallowFilters()

Gets "Disallow" filters.

Returns:

disallow filters (never null)

Since:

2.4.0

getAllowFilters

public List<IRobotsTxtFilter> getAllowFilters()

Gets "Allow" filters.

Returns:

allow filters (never null)

Since:

2.4.0

getSitemapLocations

public List<String> getSitemapLocations()

getCrawlDelay

public float getCrawlDelay()

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object