RobotsTxt (Norconex HTTP Collector 3.1.0 API)

java.lang.Object
- com.norconex.collector.http.robot.RobotsTxt

```
public class RobotsTxt
extends Object
```

Field Summary

Fields
Modifier and Type Field Description

static float UNSPECIFIED_CRAWL_DELAY

Constructor Summary

Constructors
Constructor	Description
`RobotsTxt(IRobotsTxtFilter... filters)`	Creates a new robot txt object with the supplied filters.
`RobotsTxt(List<IRobotsTxtFilter> filters)`	Creates a new robot txt object with the supplied filters.
`RobotsTxt(List<IRobotsTxtFilter> filters, float crawlDelay)`
`RobotsTxt(List<IRobotsTxtFilter> filters, List<String> sitemapLocations)`
`RobotsTxt(List<IRobotsTxtFilter> filters, List<String> sitemapLocations, float crawlDelay)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`boolean`	`equals(Object other)`
`List<IRobotsTxtFilter>`	`getAllowFilters()`	Gets "Allow" filters.
`float`	`getCrawlDelay()`
`List<IRobotsTxtFilter>`	`getDisallowFilters()`	Gets "Disallow" filters.
`List<IRobotsTxtFilter>`	`getFilters()`	Gets all filters.
`List<String>`	`getSitemapLocations()`
`int`	`hashCode()`
`String`	`toString()`

Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail
- UNSPECIFIED_CRAWL_DELAY
```
public static final float UNSPECIFIED_CRAWL_DELAY
```
  See Also:
  
  Constant Field Values

Constructor Detail

RobotsTxt
```
public RobotsTxt(IRobotsTxtFilter... filters)
```
Creates a new robot txt object with the supplied filters.

Parameters:

filters - filters

RobotsTxt
```
public RobotsTxt(List<IRobotsTxtFilter> filters)
```
Creates a new robot txt object with the supplied filters.

Parameters:

filters - filters

Since:

3.0.0

RobotsTxt

public RobotsTxt(List<IRobotsTxtFilter> filters,
                 float crawlDelay)

RobotsTxt

public RobotsTxt(List<IRobotsTxtFilter> filters,
                 List<String> sitemapLocations)

RobotsTxt

public RobotsTxt(List<IRobotsTxtFilter> filters,
                 List<String> sitemapLocations,
                 float crawlDelay)

Method Detail
- getFilters
```
public List<IRobotsTxtFilter> getFilters()
```
  Gets all filters.
  
  Returns:
  
  filters (never null)
- getDisallowFilters
```
public List<IRobotsTxtFilter> getDisallowFilters()
```
  Gets "Disallow" filters.
  
  Returns:
  
  disallow filters (never null)
  
  Since:
  
  2.4.0
- getAllowFilters
```
public List<IRobotsTxtFilter> getAllowFilters()
```
  Gets "Allow" filters.
  
  Returns:
  
  allow filters (never null)
  
  Since:
  
  2.4.0
- getSitemapLocations
```
public List<String> getSitemapLocations()
```
- getCrawlDelay
```
public float getCrawlDelay()
```
- equals
```
public boolean equals(Object other)
```
  Overrides:
  
  equals in class Object
- hashCode
```
public int hashCode()
```
  Overrides:
  
  hashCode in class Object
- toString
```
public String toString()
```
  Overrides:
  
  toString in class Object