com.norconex.collector.http.delay.impl

Class GenericDelayResolver

java.lang.Object

com.norconex.collector.http.delay.impl.AbstractDelayResolver

com.norconex.collector.http.delay.impl.GenericDelayResolver

All Implemented Interfaces:

IDelayResolver, IXMLConfigurable

public class GenericDelayResolver
extends AbstractDelayResolver

Default implementation for creating voluntary delays between URL downloads. There are a few ways the actual delay value can be defined (in order):

1. Takes the delay specify by a robots.txt file. Only applicable if robots.txt files and its robots crawl delays are not ignored.
2. Takes an explicitly scheduled delay, if any (picks the first one matching).
3. Use the specified default delay or 3 seconds, if none is specified.

In a delay schedule, the days of weeks are spelled out (in English): Monday, Tuesday, etc. Time ranges are using the 24h format.

One of these following scope dictates how the delay is applied, listed in order from the best behaved to the least.

• crawler: the delay is applied between each URL download within a crawler instance, regardless how many threads are defined within that crawler, or whether URLs are from the same site or not. This is the default scope.
• site: the delay is applied between each URL download from the same site within a crawler instance, regardless how many threads are defined. A site is defined by a URL protocol and its domain (e.g. http://example.com).
• thread: the delay is applied between each URL download from any given thread. The more threads you have the less of an impact the delay will have.

As of 2.7.0, XML configuration entries expecting millisecond durations can be provided in human-readable format (English only), as per DurationParser (e.g., "5 minutes and 30 seconds" or "5m30s").

XML configuration usage:

  <delay class="com.norconex.collector.http.delay.impl.GenericDelayResolver"
          default="(milliseconds)" 
          ignoreRobotsCrawlDelay="[false|true]"
          scope="[crawler|site|thread]" >
      <schedule 
          dayOfWeek="from (week day) to (week day)"
          dayOfMonth="from [1-31] to [1-31]"
          time="from (HH:mm) to (HH:mm)">
        (delay in milliseconds)
      </schedule>
       
      (... repeat schedule tag as needed ...)
  </delay>
 

Usage example:

The following set the minimum delay between each document download on a given site to 5 seconds, no matter what the crawler robots.txt may say, except on weekend, where it is more agressive (1 second).

  <delay class="com.norconex.collector.http.delay.impl.GenericDelayResolver"
          default="5 seconds" ignoreRobotsCrawlDelay="true" scope="site" >
      <schedule dayOfWeek="from Saturday to Sunday">1 second</schedule>
  </delay>
 

Author:

Pascal Essiembre

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GenericDelayResolver.DelaySchedule

Field Summary

Fields inherited from class com.norconex.collector.http.delay.impl.AbstractDelayResolver

DEFAULT_DELAY, SCOPE_CRAWLER, SCOPE_SITE, SCOPE_THREAD

Constructor Summary

Constructors

Constructor and Description
GenericDelayResolver()

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object other)
List<GenericDelayResolver.DelaySchedule>	getSchedules()
int	hashCode()
protected void	loadDelaysFromXML(XMLConfiguration xml) Loads explicit configuration of delays form XML.
protected long	resolveExplicitDelay(String url) Resolves explicitly specified delay, in milliseconds.
protected void	saveDelaysToXML(EnhancedXMLStreamWriter writer) Saves explicit configuration of delays to XML.
void	setSchedules(List<GenericDelayResolver.DelaySchedule> schedules)
String	toString()

Methods inherited from class com.norconex.collector.http.delay.impl.AbstractDelayResolver

delay, getDefaultDelay, getScope, isIgnoreRobotsCrawlDelay, loadFromXML, saveToXML, setDefaultDelay, setIgnoreRobotsCrawlDelay, setScope

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

GenericDelayResolver

public GenericDelayResolver()

Method Detail

getSchedules

public List<GenericDelayResolver.DelaySchedule> getSchedules()

setSchedules

public void setSchedules(List<GenericDelayResolver.DelaySchedule> schedules)

resolveExplicitDelay

protected long resolveExplicitDelay(String url)

Description copied from class: AbstractDelayResolver

Resolves explicitly specified delay, in milliseconds. This method is only invoked when there are no delays from robots.txt. If the implementing class does not have a delay resolution, -1 is returned (the default delay will be used).

Specified by:

resolveExplicitDelay in class AbstractDelayResolver

Parameters:

url - URL for which to resolve delay

Returns:

delay in millisecond, or -1

loadDelaysFromXML

protected void loadDelaysFromXML(XMLConfiguration xml)
                          throws IOException

Description copied from class: AbstractDelayResolver

Loads explicit configuration of delays form XML. Implementors should override this method if they wish to add extra configurable elements. Default implementation does nothing.

Overrides:

loadDelaysFromXML in class AbstractDelayResolver

Parameters:

xml - configuration

Throws:

IOException - problem loading delays

saveDelaysToXML

protected void saveDelaysToXML(EnhancedXMLStreamWriter writer)
                        throws IOException

Description copied from class: AbstractDelayResolver

Saves explicit configuration of delays to XML. Implementors should override this method if they wish to add extra configurable elements. Default implementation does nothing.

Overrides:

saveDelaysToXML in class AbstractDelayResolver

Parameters:

writer - a writer

Throws:

IOException - problem saving delays

equals

public boolean equals(Object other)

Overrides:

equals in class AbstractDelayResolver

hashCode

public int hashCode()

Overrides:

hashCode in class AbstractDelayResolver

toString

public String toString()

Overrides:

toString in class AbstractDelayResolver