Package com.norconex.collector.http.delay.impl

Class ReferenceDelayResolver

java.lang.Object
com.norconex.collector.http.delay.impl.AbstractDelayResolver
com.norconex.collector.http.delay.impl.ReferenceDelayResolver
All Implemented Interfaces:
IDelayResolver, IXMLConfigurable
public class ReferenceDelayResolver extends AbstractDelayResolver

Introduces different delays between document downloads based on matching document reference (URL) patterns. There are a few ways the actual delay value can be defined (in order):

  1. Takes the delay specify by a robots.txt file. Only applicable if robots.txt files and its robots crawl delays are not ignored.
  2. Takes the delay matching a reference pattern, if any (picks the first one matching).
  3. Used the specified default delay or 3 seconds, if none is specified.

One of these following scope dictates how the delay is applied, listed in order from the best behaved to the least.

  • crawler: the delay is applied between each URL download within a crawler instance, regardless how many threads are defined within that crawler, or whether URLs are from the same site or not. This is the default scope.
  • site: the delay is applied between each URL download from the same site within a crawler instance, regardless how many threads are defined. A site is defined by a URL protocol and its domain (e.g. http://example.com).
  • thread: the delay is applied between each URL download from any given thread. The more threads you have the less of an impact the delay will have.

As of 2.7.0, XML configuration entries expecting millisecond durations can be provided in human-readable format (English only), as per DurationParser (e.g., "5 minutes and 30 seconds" or "5m30s").

XML configuration usage:


<delay
    class="com.norconex.collector.http.delay.impl.ReferenceDelayResolver"
    default="(milliseconds)"
    ignoreRobotsCrawlDelay="[false|true]"
    scope="[crawler|site|thread]">
  <pattern
      delay="(delay in milliseconds)">
    (regular expression applied against document reference)
  </pattern>
  (... repeat pattern tag as needed ...)
</delay>

XML usage example:


<pre>
  <delay
      class="ReferenceDelayResolver"
      default="3 seconds">
    <pattern
        delay="10 seconds">
      .*\.pdf
    </pattern>
  </delay>

The above examlpe will increase the delay to 10 seconds when encountering PDFs from a default of 3 seconds.

Since:
2.5.0
Author:
Pascal Essiembre