Package com.norconex.collector.http.recrawl.impl

Class GenericRecrawlableResolver

java.lang.Object
com.norconex.collector.http.recrawl.impl.GenericRecrawlableResolver
All Implemented Interfaces:
IRecrawlableResolver, IXMLConfigurable
public class GenericRecrawlableResolver extends Object implements IRecrawlableResolver, IXMLConfigurable

Relies on both sitemap directives and custom instructions for establishing the minimum frequency between each document recrawl.

Sitemap support:

Provided crawler support for sitemaps has not been disabled, this class tries to honor last modified and frequency directives found in sitemap files.

By default, existing sitemap directives take precedence over custom ones. You chose to have sitemap directives be considered last or even disable sitemap directives using the setSitemapSupport(SitemapSupport) method.

Custom recrawl frequencies:

You can chose to have some of your crawled documents be re-crawled less frequently than others by specifying custom minimum frequencies (setMinFrequencies(MinFrequency...)). Minimum frequencies are processed in the order specified and must each have to following:

  • applyTo: Either "reference" or "contentType" (defaults to "reference").
  • pattern: A regular expression.
  • value: one of "always", "hourly", "daily", "weekly", "monthly", "yearly", "never", or a numeric value in milliseconds.

As of 2.7.0, XML configuration entries expecting millisecond durations can be provided in human-readable format (English only), as per DurationParser (e.g., "5 minutes and 30 seconds" or "5m30s").

XML configuration usage:


<recrawlableResolver
    class="com.norconex.collector.http.recrawl.impl.GenericRecrawlableResolver"
    sitemapSupport="[first|last|never]">
  <minFrequency
      applyTo="[reference|contentType]"
      caseSensitive="[false|true]"
      value="([always|hourly|daily|weekly|monthly|yearly|never] or milliseconds)">
    (regex pattern)
  </minFrequency>
  (... repeat frequency tag as needed ...)
</recrawlableResolver>

XML usage example:


<recrawlableResolver
    class="com.norconex.collector.http.recrawl.impl.GenericRecrawlableResolver"
    sitemapSupport="last">
  <minFrequency
      applyTo="contentType"
      value="monthly">
    application/pdf
  </minFrequency>
  <minFrequency
      applyTo="reference"
      value="1800000">
    .*latest-news.*\.html
  </minFrequency>
</recrawlableResolver>

The above example ensures PDFs are re-crawled no more frequently than once a month, while HTML news can be re-crawled as fast at every half hour. For the rest, it relies on the website sitemap directives (if any).

Since:
2.5.0
Author:
Pascal Essiembre