com.norconex.collector.http.recrawl.impl

Class GenericRecrawlableResolver

java.lang.Object

com.norconex.collector.http.recrawl.impl.GenericRecrawlableResolver

All Implemented Interfaces:

IRecrawlableResolver, IXMLConfigurable

public class GenericRecrawlableResolver
extends Object
implements IRecrawlableResolver, IXMLConfigurable

Relies on both sitemap directives and custom instructions for establishing the minimum frequency between each document recrawl.

Sitemap support:

Provided crawler support for sitemaps has not been disabled, this class tries to honor last modified and frequency directives found in sitemap files.

By default, existing sitemap directives take precedence over custom ones. You chose to have sitemap directives be considered last or even disable sitemap directives using the setSitemapSupport(SitemapSupport) method.

Custom recrawl frequencies:

You can chose to have some of your crawled documents be re-crawled less frequently than others by specifying custom minimum frequencies (setMinFrequencies(MinFrequency...)). Minimum frequencies are processed in the order specified and must each have to following:

• applyTo: Either "reference" or "contentType" (defaults to "reference").
• pattern: A regular expression.
• value: one of "always", "hourly", "daily", "weekly", "monthly", "yearly", "never", or a numeric value in milliseconds.

As of 2.7.0, XML configuration entries expecting millisecond durations can be provided in human-readable format (English only), as per DurationParser (e.g., "5 minutes and 30 seconds" or "5m30s").

XML configuration usage:

  <recrawlableResolver
         class="com.norconex.collector.http.recrawl.impl.GenericRecrawlableResolver"
         sitemapSupport="[first|last|never]" >

     <minFrequency applyTo="[reference|contentType]" caseSensitive="[false|true]"
             value="([always|hourly|daily|weekly|monthly|yearly|never] or milliseconds)" >
         (regex pattern)
     </minFrequency>
     (... repeat frequency tag as needed ...)

  </recrawlableResolver>
 

Usage example:

The following example ensures PDFs recrawled no more frequently than once a month, while HTML news can be crawled as fast at every half hour. For the rest, it relies on the website sitemap directives (if any).

  <recrawlableResolver
         class="com.norconex.collector.http.recrawl.impl.GenericRecrawlableResolver"
         sitemapSupport="last" >
     <minFrequency applyTo="contentType" value="monthly">application/pdf</minFrequency>
     <minFrequency applyTo="reference" value="1800000">.*latest-news.*\.html</minFrequency>
  </recrawlableResolver>
 

Since:

2.5.0

Author:

Pascal Essiembre

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GenericRecrawlableResolver.MinFrequency
static class	GenericRecrawlableResolver.SitemapSupport

Constructor Summary

Constructors

Constructor and Description
GenericRecrawlableResolver()

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object other)
GenericRecrawlableResolver.MinFrequency[]	getMinFrequencies()
GenericRecrawlableResolver.SitemapSupport	getSitemapSupport() Gets the sitemap support strategy.
int	hashCode()
boolean	isRecrawlable(PreviousCrawlData prevData) Whether a document recrawlable or not.
void	loadFromXML(Reader in)
void	saveToXML(Writer out)
void	setMinFrequencies(GenericRecrawlableResolver.MinFrequency... frequencies)
void	setSitemapSupport(GenericRecrawlableResolver.SitemapSupport sitemapSupport) Sets the sitemap support strategy.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

GenericRecrawlableResolver

public GenericRecrawlableResolver()

Method Detail

getSitemapSupport

public GenericRecrawlableResolver.SitemapSupport getSitemapSupport()

Gets the sitemap support strategy. Defualt is GenericRecrawlableResolver.SitemapSupport.FIRST.

Returns:

sitemap support strategy

setSitemapSupport

public void setSitemapSupport(GenericRecrawlableResolver.SitemapSupport sitemapSupport)

Sets the sitemap support strategy. A null value is equivalent to specifying the default GenericRecrawlableResolver.SitemapSupport.FIRST.

Parameters:

sitemapSupport - sitemap support strategy

getMinFrequencies

public GenericRecrawlableResolver.MinFrequency[] getMinFrequencies()

setMinFrequencies

public void setMinFrequencies(GenericRecrawlableResolver.MinFrequency... frequencies)

isRecrawlable

public boolean isRecrawlable(PreviousCrawlData prevData)

Description copied from interface: IRecrawlableResolver

Whether a document recrawlable or not.

Specified by:

isRecrawlable in interface IRecrawlableResolver

Parameters:

prevData - data about previously crawled document

Returns:

true if recrawlable

loadFromXML

public void loadFromXML(Reader in)
                 throws IOException

Specified by:

loadFromXML in interface IXMLConfigurable

Throws:

IOException

saveToXML

public void saveToXML(Writer out)
               throws IOException

Specified by:

saveToXML in interface IXMLConfigurable

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object