com.norconex.collector.http.sitemap.impl

Class StandardSitemapResolver

java.lang.Object

com.norconex.collector.http.sitemap.impl.StandardSitemapResolver

All Implemented Interfaces:

ISitemapResolver

public class StandardSitemapResolver
extends Object
implements ISitemapResolver

Implementation of ISitemapResolver as per sitemap.xml standard defined at http://www.sitemaps.org/protocol.html.

Sitemaps are only resolved if they have not been resolved already for the same URL "root" (the protocol, host and possible port).

The Sitemap specifications dictates that a sitemap.xml file defined in a sub-directory applies only to URLs found in that sub-directory and its children. This behavior is respected by default. Setting lenient to true no longer honors this restriction. Since 2.9.1, setting lenient will also attempt to parse XML values with invalid entities.

Since 2.3.0

Paths relative to URL roots can be specified and an attempt will be made to load and parse any sitemap found at those locations for each root URLs encountered (except for "start URLs" sitemaps, see below). Default paths are /sitemap.xml and /sitemap_index.xml. Setting null or an empty path array on setSitemapPaths(String...) will prevent attempts to locate sitemaps and only sitemaps found in robots.txt or defined as start URLs will be considered.

Sitemaps can be specified as "start URLs" (defined in HttpCrawlerConfig.getStartSitemapURLs()). Sitemaps defined that way will be the only ones resolved for the root URL they represent (sitemap paths or sitemaps defined in robots.txt won't apply).

Sitemaps are first stored in a local temporary file before being parsed. The tempDir constructor argument is used as the location where to store those files. When null, the system temporary directory is used, as returned by FileUtils.getTempDirectoryPath().

Author:

Pascal Essiembre

Field Summary

Fields

Modifier and Type	Field and Description
static String[]	DEFAULT_SITEMAP_PATHS

Constructor Summary

Constructors

Constructor and Description
StandardSitemapResolver(File tempDir, SitemapStore sitemapStore)

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object other)
long	getFromDate() Gets the minimum EPOCH date (in milliseconds) a sitemap entry should have to be considered.
String[]	getSitemapLocations() Deprecated. Since 2.3.0, use HttpCrawlerConfig.getStartSitemapURLs()
String[]	getSitemapPaths() Gets the URL paths, relative to the URL root, from which to try locate and resolve sitemaps.
File	getTempDir() Gets the directory where temporary sitemap files are written.
int	hashCode()
boolean	isEscalateErrors() Gets whether errors should be thrown instead of logged.
boolean	isLenient()
void	resolveSitemaps(org.apache.http.client.HttpClient httpClient, String urlRoot, String[] sitemapLocations, SitemapURLAdder sitemapURLAdder, boolean startURLs) Resolves the sitemap instructions for a URL "root" (e.g.
void	setEscalateErrors(boolean escalateErrors) Sets whether errors should be thrown instead of logged.
void	setFromDate(long fromDate) Sets the minimum EPOCH date (in milliseconds) a sitemap entry should have to be considered.
void	setLenient(boolean lenient)
void	setSitemapLocations(String... sitemapLocations) Deprecated. Since 2.3.0, use HttpCrawlerConfig.setStartSitemapURLs(String[])
void	setSitemapPaths(String... sitemapPaths) Sets the URL paths, relative to the URL root, from which to try locate and resolve sitemaps.
void	setTempDir(File tempDir) Sets the directory where temporary sitemap files are written.
void	stop() Stops any ongoing sitemap resolution.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_SITEMAP_PATHS

public static final String[] DEFAULT_SITEMAP_PATHS

Constructor Detail

StandardSitemapResolver

public StandardSitemapResolver(File tempDir,
                               SitemapStore sitemapStore)

Method Detail

getSitemapPaths

public String[] getSitemapPaths()

Gets the URL paths, relative to the URL root, from which to try locate and resolve sitemaps. Default paths are "/sitemap.xml" and "/sitemap-index.xml".

Returns:

sitemap paths.

Since:

2.3.0

setSitemapPaths

public void setSitemapPaths(String... sitemapPaths)

Sets the URL paths, relative to the URL root, from which to try locate and resolve sitemaps.

Parameters:

sitemapPaths - sitemap paths.

Since:

2.3.0

resolveSitemaps

public void resolveSitemaps(org.apache.http.client.HttpClient httpClient,
                            String urlRoot,
                            String[] sitemapLocations,
                            SitemapURLAdder sitemapURLAdder,
                            boolean startURLs)

Description copied from interface: ISitemapResolver

Resolves the sitemap instructions for a URL "root" (e.g. http://www.example.com).

Specified by:

resolveSitemaps in interface ISitemapResolver

Parameters:

httpClient - the http client to use to stream Internet files if needed

urlRoot - the URL root for which to resolve the sitemap

sitemapLocations - sitemap locations to resolve

sitemapURLAdder - where to store retrieved site map URLs

startURLs - whether the sitemapLocations provided (if any) are start URLs (defined in HttpCrawlerConfig.getStartSitemapURLs())

getSitemapLocations

@Deprecated
public String[] getSitemapLocations()

Deprecated. Since 2.3.0, use HttpCrawlerConfig.getStartSitemapURLs()

Get sitemap locations.

Returns:

sitemap locations

setSitemapLocations

@Deprecated
public void setSitemapLocations(String... sitemapLocations)

Deprecated. Since 2.3.0, use HttpCrawlerConfig.setStartSitemapURLs(String[])

Set sitemap locations.

Parameters:

sitemapLocations - sitemap locations

isLenient

public boolean isLenient()

setLenient

public void setLenient(boolean lenient)

getFromDate

public long getFromDate()

Gets the minimum EPOCH date (in milliseconds) a sitemap entry should have to be considered.

Returns:

from date

Since:

2.9.1

setFromDate

public void setFromDate(long fromDate)

Sets the minimum EPOCH date (in milliseconds) a sitemap entry should have to be considered.

Parameters:

fromDate - from date

Since:

2.9.1

isEscalateErrors

public boolean isEscalateErrors()

Gets whether errors should be thrown instead of logged.

Returns:

true if throwing errors

Since:

2.9.1

setEscalateErrors

public void setEscalateErrors(boolean escalateErrors)

Sets whether errors should be thrown instead of logged.

Parameters:

escalateErrors - true if throwing errors

Since:

2.9.1

getTempDir

public File getTempDir()

Gets the directory where temporary sitemap files are written.

Returns:

directory

Since:

2.3.0

setTempDir

public void setTempDir(File tempDir)

Sets the directory where temporary sitemap files are written.

Parameters:

tempDir - directory

Since:

2.3.0

stop

public void stop()

Description copied from interface: ISitemapResolver

Stops any ongoing sitemap resolution. Some sitemaps can be huge, and they may take a while to process. Upon the crawler receiving a stop request, this method will be invoked and implementors should try to exit cleanly without much delay.

Specified by:

stop in interface ISitemapResolver

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object