com.norconex.collector.http.sitemap

Interface ISitemapResolver

All Known Implementing Classes:

StandardSitemapResolver

public interface ISitemapResolver

Given a URL root, resolve the corresponding sitemap(s), if any, and only if it has not yet been resolved for a crawling session.

Sitemaps URLs can be specified as "start URLs" (defined in HttpCrawlerConfig.getStartSitemapURLs()). It is up to the selected implementation to decide whether to treat sitemaps specified as start URLs any differently.

When ignoring sitemap (HttpCrawlerConfig.isIgnoreSitemap()), the selected sitemap resolver implementation will still be invoked for sitemaps specified as start URLs.

Sitemaps locations to resolved may also come from a site robots.txt (provided robots.txt files are not ignored).

Is it possible for implementations to not attempt to resolve sitemaps for some URLs. Refer to specific implementation for more details.

Author:

Pascal Essiembre

See Also:

StandardSitemapResolver

Method Summary

Modifier and Type	Method and Description
void	resolveSitemaps(org.apache.http.client.HttpClient httpClient, String urlRoot, String[] sitemapLocations, SitemapURLAdder sitemapURLAdder, boolean startURLs) Resolves the sitemap instructions for a URL "root" (e.g.
void	stop() Stops any ongoing sitemap resolution.

Method Detail

resolveSitemaps

void resolveSitemaps(org.apache.http.client.HttpClient httpClient,
                     String urlRoot,
                     String[] sitemapLocations,
                     SitemapURLAdder sitemapURLAdder,
                     boolean startURLs)

Resolves the sitemap instructions for a URL "root" (e.g. http://www.example.com).

Parameters:

httpClient - the http client to use to stream Internet files if needed

urlRoot - the URL root for which to resolve the sitemap

sitemapLocations - sitemap locations to resolve

sitemapURLAdder - where to store retrieved site map URLs

startURLs - whether the sitemapLocations provided (if any) are start URLs (defined in HttpCrawlerConfig.getStartSitemapURLs())

stop

void stop()

Stops any ongoing sitemap resolution. Some sitemaps can be huge, and they may take a while to process. Upon the crawler receiving a stop request, this method will be invoked and implementors should try to exit cleanly without much delay.