com.norconex.collector.http.crawler.event.impl

Class URLStatusCrawlerEventListener

java.lang.Object

com.norconex.collector.http.crawler.event.impl.URLStatusCrawlerEventListener

All Implemented Interfaces:

ICrawlerEventListener, IXMLConfigurable

public class URLStatusCrawlerEventListener
extends Object
implements ICrawlerEventListener, IXMLConfigurable

Store on file all URLs that were "fetched", along with their HTTP response code, usually for reporting purposes (e.g. finding broken links). A short summary of all HTTP status codes can be found here.

Filter by status codes

By default, the status of all fetched URLs are stored by this listener, regardless what were those statuses. This can generate very lengthy reports on large crawls. If you are only interested in certain status codes, you can listen only for those using the setStatusCodes(String) method or XML configuration equivalent. You specify the codes you want to listen for as coma-separated values. Ranges are also supported: specify two range values (both inclusive) separated by an hyphen. For instance, if you want to store all "bad" URLs, you can quickly specify all codes except 200 (OK) this way:

   100-199,201-599

Output location

The generated report will be stored in the directory specified by using setOutputDir(String). By default, the file generated will use this naming pattern:

   urlstatuses-[crawlerId]-[timestamp].tsv
 

The filename prefix can be changed from "urlstatuses-" to anything else using setFileNamePrefix(String).

Referring/parent URLs and custom link extractor

To capture the referring pages you have to use a link extractor that extracts referrer information. The default link extractor GenericLinkExtractor properly extracts this information. Same with TikaLinkExtractor. This is only a consideration when using a custom link extractor.

XML configuration usage:

  <listener  
      class="com.norconex.collector.http.crawler.event.impl.URLStatusCrawlerEventListener">
      <statusCodes>(CSV list of status codes)</statusCodes>
      <outputDir>(path to a directory of your choice)</outputDir>
      <fileNamePrefix>(report file name prefix)</fileNamePrefix>
  </listener>
 

Usage example:

The following example will generate a broken links report by recording 404 status codes (from HTTP response).

  <listener  
      class="com.norconex.collector.http.crawler.event.impl.URLStatusCrawlerEventListener">
      <statusCodes>404</statusCodes>
      <outputDir>/report/path/</outputDir>
      <fileNamePrefix>brokenLinks</fileNamePrefix>
  </listener>
 

Since:

2.2.0

Author:

Pascal Essiembre

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_FILENAME_PREFIX

Constructor Summary

Constructors

Constructor and Description
URLStatusCrawlerEventListener()

Method Summary

Modifier and Type	Method and Description
void	crawlerEvent(ICrawler crawler, CrawlerEvent event)
boolean	equals(Object other)
String	getFileNamePrefix() Gets the generated report file name prefix.
String	getOutputDir() Gets the local directory where this listener report will be written.
String	getStatusCodes() Gets the status codes to listen for.
int	hashCode()
void	loadFromXML(Reader in)
void	saveToXML(Writer out)
void	setFileNamePrefix(String fileNamePrefix) Sets the generated report file name prefix.
void	setOutputDir(String outputDir) Sets the local directory where this listener report will be written.
void	setStatusCodes(String statusCodes) Sets a coma-separated list of status codes to listen to.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_FILENAME_PREFIX

public static final String DEFAULT_FILENAME_PREFIX

See Also:

Constant Field Values

Constructor Detail

URLStatusCrawlerEventListener

public URLStatusCrawlerEventListener()

Method Detail

getStatusCodes

public String getStatusCodes()

Gets the status codes to listen for. Default is null (listens for all status codes).

Returns:

status codes

setStatusCodes

public void setStatusCodes(String statusCodes)

Sets a coma-separated list of status codes to listen to. See class documentation for how to specify code ranges.

Parameters:

statusCodes - the status codes to listen for

getOutputDir

public String getOutputDir()

Gets the local directory where this listener report will be written.

Returns:

directory path

setOutputDir

public void setOutputDir(String outputDir)

Sets the local directory where this listener report will be written.

Parameters:

outputDir - directory path

getFileNamePrefix

public String getFileNamePrefix()

Gets the generated report file name prefix. See class documentation for default prefix.

Returns:

file name prefix

setFileNamePrefix

public void setFileNamePrefix(String fileNamePrefix)

Sets the generated report file name prefix.

Parameters:

fileNamePrefix - file name prefix

crawlerEvent

public void crawlerEvent(ICrawler crawler,
                         CrawlerEvent event)

Specified by:

crawlerEvent in interface ICrawlerEventListener

loadFromXML

public void loadFromXML(Reader in)
                 throws IOException

Specified by:

loadFromXML in interface IXMLConfigurable

Throws:

IOException

saveToXML

public void saveToXML(Writer out)
               throws IOException

Specified by:

saveToXML in interface IXMLConfigurable

Throws:

IOException

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object