com.norconex.collector.http.fetch.impl

Class GenericMetadataFetcher

java.lang.Object

com.norconex.collector.http.fetch.impl.GenericMetadataFetcher

All Implemented Interfaces:

IHttpMetadataFetcher, IXMLConfigurable

public class GenericMetadataFetcher
extends Object
implements IHttpMetadataFetcher, IXMLConfigurable

Basic implementation of IHttpMetadataFetcher.

XML configuration usage:

  <metadataFetcher
      class="com.norconex.collector.http.fetch.impl.GenericMetadataFetcher"
      skipOnBadStatus="[false|true]" >
      <validStatusCodes>(defaults to 200)</validStatusCodes>
      <notFoundStatusCodes>(defaults to 404)</notFoundStatusCodes>
      <headersPrefix>(string to prefix headers)</headersPrefix>
  </metadataFetcher>
 

The "validStatusCodes" and "notFoundStatusCodes" elements expect a coma-separated list of HTTP response code. If a code is added in both elements, the valid list takes precedence.

The "notFoundStatusCodes" element was added in 2.6.0.

Usage example:

The following configures a crawler to use this fetcher with the default settings.

  <metadataFetcher
      class="com.norconex.collector.http.fetch.impl.GenericMetadataFetcher" />
 

Author:

Pascal Essiembre

Constructor Summary

Constructors

Constructor and Description
GenericMetadataFetcher()
GenericMetadataFetcher(int[] validStatusCodes)

Method Summary

Modifier and Type	Method and Description
protected org.apache.http.client.methods.HttpRequestBase	createUriRequest(String url) Creates the HTTP request to be executed.
boolean	equals(Object other)
HttpFetchResponse	fetchHTTPHeaders(org.apache.http.client.HttpClient httpClient, String url, Properties metadata) Fetches the HTTP headers for a URL and stores it in the provided Properties.
String	getHeadersPrefix()
int[]	getNotFoundStatusCodes() Gets HTTP status codes to be considered as "Not found" state.
int[]	getValidStatusCodes()
int	hashCode()
void	loadFromXML(Reader in)
void	saveToXML(Writer out)
void	setHeadersPrefix(String headersPrefix)
void	setNotFoundStatusCodes(int... notFoundStatusCodes) Sets HTTP status codes to be considered as "Not found" state.
void	setValidStatusCodes(int... validStatusCodes)
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

GenericMetadataFetcher

public GenericMetadataFetcher()

GenericMetadataFetcher

public GenericMetadataFetcher(int[] validStatusCodes)

Method Detail

getValidStatusCodes

public int[] getValidStatusCodes()

setValidStatusCodes

public void setValidStatusCodes(int... validStatusCodes)

getNotFoundStatusCodes

public int[] getNotFoundStatusCodes()

Gets HTTP status codes to be considered as "Not found" state. Default is 404.

Returns:

"Not found" codes

Since:

2.6.0

setNotFoundStatusCodes

public final void setNotFoundStatusCodes(int... notFoundStatusCodes)

Sets HTTP status codes to be considered as "Not found" state.

Parameters:

notFoundStatusCodes - "Not found" codes

Since:

2.6.0

getHeadersPrefix

public String getHeadersPrefix()

setHeadersPrefix

public void setHeadersPrefix(String headersPrefix)

fetchHTTPHeaders

public HttpFetchResponse fetchHTTPHeaders(org.apache.http.client.HttpClient httpClient,
                                          String url,
                                          Properties metadata)

Description copied from interface: IHttpMetadataFetcher

Fetches the HTTP headers for a URL and stores it in the provided Properties.

Specified by:

fetchHTTPHeaders in interface IHttpMetadataFetcher

Parameters:

httpClient - the HTTP Client

url - the url from which to fetch the headers

metadata - recipient for storing HTTP headers as metadata

Returns:

fetch response

createUriRequest

protected org.apache.http.client.methods.HttpRequestBase createUriRequest(String url)

Creates the HTTP request to be executed. Default implementation returns an HttpHead request around the provided URL. This method can be overwritten to return another type of request.

Parameters:

url - the URL to create the request for

Returns:

HTTP request

loadFromXML

public void loadFromXML(Reader in)

Specified by:

loadFromXML in interface IXMLConfigurable

saveToXML

public void saveToXML(Writer out)
               throws IOException

Specified by:

saveToXML in interface IXMLConfigurable

Throws:

IOException

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object