com.norconex.collector.http.redirect.impl

Class GenericRedirectURLProvider

java.lang.Object

com.norconex.collector.http.redirect.impl.GenericRedirectURLProvider

All Implemented Interfaces:

IRedirectURLProvider, IXMLConfigurable

public class GenericRedirectURLProvider
extends Object
implements IRedirectURLProvider, IXMLConfigurable

Provide redirect URLs by grabbing them from the HTTP Response Location header value. The URL is made absolute and an attempt is made to fix possible character encoding issues.

The RFC 2616 specification mentions that the Location header should contain a URI as defined by the RFC 1630 specification. The later requires that a URI be 7-bit ASCII with any special characters URL encoded. Some redirect URLs do not conform to that so we apply the following logic in an attempt to fix them:

• Does the URL contains only ASCII characters (code points <= 128)?
  • Yes: No attempt to fix it is made.
  • No: Does the HTTP response specify a character encoding in the Content-Type header?
    • Yes: Treat URL as being encoded in that character encoding.
    • No: Is the fallback character encoding set?
      • Yes: Treat URL as being encoded with the fallback character encoding.
      • No: Treat URL as being encoded with UTF-8.

XML configuration usage:

  <redirectURLProvider 
      class="com.norconex.collector.http.redirect.impl.GenericRedirectURLProvider"
      fallbackCharset="(character encoding)" />
 

Usage example:

The following sets the default character encoding to be "ISO-8859-1" when it could not be detected.

  <redirectURLProvider fallbackCharset="ISO-8859-1" />
 

Since:

2.4.0

Author:

Pascal Essiembre

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_FALLBACK_CHARSET

Constructor Summary

Constructors

Constructor and Description
GenericRedirectURLProvider()

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object obj)
String	getFallbackCharset()
int	hashCode()
void	loadFromXML(Reader in)
String	provideRedirectURL(org.apache.http.HttpRequest request, org.apache.http.HttpResponse response, org.apache.http.protocol.HttpContext context) Provides the redirect URL that the crawler must follow.
void	saveToXML(Writer out)
void	setFallbackCharset(String fallbackCharset)
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_FALLBACK_CHARSET

public static final String DEFAULT_FALLBACK_CHARSET

Constructor Detail

GenericRedirectURLProvider

public GenericRedirectURLProvider()

Method Detail

getFallbackCharset

public String getFallbackCharset()

setFallbackCharset

public void setFallbackCharset(String fallbackCharset)

provideRedirectURL

public String provideRedirectURL(org.apache.http.HttpRequest request,
                                 org.apache.http.HttpResponse response,
                                 org.apache.http.protocol.HttpContext context)

Description copied from interface: IRedirectURLProvider

Provides the redirect URL that the crawler must follow. This method is only invoked when a redirect has been detected. As such, it should rarely return null. Returning null effectively prevents a redirect from happening, but it is an efficient way to disable redirects. The recommended approach to disable redirects is to set zero on GenericHttpClientFactory.setMaxRedirects(int)

Specified by:

provideRedirectURL in interface IRedirectURLProvider

Parameters:

request - the HTTP request that led to the redirect

response - original URL HTTP response

context - execution state of an HTTP process

Returns:

redirect URL

loadFromXML

public void loadFromXML(Reader in)

Specified by:

loadFromXML in interface IXMLConfigurable

saveToXML

public void saveToXML(Writer out)
               throws IOException

Specified by:

saveToXML in interface IXMLConfigurable

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

equals

public boolean equals(Object obj)

Overrides:

equals in class Object