com.norconex.collector.http.url.impl

Class GenericURLNormalizer

java.lang.Object

com.norconex.collector.http.url.impl.GenericURLNormalizer

All Implemented Interfaces:

IURLNormalizer, IXMLConfigurable

public class GenericURLNormalizer
extends Object
implements IURLNormalizer, IXMLConfigurable

Generic implementation of IURLNormalizer that should satisfy most URL normalization needs. This implementation relies on URLNormalizer. Please refer to it for complete documentation and examples.

This class is in effect by default. To skip its usage, you can explicitly set the URL Normalizer to null in the HttpCrawlerConfig, or you can disable it using setDisabled(boolean).

By default, this class removes the URL fragment and applies these RFC 3986 normalizations:

• Converting the scheme and host to lower case
• Capitalizing letters in escape sequences
• Decoding percent-encoded unreserved characters
• Removing the default port
• Encoding non-URI characters

To overwrite this default, you have to specify a new list of normalizations to apply, via the setNormalizations(Normalization...) method, or via XML configuration. Each normalizations is identified by a code name. The following is the complete code name list for supported normalizations. Click on any code name to get a full description from URLNormalizer:

• addDirectoryTrailingSlash (since 2.6.0)
• addDomainTrailingSlash (since 2.6.1)
• addWWW
• decodeUnreservedCharacters
• encodeNonURICharacters
• encodeSpaces
• lowerCaseSchemeHost
• removeDefaultPort
• removeDirectoryIndex
• removeDotSegments
• removeDuplicateSlashes
• removeEmptyParameters
• removeFragment
• removeSessionIds
• removeTrailingQuestionMark
• removeTrailingSlash (since 2.6.0)
• removeTrailingHash (since 2.7.0)
• removeWWW
• replaceIPWithDomainName
• secureScheme
• sortQueryParameters
• unsecureScheme
• upperCaseEscapeSequence
• removeQueryString (since 2.9.0)
• lowerCase (since 2.9.0)
• lowerCasePath (since 2.9.0)
• lowerCaseQuery (since 2.9.0)
• lowerCaseQueryParameterNames (since 2.9.0)
• lowerCaseQueryParameterValues (since 2.9.0)

In addition, this class allows you to specify any number of URL value replacements using regular expressions.

XML configuration usage:

  <urlNormalizer
      class="com.norconex.collector.http.url.impl.GenericURLNormalizer"
      disabled="[false|true]">
    <normalizations>
      (normalization code names, coma separated)
    </normalizations>
    <replacements>
      <replace>
         <match>(regex pattern to match)</match>
         <replacement>(optional replacement value, default to blank)</replacement>
      </replace>
      (... repeat replace tag  as needed ...)
    </replacements>
  </urlNormalizer>
 

Since 2.7.2, having an empty "normalizations" tag will effectively remove any normalizations rules previously set (like default ones). Not having the tag at all will keep existing/default normalizations.

Usage example:

The following adds a normalization to add "www." to URL domains when missing, to the default set of normalizations. It also add custom URL "search-and-replace" to remove any "&view=print" strings from URLs as well as replace "&type=summary" with "&type=full".

  <urlNormalizer class="com.norconex.collector.http.url.impl.GenericURLNormalizer">
    <normalizations>
        removeFragment, lowerCaseSchemeHost, upperCaseEscapeSequence,
        decodeUnreservedCharacters, removeDefaultPort,
        encodeNonURICharacters, addWWW
    </normalizations>
    <replacements>
      <replace><match>&amp;view=print</match></replace>
      <replace>
         <match>(&amp;type=)(summary)</match>
         <replacement>$1full</replacement>
      </replace>
    </replacements>
  </urlNormalizer>
 

Author:

Pascal Essiembre

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GenericURLNormalizer.Normalization
static class	GenericURLNormalizer.Replace

Constructor Summary

Constructors

Constructor and Description
GenericURLNormalizer()

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object other)
GenericURLNormalizer.Normalization[]	getNormalizations()
GenericURLNormalizer.Replace[]	getReplaces()
int	hashCode()
boolean	isDisabled() Whether this URL Normalizer is disabled or not.
void	loadFromXML(Reader in)
String	normalizeURL(String url) Normalize the given URL.
void	saveToXML(Writer out)
void	setDisabled(boolean disabled) Sets whether this URL Normalizer is disabled or not.
void	setNormalizations(GenericURLNormalizer.Normalization... normalizations)
void	setReplaces(GenericURLNormalizer.Replace... replaces)
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

GenericURLNormalizer

public GenericURLNormalizer()

Method Detail

normalizeURL

public String normalizeURL(String url)

Description copied from interface: IURLNormalizer

Normalize the given URL.

Specified by:

normalizeURL in interface IURLNormalizer

Parameters:

url - the URL to normalize

Returns:

the normalized URL

getNormalizations

public GenericURLNormalizer.Normalization[] getNormalizations()

setNormalizations

public void setNormalizations(GenericURLNormalizer.Normalization... normalizations)

getReplaces

public GenericURLNormalizer.Replace[] getReplaces()

setReplaces

public void setReplaces(GenericURLNormalizer.Replace... replaces)

isDisabled

public boolean isDisabled()

Whether this URL Normalizer is disabled or not.

Returns:

true if disabled

Since:

2.3.0

setDisabled

public void setDisabled(boolean disabled)

Sets whether this URL Normalizer is disabled or not.

Parameters:

disabled - true if disabled

Since:

2.3.0

loadFromXML

public void loadFromXML(Reader in)

Specified by:

loadFromXML in interface IXMLConfigurable

saveToXML

public void saveToXML(Writer out)
               throws IOException

Specified by:

saveToXML in interface IXMLConfigurable

Throws:

IOException

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object