com.norconex.collector.http.filter.impl

Class SegmentCountURLFilter

java.lang.Object

com.norconex.importer.handler.filter.AbstractOnMatchFilter

com.norconex.collector.http.filter.impl.SegmentCountURLFilter

All Implemented Interfaces:

IDocumentFilter, IMetadataFilter, IReferenceFilter, IXMLConfigurable, IOnMatchFilter

public class SegmentCountURLFilter
extends AbstractOnMatchFilter
implements IReferenceFilter, IDocumentFilter, IMetadataFilter, IXMLConfigurable

Filters URL based based on the number of URL segments. A URL with a number of segments equal or more than the specified count will either be included or excluded, as specified.

By default segments are obtained by breaking the URL text at each forward slashes (/), starting after the host name. You can define different or additional segment separator characters.

When duplicate is true, it will count the maximum number of duplicate segments found.

XML configuration usage:

  <filter class="com.norconex.collector.http.filter.impl.SegmentCountURLFilter"
          onMatch="[include|exclude]"
          count="(numeric value)"
          duplicate="[false|true]"
          separator="(a regex identifying segment separator)" />
 

Usage example:

The following will reject URLs with more than 5 forward slashes after the domain.

  <filter class="com.norconex.collector.http.filter.impl.SegmentCountURLFilter"
          onMatch="exclude" count="5" />
 

Since:

1.2

Author:

Pascal Essiembre

See Also:

Pattern

Field Summary

Fields

Modifier and Type	Field and Description
static int	DEFAULT_SEGMENT_COUNT Default segment count.
static String	DEFAULT_SEGMENT_SEPARATOR_PATTERN Default segment separator pattern.

Constructor Summary

Constructors

Constructor and Description
SegmentCountURLFilter() Constructor.
SegmentCountURLFilter(int count) Constructor.
SegmentCountURLFilter(int count, OnMatch onMatch) Constructor.
SegmentCountURLFilter(int count, OnMatch onMatch, boolean duplicate) Constructor.

Method Summary

Modifier and Type	Method and Description
boolean	acceptDocument(ImporterDocument document)
boolean	acceptMetadata(String reference, Properties metadata)
boolean	acceptReference(String url)
boolean	equals(Object obj)
int	getCount()
String	getSeparator() Gets the segment separator pattern
int	hashCode()
boolean	isDuplicate()
void	loadFromXML(Reader in)
void	saveToXML(Writer out)
void	setCount(int count)
void	setDuplicate(boolean duplicate)
void	setSeparator(String separator)
String	toString()

Methods inherited from class com.norconex.importer.handler.filter.AbstractOnMatchFilter

getOnMatch, loadFromXML, saveToXML, setOnMatch

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_SEGMENT_SEPARATOR_PATTERN

public static final String DEFAULT_SEGMENT_SEPARATOR_PATTERN

Default segment separator pattern.

See Also:

Constant Field Values

DEFAULT_SEGMENT_COUNT

public static final int DEFAULT_SEGMENT_COUNT

Default segment count.

See Also:

Constant Field Values

Constructor Detail

SegmentCountURLFilter

public SegmentCountURLFilter()

Constructor.

SegmentCountURLFilter

public SegmentCountURLFilter(int count)

Constructor.

Parameters:

count - how many segment

SegmentCountURLFilter

public SegmentCountURLFilter(int count,
                             OnMatch onMatch)

Constructor.

Parameters:

count - how many segment

onMatch - what to do on match

SegmentCountURLFilter

public SegmentCountURLFilter(int count,
                             OnMatch onMatch,
                             boolean duplicate)

Constructor.

Parameters:

count - how many segment

onMatch - what to do on match

duplicate - whether to handle duplicates

Method Detail

getSeparator

public String getSeparator()

Gets the segment separator pattern

Returns:

the pattern

setSeparator

public final void setSeparator(String separator)

getCount

public int getCount()

setCount

public final void setCount(int count)

isDuplicate

public boolean isDuplicate()

setDuplicate

public final void setDuplicate(boolean duplicate)

acceptDocument

public boolean acceptDocument(ImporterDocument document)

Specified by:

acceptDocument in interface IDocumentFilter

acceptMetadata

public boolean acceptMetadata(String reference,
                              Properties metadata)

Specified by:

acceptMetadata in interface IMetadataFilter

acceptReference

public boolean acceptReference(String url)

Specified by:

acceptReference in interface IReferenceFilter

loadFromXML

public void loadFromXML(Reader in)

Specified by:

loadFromXML in interface IXMLConfigurable

saveToXML

public void saveToXML(Writer out)
               throws IOException

Specified by:

saveToXML in interface IXMLConfigurable

Throws:

IOException

toString

public String toString()

Overrides:

toString in class AbstractOnMatchFilter

hashCode

public int hashCode()

Overrides:

hashCode in class AbstractOnMatchFilter

equals

public boolean equals(Object obj)

Overrides:

equals in class AbstractOnMatchFilter