com.norconex.importer.handler.filter.impl

Class DOMContentFilter

java.lang.Object

com.norconex.importer.handler.AbstractImporterHandler

com.norconex.importer.handler.filter.AbstractDocumentFilter

com.norconex.importer.handler.filter.impl.DOMContentFilter

All Implemented Interfaces:

IXMLConfigurable, IDocumentFilter, IOnMatchFilter, IImporterHandler

public class DOMContentFilter
extends AbstractDocumentFilter

Uses a Document Object Model (DOM) representation of an HTML, XHTML, or XML document content to perform filtering based on matching an element/attribute or element/attribute value.

In order to construct a DOM tree, a document content is loaded entirely into memory. Use this filter with caution if you know you'll need to parse huge files. You can use RegexContentFilter instead if this is a concern.

The jsoup parser library is used to load a document content into a DOM tree. Elements are referenced using a CSS or JQuery-like syntax.

If an element is referenced without a value to match, its mere presence constitutes a match. If both an element and a regular expression is provided the element value will be retrieved and the regular expression will be applied against it for a match.

Refer to AbstractDocumentFilter for the inclusion/exclusion logic.

Should be used as a pre-parse handler.

Content-types

By default, this filter is restricted to (applies only to) documents matching the restrictions returned by CommonRestrictions.domContentTypes(). You can specify your own content types if you know they represent a file with HTML or XML-like markup tags. For documents that are incompatible, consider using RegexContentFilter instead.

Since 2.5.0, when used as a pre-parse handler, this class attempts to detect the content character encoding unless the character encoding was specified using setSourceCharset(String). Since document parsing converts content to UTF-8, UTF-8 is always assumed when used as a post-parse handler.

Since 2.5.0, it is possible to control what gets extracted exactly for matching purposes thanks to the "extract" argument of the new method setExtract(String). Version 2.6.0 introduced several more extract options. Possible values are:

• text: Default option when extract is blank. The text of the element, including combined children.
• html: Extracts an element inner HTML (including children).
• outerHtml: Extracts an element outer HTML (like "html", but includes the "current" tag).
• ownText: Extracts the text owned by this element only; does not get the combined text of all children.
• data: Extracts the combined data of a data-element (e.g. <script>).
• id: Extracts the ID attribute of the element (if any).
• tagName: Extract the name of the tag of the element.
• val: Extracts the value of a form element (input, textarea, etc).
• className: Extracts the literal value of the element's "class" attribute, which may include multiple class names, space separated.
• cssSelector: Extracts a CSS selector that will uniquely select (identify) this element.
• attr(attributeKey): Extracts the value of the element attribute matching your replacement for "attributeKey" (e.g. "attr(title)" will extract the "title" attribute).

Since 2.8.0, you can specify which parser to use when reading documents. The default is "html" and will normalize the content as HTML. This is generally a desired behavior, but this can sometimes have your selector fail. If you encounter this problem, try switching to "xml" parser, which does not attempt normalization on the content. The drawback with "xml" is you may not get all HTML-specific selector options to work. If you know you are dealing with XML to begin with, specifying "xml" should be a good option.

XML configuration usage:

  <filter class="com.norconex.importer.handler.filter.impl.DOMContentFilter"
          onMatch="[include|exclude]" 
          caseSensitive="[false|true]"
          sourceCharset="(character encoding)"          
          selector="(selector syntax)"
          parser="[html|xml]"
          extract="[text|html|outerHtml|ownText|data|tagName|val|className|cssSelector|attr(attributeKey)]" >
          
    <restrictTo caseSensitive="[false|true]"
            field="(name of header/metadata field name to match)">
        (regular expression of value to match)
    </restrictTo>
    <!-- multiple "restrictTo" tags allowed (only one needs to match) -->

    <regex>(optional regular expression matching selector extracted value)</regex>
  </filter>
 

Examples:

To exclude an HTML page that has one or more GIF images in it:

  <filter class="com.norconex.importer.handler.filter.impl.DOMContentFilter"
          selector="img[src$=.gif]" onMatch="exclude" />
 

To exclude an HTML page that has a paragraph tag with a class called "disclaimer" and a value containing "skip me":

  <filter class="com.norconex.importer.handler.filter.impl.DOMContentFilter"
          selector="p.disclaimer" onMatch="exclude" >
    <regex>\bskip me\b</regex>
  </filter>
 

Since:

2.4.0

Author:

Pascal Essiembre

Constructor Summary

Constructors

Constructor and Description
DOMContentFilter()
DOMContentFilter(String regex)
DOMContentFilter(String regex, OnMatch onMatch)
DOMContentFilter(String regex, OnMatch onMatch, boolean caseSensitive)

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object other)
String	getExtract() Gets what should be extracted for the value.
String	getParser() Gets the parser to use when creating the DOM-tree.
String	getRegex()
String	getSelector()
String	getSourceCharset() Gets the assumed source character encoding.
int	hashCode()
boolean	isCaseSensitive()
protected boolean	isDocumentMatched(String reference, InputStream input, ImporterMetadata metadata, boolean parsed)
protected void	loadFilterFromXML(org.apache.commons.configuration.XMLConfiguration xml)
protected void	saveFilterToXML(EnhancedXMLStreamWriter writer)
void	setCaseSensitive(boolean caseSensitive)
void	setExtract(String extract) Sets what should be extracted for the value.
void	setParser(String parser) Sets the parser to use when creating the DOM-tree.
void	setRegex(String regex)
void	setSelector(String selector)
void	setSourceCharset(String sourceCharset) Sets the assumed source character encoding.
String	toString()

Methods inherited from class com.norconex.importer.handler.filter.AbstractDocumentFilter

acceptDocument, getOnMatch, loadHandlerFromXML, saveHandlerToXML, setOnMatch

Methods inherited from class com.norconex.importer.handler.AbstractImporterHandler

addRestriction, addRestriction, addRestrictions, clearRestrictions, detectCharsetIfBlank, getRestrictions, isApplicable, loadFromXML, removeRestriction, removeRestriction, saveToXML

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

DOMContentFilter

public DOMContentFilter()

DOMContentFilter

public DOMContentFilter(String regex)

DOMContentFilter

public DOMContentFilter(String regex,
                        OnMatch onMatch)

DOMContentFilter

public DOMContentFilter(String regex,
                        OnMatch onMatch,
                        boolean caseSensitive)

Method Detail

getRegex

public String getRegex()

setRegex

public final void setRegex(String regex)

isCaseSensitive

public boolean isCaseSensitive()

setCaseSensitive

public void setCaseSensitive(boolean caseSensitive)

getSelector

public String getSelector()

setSelector

public void setSelector(String selector)

getExtract

public String getExtract()

Gets what should be extracted for the value. One of "text" (default), "html", or "outerHtml". null means this class will use the default ("text").

Returns:

what should be extracted for the value

Since:

2.5.0

setExtract

public void setExtract(String extract)

Sets what should be extracted for the value. One of "text" (default), "html", or "outerHtml". null means this class will use the default ("text").

Parameters:

extract - what should be extracted for the value

Since:

2.5.0

getSourceCharset

public String getSourceCharset()

Gets the assumed source character encoding.

Returns:

character encoding of the source to be transformed

Since:

2.5.0

setSourceCharset

public void setSourceCharset(String sourceCharset)

Sets the assumed source character encoding.

Parameters:

sourceCharset - character encoding of the source to be transformed

Since:

2.5.0

getParser

public String getParser()

Gets the parser to use when creating the DOM-tree.

Returns:

html (default) or xml.

Since:

2.8.0

setParser

public void setParser(String parser)

Sets the parser to use when creating the DOM-tree.

Parameters:

parser - html or xml.

Since:

2.8.0

isDocumentMatched

protected boolean isDocumentMatched(String reference,
                                    InputStream input,
                                    ImporterMetadata metadata,
                                    boolean parsed)
                             throws ImporterHandlerException

Specified by:

isDocumentMatched in class AbstractDocumentFilter

Throws:

ImporterHandlerException

saveFilterToXML

protected void saveFilterToXML(EnhancedXMLStreamWriter writer)
                        throws XMLStreamException

Specified by:

saveFilterToXML in class AbstractDocumentFilter

Throws:

XMLStreamException

loadFilterFromXML

protected void loadFilterFromXML(org.apache.commons.configuration.XMLConfiguration xml)
                          throws IOException

Specified by:

loadFilterFromXML in class AbstractDocumentFilter

Throws:

IOException

equals

public boolean equals(Object other)

Overrides:

equals in class AbstractDocumentFilter

hashCode

public int hashCode()

Overrides:

hashCode in class AbstractDocumentFilter

toString

public String toString()

Overrides:

toString in class AbstractDocumentFilter