com.norconex.importer.handler.tagger.impl

Class TextPatternTagger

java.lang.Object

com.norconex.importer.handler.AbstractImporterHandler

com.norconex.importer.handler.tagger.AbstractDocumentTagger

com.norconex.importer.handler.tagger.AbstractCharStreamTagger

com.norconex.importer.handler.tagger.AbstractStringTagger

com.norconex.importer.handler.tagger.impl.TextPatternTagger

All Implemented Interfaces:

IXMLConfigurable, IImporterHandler, IDocumentTagger

public class TextPatternTagger
extends AbstractStringTagger
implements IXMLConfigurable

Extracts and add all text values matching the regular expression provided in to a field provided explicitely, or also matching a regular expression. The target field is considered a multi-value field.

Since 2.8.0, it is now possible to extract both the field names and their values with regular expression. This is done by using match groups in your regular expressions (parenthesis). For each pattern you define, you can specify which match group hold the field name and which one holds the value. Specifying a field match group is optional if a field is provided. If no match groups are specified, a field is expected.

Since 2.8.0, case-sensitivity for regular expressions is now set on each patterns.

This class can be used as a pre-parsing handler on text documents only or a post-parsing handler.

XML configuration usage:

  <tagger class="com.norconex.importer.handler.tagger.impl.TextPatternTagger"
          sourceCharset="(character encoding)"
          maxReadSize="(max characters to read at once)" >
      
      <restrictTo caseSensitive="[false|true]"
              field="(name of header/metadata field name to match)">
          (regular expression of value to match)
      </restrictTo>
      <!-- multiple "restrictTo" tags allowed (only one needs to match) -->
      
      <pattern field="(target field name)" 
              fieldGroup="(field name match group index)"
              valueGroup="(field value match group index)"
              caseSensitive="[false|true]">
          (regular expression)
      </pattern>
      <!-- multiple pattern tags allowed -->
 
  </tagger>
 

Usage example:

The first pattern in the following example extracts what look like email addresses in to an "email" field (simplified regex). The second pattern extracts field names and values from "label" and "value" cells on a given HTML table:

  <tagger class="com.norconex.importer.handler.tagger.impl.TextPatternTagger" >
      <pattern field="emails">
          [A-Za-z0-9+_.-]+?@[a-zA-Z0-9.-]+
      </pattern>
      <pattern fieldGroup="1" valueGroup="2"><![CDATA[
        <tr><td class="label">(.*?)</td><td class="value">(.*?)</td></tr>
      ]]></pattern>
  </tagger>
 

Since:

2.3.0

Author:

Pascal Essiembre

Constructor Summary

Constructors

Constructor and Description
TextPatternTagger()

Method Summary

Modifier and Type	Method and Description
void	addPattern(RegexFieldExtractor... pattern) Adds one or more pattern that will extract matching field names/values.
void	addPattern(String field, String pattern) Adds a pattern that will extract the whole text matched into given field.
void	addPattern(String field, String pattern, int valueGroup) Adds a new pattern, which will extract the value from the specified group index upon matching.
boolean	equals(Object other)
List<RegexFieldExtractor>	getPatterns() Gets the patterns used to extract matching field names/values.
int	hashCode()
boolean	isCaseSensitive() Deprecated. Always false. Case sensitivity is now set from each pattern
protected void	loadStringTaggerFromXML(org.apache.commons.configuration.XMLConfiguration xml) Loads configuration settings specific to the implementing class.
protected void	saveStringTaggerToXML(EnhancedXMLStreamWriter writer) Saves configuration settings specific to the implementing class.
void	setCaseSensitive(boolean caseSensitive) Deprecated. Always false. Case sensitivity is now set on each pattern
void	setPattern(RegexFieldExtractor... pattern) Sets one or more patterns that will extract matching field names/values.
protected void	tagStringContent(String reference, StringBuilder content, ImporterMetadata metadata, boolean parsed, int sectionIndex)
String	toString()

Methods inherited from class com.norconex.importer.handler.tagger.AbstractStringTagger

getMaxReadSize, loadCharStreamTaggerFromXML, saveCharStreamTaggerToXML, setMaxReadSize, tagTextDocument

Methods inherited from class com.norconex.importer.handler.tagger.AbstractCharStreamTagger

getSourceCharset, loadHandlerFromXML, saveHandlerToXML, setSourceCharset, tagApplicableDocument

Methods inherited from class com.norconex.importer.handler.tagger.AbstractDocumentTagger

tagDocument

Methods inherited from class com.norconex.importer.handler.AbstractImporterHandler

addRestriction, addRestriction, addRestrictions, clearRestrictions, detectCharsetIfBlank, getRestrictions, isApplicable, loadFromXML, removeRestriction, removeRestriction, saveToXML

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.norconex.commons.lang.config.IXMLConfigurable

loadFromXML, saveToXML

Constructor Detail

TextPatternTagger

public TextPatternTagger()

Method Detail

tagStringContent

protected void tagStringContent(String reference,
                                StringBuilder content,
                                ImporterMetadata metadata,
                                boolean parsed,
                                int sectionIndex)

Specified by:

tagStringContent in class AbstractStringTagger

isCaseSensitive

@Deprecated
public boolean isCaseSensitive()

Deprecated. Always false. Case sensitivity is now set from each pattern

Gets whether to ignore case when matching text.

Returns:

true if case sensitive.

setCaseSensitive

@Deprecated
public void setCaseSensitive(boolean caseSensitive)

Deprecated. Always false. Case sensitivity is now set on each pattern

Sets whether to ignore case when matching text.

Parameters:

caseSensitive - true to consider character case

addPattern

public void addPattern(String field,
                       String pattern)

Adds a pattern that will extract the whole text matched into given field.

Parameters:

field - target field to store the matching pattern.

pattern - the pattern

addPattern

public void addPattern(String field,
                       String pattern,
                       int valueGroup)

Adds a new pattern, which will extract the value from the specified group index upon matching.

Parameters:

field - target field to store the matching pattern.

pattern - the pattern

valueGroup - which pattern group to return.

addPattern

public void addPattern(RegexFieldExtractor... pattern)

Adds one or more pattern that will extract matching field names/values.

Parameters:

pattern - field extractor pattern

setPattern

public void setPattern(RegexFieldExtractor... pattern)

Sets one or more patterns that will extract matching field names/values. Clears previously set pattterns.

Parameters:

pattern - field extractor pattern

getPatterns

public List<RegexFieldExtractor> getPatterns()

Gets the patterns used to extract matching field names/values.

Returns:

patterns

loadStringTaggerFromXML

protected void loadStringTaggerFromXML(org.apache.commons.configuration.XMLConfiguration xml)
                                throws IOException

Description copied from class: AbstractStringTagger

Loads configuration settings specific to the implementing class.

Specified by:

loadStringTaggerFromXML in class AbstractStringTagger

Parameters:

xml - xml configuration

Throws:

IOException - could not load from XML

saveStringTaggerToXML

protected void saveStringTaggerToXML(EnhancedXMLStreamWriter writer)
                              throws XMLStreamException

Description copied from class: AbstractStringTagger

Saves configuration settings specific to the implementing class. The parent tag along with the "class" attribute are already written. Implementors must not close the writer.

Specified by:

saveStringTaggerToXML in class AbstractStringTagger

Parameters:

writer - the xml writer

Throws:

XMLStreamException - could not save to XML

equals

public boolean equals(Object other)

Overrides:

equals in class AbstractStringTagger

hashCode

public int hashCode()

Overrides:

hashCode in class AbstractStringTagger

toString

public String toString()

Overrides:

toString in class AbstractStringTagger