com.norconex.importer.handler.tagger.impl

Class TextStatisticsTagger

java.lang.Object

com.norconex.importer.handler.AbstractImporterHandler

com.norconex.importer.handler.tagger.AbstractDocumentTagger

com.norconex.importer.handler.tagger.AbstractCharStreamTagger

com.norconex.importer.handler.tagger.impl.TextStatisticsTagger

All Implemented Interfaces:

IXMLConfigurable, IImporterHandler, IDocumentTagger

public class TextStatisticsTagger
extends AbstractCharStreamTagger
implements IXMLConfigurable

Analyzes the content of the supplied document and adds statistical information about its content or field as metadata fields. Default behavior provide the statistics about the content. Refer to the following for the new metadata fields to be created along with their description.

Statistic fields

Field name	Description
document.stat.characterCount	Total number of characters (excluding carriage returns/line feed).
document.stat.wordCount	Total number of words.
document.stat.sentenceCount	Total number of sentences.
document.stat.paragraphCount	Total number of paragraph.
document.stat.averageWordCharacterCount	Average number of character in every words.
document.stat.averageSentenceCharacterCount	Average number of character in sentences (including non-word characters, such as spaces, or slashes).
document.stat.averageSentenceWordCount	Average number of words per sentences.
document.stat.averageParagraphCharacterCount	Average number of characters in paragraphs (including non-word characters, such as spaces, or slashes).
document.stat.averageParagraphSentenceCount	Average number of sentences per paragraphs.
document.stat.averageParagraphWordCount	Average number of words per paragraphs.

You can specify a field name to obtain statistics about that field instead. When you do so, the field name will be inserted in the above names, right after "document.stat.". E.g.: document.stat.myfield.characterCount

Can be used both as a pre-parse (text-only) or post-parse handler.

XML configuration usage:

  <tagger class="com.norconex.importer.handler.tagger.impl.TextStatisticsTagger"
          sourceCharset="(character encoding)"
          fieldName="(optional field name instead of using content)" >
      
      <restrictTo caseSensitive="[false|true]"
              field="(name of header/metadata field name to match)">
          (regular expression of value to match)
      </restrictTo>
      <!-- multiple "restrictTo" tags allowed (only one needs to match) -->
  </tagger>
 

Usage example:

The following store the statistics in a field called "statistics".

  <tagger class="com.norconex.importer.handler.tagger.impl.TextStatisticsTagger"
          fieldName="statistics" />
 

Since:

2.0.0

Author:

Pascal Essiembre

Constructor Summary

Constructors

Constructor and Description
TextStatisticsTagger()

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object other)
String	getFieldName()
int	hashCode()
protected void	loadCharStreamTaggerFromXML(org.apache.commons.configuration.XMLConfiguration xml) Loads configuration settings specific to the implementing class.
protected void	saveCharStreamTaggerToXML(EnhancedXMLStreamWriter writer) Saves configuration settings specific to the implementing class.
void	setFieldName(String fieldName)
protected void	tagTextDocument(String reference, Reader input, ImporterMetadata metadata, boolean parsed)
String	toString()

Methods inherited from class com.norconex.importer.handler.tagger.AbstractCharStreamTagger

getSourceCharset, loadHandlerFromXML, saveHandlerToXML, setSourceCharset, tagApplicableDocument

Methods inherited from class com.norconex.importer.handler.tagger.AbstractDocumentTagger

tagDocument

Methods inherited from class com.norconex.importer.handler.AbstractImporterHandler

addRestriction, addRestriction, addRestrictions, clearRestrictions, detectCharsetIfBlank, getRestrictions, isApplicable, loadFromXML, removeRestriction, removeRestriction, saveToXML

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.norconex.commons.lang.config.IXMLConfigurable

loadFromXML, saveToXML

Constructor Detail

TextStatisticsTagger

public TextStatisticsTagger()

Method Detail

tagTextDocument

protected void tagTextDocument(String reference,
                               Reader input,
                               ImporterMetadata metadata,
                               boolean parsed)
                        throws ImporterHandlerException

Specified by:

tagTextDocument in class AbstractCharStreamTagger

Throws:

ImporterHandlerException

getFieldName

public String getFieldName()

setFieldName

public void setFieldName(String fieldName)

loadCharStreamTaggerFromXML

protected void loadCharStreamTaggerFromXML(org.apache.commons.configuration.XMLConfiguration xml)
                                    throws IOException

Description copied from class: AbstractCharStreamTagger

Loads configuration settings specific to the implementing class.

Specified by:

loadCharStreamTaggerFromXML in class AbstractCharStreamTagger

Parameters:

xml - xml configuration

Throws:

IOException - could not load from XML

saveCharStreamTaggerToXML

protected void saveCharStreamTaggerToXML(EnhancedXMLStreamWriter writer)
                                  throws XMLStreamException

Description copied from class: AbstractCharStreamTagger

Saves configuration settings specific to the implementing class. The parent tag along with the "class" attribute are already written. Implementors must not close the writer.

Specified by:

saveCharStreamTaggerToXML in class AbstractCharStreamTagger

Parameters:

writer - the xml writer

Throws:

XMLStreamException - could not save to XML

equals

public boolean equals(Object other)

Overrides:

equals in class AbstractCharStreamTagger

hashCode

public int hashCode()

Overrides:

hashCode in class AbstractCharStreamTagger

toString

public String toString()

Overrides:

toString in class AbstractCharStreamTagger