Package com.norconex.importer.handler.tagger.impl

Class URLExtractorTagger

  • All Implemented Interfaces:
    IXMLConfigurable, IImporterHandler, IDocumentTagger
    public class URLExtractorTagger
extends AbstractCharStreamTagger
implements IXMLConfigurable

    Extracts unique URLs matching specific patterns in plain text content and store them in a given field.

    URL-matching patterns used are relatively simple. It looks for strings starting with http://, https://, or www.. The later is prefixed with https:// when encountered (to make it absolute).

    The matching is case-insensitive. If you need alternate ways to detect URLs, you can use a combination of RegexTagger, ReplaceTagger, or create your own implementation.

    Storing values in an existing field

    If a target field with the same name already exists for a document, values will be added to the end of the existing value list. It is possible to change this default behavior by supplying a PropertySetter.

    If no URLs are found, the target field values (if any) are left intact.

    Content source

    It is possible to specify a fromField as the source of the text to use instead of using the document content.

    This class is typically e used as a post-parsing handler only (to ensure we are dealing with text).

    XML configuration usage:

    
<handler
    class="com.norconex.importer.handler.tagger.impl.URLExtractorTagger"
    toField="(target field where to store extracted URLs)"
    maxReadSize="(max characters to read at once)"
    sourceCharset="(character encoding)">
  <!-- multiple "restrictTo" tags allowed (only one needs to match) -->
  <restrictTo>
    <fieldMatcher>(field-matching expression)</fieldMatcher>
    <valueMatcher>(value-matching expression)</valueMatcher>
  </restrictTo>
  <fieldMatcher>
    (Optional field of text to use. Default uses document content.)
  </fieldMatcher>
</handler>

    XML usage example:

    
<handler
    class="URLExtractorTagger"
    toField="documentURLs">
  <restrictTo>
    <fieldMatcher>document.contentType</fieldMatcher>
    <valueMatcher>application/pdf</valueMatcher>
  </restrictTo>
</handler>

    The above example is used as a post-parse handler. It detects URLs in parsed PDFs and store those URLs in a field call "documentURLs".

    Since:
    3.0.0
    Author:
    Pascal Essiembre