ExternalParser (Norconex Importer 2.11.0 API)

java.lang.Object
- com.norconex.importer.parser.impl.ExternalParser

All Implemented Interfaces:: IXMLConfigurable, IDocumentParser

public class ExternalParser
extends Object
implements IDocumentParser, IXMLConfigurable

Parses and extracts text from a file using an external application to do so.

This parser relies heavily on the mechanics of ExternalTransformer. Refer to that class for documentation.

This parser can be made configurable via XML. See GenericDocumentParserFactory for general indications how to configure parsers.

To use an external application to change a file content after parsing has already occurred, consider using ExternalTransformer instead.

XML configuration usage:

  <parser contentType="(content type this parser is associated to)" 
          class="com.norconex.importer.parser.impl.ExternalParser" >
          
      <command>
          c:\Apps\myapp.exe ${INPUT} ${OUTPUT} ${INPUT_META} ${OUTPUT_META} ${REFERENCE}
      </command>
      
      <metadata 
              inputFormat="[json|xml|properties]" 
              outputFormat="[json|xml|properties]">
          <!-- pattern only used when no output format is specified -->
          <pattern field="(target field name)" 
                  fieldGroup="(field name match group index)"
                  valueGroup="(field value match group index)"
                  caseSensitive="[false|true]">
              (regular expression)
          </pattern>
          <!-- repeat pattern tag as needed -->
      </metadata>
      
      <environment>
          <variable name="(environment variable name)">
              (environment variable value)
          </variable>
          <!-- repeat variable tag as needed -->
      </environment>
      
  </parser>

Usage example:

The following example invokes an external application processing for simple text files that accepts two files as arguments: the first one being the file to transform, the second one being holding the transformation result. It also extract a document number from STDOUT, found as "DocNo:1234" and storing it as "docnumber".

  <parser contentType="text/plain" 
          class="com.norconex.importer.parser.impl.ExternalParser" >
      <command>/path/transform/app ${INPUT} ${OUTPUT}</command>
      <metadata>
          <pattern field="docnumber" valueGroup="1">DocNo:(\d+)</match>
      </metadata>
  </parser>

Since:: 2.2.0
Author:: Pascal Essiembre
See Also:: ExternalTransformer

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`META_FORMAT_JSON`
`static String`	`META_FORMAT_PROPERTIES`
`static String`	`META_FORMAT_XML`
`static String`	`TOKEN_INPUT`
`static String`	`TOKEN_INPUT_META`
`static String`	`TOKEN_OUTPUT`
`static String`	`TOKEN_OUTPUT_META`
`static String`	`TOKEN_REFERENCE`

Constructor Summary

Constructors
Constructor and Description

ExternalParser()

Constructors
Constructor and Description
`ExternalParser()`

Method Summary

All Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`void`	`addEnvironmentVariable(String name, String value)` Adds an environment variables to the list of previously assigned variables (if any).
`void`	`addEnvironmentVariables(Map<String,String> environmentVariables)` Adds the environment variables, keeping environment variables previously assigned.
`void`	`addMetadataExtractionPattern(Pattern pattern, boolean reverse)` Deprecated. Since 2.8.0, use `addMetadataExtractionPatterns(RegexFieldExtractor...)`
`void`	`addMetadataExtractionPattern(Pattern pattern, String field)` Deprecated. Since 2.8.0, use `addMetadataExtractionPatterns(RegexFieldExtractor...)`
`void`	`addMetadataExtractionPattern(String field, String pattern)` Adds a metadata extraction pattern that will extract the whole text matched into the given field.
`void`	`addMetadataExtractionPattern(String field, String pattern, int valueGroup)` Adds a metadata extraction pattern, which will extract the value from the specified group index upon matching.
`void`	`addMetadataExtractionPatterns(Map<Pattern,String> patterns)` Deprecated. Since 2.8.0, use `addMetadataExtractionPatterns(RegexFieldExtractor...)`
`void`	`addMetadataExtractionPatterns(RegexFieldExtractor... patterns)` Adds a metadata extraction pattern that will extract matching field names/values.
`boolean`	`equals(Object other)`
`String`	`getCommand()` Gets the command to execute.
`Map<String,String>`	`getEnvironmentVariables()` Gets environment variables.
`List<RegexFieldExtractor>`	`getMetadataExtractionPatterns()` Gets metadata extraction patterns.
`String`	`getMetadataInputFormat()` Gets the format of the metadata input file sent to the external application.
`String`	`getMetadataOutputFormat()` Gets the format of the metadata output file from the external application.
`File`	`getTempDir()` Gets directory where to store temporary files used for transformation.
`int`	`hashCode()`
`void`	`loadFromXML(Reader in)`
`List<ImporterDocument>`	`parseDocument(ImporterDocument doc, Writer output)` Parses a document.
`void`	`saveToXML(Writer out)`
`void`	`setCommand(String command)` Sets the command to execute.
`void`	`setEnvironmentVariables(Map<String,String> environmentVariables)` Sets the environment variables.
`void`	`setMetadataExtractionPatterns(Map<Pattern,String> patterns)` Deprecated. Since 2.8.0, use `addMetadataExtractionPatterns(RegexFieldExtractor...)`
`void`	`setMetadataExtractionPatterns(RegexFieldExtractor... patterns)` Sets metadata extraction patterns.
`void`	`setMetadataInputFormat(String metadataInputFormat)` Sets the format of the metadata input file sent to the external application.
`void`	`setMetadataOutputFormat(String metadataOutputFormat)` Sets the format of the metadata output file from the external application.
`void`	`setTempDir(File tempDir)` Sets directory where to store temporary files used for transformation.
`String`	`toString()`

Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

- Field Detail
  - TOKEN_INPUT
```
public static final String TOKEN_INPUT
```
    See Also:
    
    Constant Field Values
  - TOKEN_OUTPUT
```
public static final String TOKEN_OUTPUT
```
    See Also:
    
    Constant Field Values
  - TOKEN_INPUT_META
```
public static final String TOKEN_INPUT_META
```
    Since:
    
    2.8.0
    
    See Also:
    
    Constant Field Values
  - TOKEN_OUTPUT_META
```
public static final String TOKEN_OUTPUT_META
```
    Since:
    
    2.8.0
    
    See Also:
    
    Constant Field Values
  - TOKEN_REFERENCE
```
public static final String TOKEN_REFERENCE
```
    Since:
    
    2.8.0
    
    See Also:
    
    Constant Field Values
  - META_FORMAT_JSON
```
public static final String META_FORMAT_JSON
```
    Since:
    
    2.8.0
    
    See Also:
    
    Constant Field Values
  - META_FORMAT_XML
```
public static final String META_FORMAT_XML
```
    Since:
    
    2.8.0
    
    See Also:
    
    Constant Field Values
  - META_FORMAT_PROPERTIES
```
public static final String META_FORMAT_PROPERTIES
```
    Since:
    
    2.8.0
    
    See Also:
    
    Constant Field Values
- Constructor Detail
  - ExternalParser
```
public ExternalParser()
```
- Method Detail
  - getCommand
```
public String getCommand()
```
    Gets the command to execute.
    
    Returns:
    
    the command
  - setCommand
```
public void setCommand(String command)
```
    Sets the command to execute. Make sure to escape spaces in executable path and its arguments as well as other special command line characters.
    
    Parameters:
    
    command - the command
  - getTempDir
```
public File getTempDir()
```
    Gets directory where to store temporary files used for transformation.
    
    Returns:
    
    temporary directory
    
    Since:
    
    2.8.0
  - setTempDir
```
public void setTempDir(File tempDir)
```
    Sets directory where to store temporary files used for transformation.
    
    Parameters:
    
    tempDir - temporary directory
    
    Since:
    
    2.8.0
  - getMetadataExtractionPatterns
```
public List<RegexFieldExtractor> getMetadataExtractionPatterns()
```
    Gets metadata extraction patterns. See class documentation.
    
    Returns:
    
    map of patterns and field names
  - setMetadataExtractionPatterns
```
@Deprecated
public void setMetadataExtractionPatterns(Map<Pattern,String> patterns)
```
    Deprecated. Since 2.8.0, use addMetadataExtractionPatterns(RegexFieldExtractor...)
    
    Sets metadata extraction patterns. Clears any previously assigned patterns. To reverse the match group order in a double match group pattern, set the field name to "reverse:true".
    
    Parameters:
    
    patterns - map of patterns and field names
  - addMetadataExtractionPatterns
```
@Deprecated
public void addMetadataExtractionPatterns(Map<Pattern,String> patterns)
```
    Deprecated. Since 2.8.0, use addMetadataExtractionPatterns(RegexFieldExtractor...)
    
    Adds metadata extraction patterns, keeping any patterns previously assigned. To reverse the match group order in a double match group pattern, set the field name to "reverse:true".
    
    Parameters:
    
    patterns - map of patterns and field names
  - addMetadataExtractionPattern
```
@Deprecated
public void addMetadataExtractionPattern(Pattern pattern,
                                                      boolean reverse)
```
    Deprecated. Since 2.8.0, use addMetadataExtractionPatterns(RegexFieldExtractor...)
    
    Adds a metadata extraction pattern. See class documentation.
    
    Parameters:
    
    pattern - pattern with two match groups
    
    reverse - whether to reverse match groups (inverse key and value).
  - addMetadataExtractionPattern
```
@Deprecated
public void addMetadataExtractionPattern(Pattern pattern,
                                                      String field)
```
    Deprecated. Since 2.8.0, use addMetadataExtractionPatterns(RegexFieldExtractor...)
    
    Adds a metadata extraction pattern. See class documentation.
    
    Parameters:
    
    pattern - pattern with no or one match group
    
    field - field name where to store the matched pattern
  - addMetadataExtractionPattern
```
public void addMetadataExtractionPattern(String field,
                                         String pattern)
```
    Adds a metadata extraction pattern that will extract the whole text matched into the given field.
    
    Parameters:
    
    field - target field to store the matching pattern.
    
    pattern - the pattern
    
    Since:
    
    2.8.0
  - addMetadataExtractionPattern
```
public void addMetadataExtractionPattern(String field,
                                         String pattern,
                                         int valueGroup)
```
    Adds a metadata extraction pattern, which will extract the value from the specified group index upon matching.
    
    Parameters:
    
    field - target field to store the matching pattern.
    
    pattern - the pattern
    
    valueGroup - which pattern group to return.
    
    Since:
    
    2.8.0
  - addMetadataExtractionPatterns
```
public void addMetadataExtractionPatterns(RegexFieldExtractor... patterns)
```
    Adds a metadata extraction pattern that will extract matching field names/values.
    
    Parameters:
    
    patterns - extraction pattern
    
    Since:
    
    2.8.0
  - setMetadataExtractionPatterns
```
public void setMetadataExtractionPatterns(RegexFieldExtractor... patterns)
```
    Sets metadata extraction patterns. Clears any previously assigned patterns.
    
    Parameters:
    
    patterns - extraction pattern
    
    Since:
    
    2.8.0
  - getEnvironmentVariables
```
public Map<String,String> getEnvironmentVariables()
```
    Gets environment variables.
    
    Returns:
    
    environment variables or null if using the current process environment variables
  - setEnvironmentVariables
```
public void setEnvironmentVariables(Map<String,String> environmentVariables)
```
    Sets the environment variables. Clearing any prevously assigned environment variables. Set null to use the current process environment variables (default).
    
    Parameters:
    
    environmentVariables - environment variables
  - addEnvironmentVariables
```
public void addEnvironmentVariables(Map<String,String> environmentVariables)
```
    Adds the environment variables, keeping environment variables previously assigned. Existing variables of the same name will be overwritten. To clear all previously assigned variables and use the current process environment variables, pass null to setEnvironmentVariables(Map).
    
    Parameters:
    
    environmentVariables - environment variables
  - addEnvironmentVariable
```
public void addEnvironmentVariable(String name,
                                   String value)
```
    Adds an environment variables to the list of previously assigned variables (if any). Existing variables of the same name will be overwritten. Setting a variable with a null name has no effect while null values are converted to empty strings.
    
    Parameters:
    
    name - environment variable name
    
    value - environment variable value
  - getMetadataInputFormat
```
public String getMetadataInputFormat()
```
    Gets the format of the metadata input file sent to the external application. One of "json" (default), "xml", or "properties" is expected. Only applicable when the ${INPUT} token is part of the command.
    
    Returns:
    
    metadata input format
    
    Since:
    
    2.8.0
  - setMetadataInputFormat
```
public void setMetadataInputFormat(String metadataInputFormat)
```
    Sets the format of the metadata input file sent to the external application. One of "json" (default), "xml", or "properties" is expected. Only applicable when the ${INPUT} token is part of the command.
    
    Parameters:
    
    metadataInputFormat - format of the metadata input file
    
    Since:
    
    2.8.0
  - getMetadataOutputFormat
```
public String getMetadataOutputFormat()
```
    Gets the format of the metadata output file from the external application. By default no format is set, and metadata extraction patterns are used to extract metadata information. One of "json", "xml", or "properties" is expected. Only applicable when the ${OUTPUT} token is part of the command.
    
    Returns:
    
    metadata output format
    
    Since:
    
    2.8.0
  - setMetadataOutputFormat
```
public void setMetadataOutputFormat(String metadataOutputFormat)
```
    Sets the format of the metadata output file from the external application. One of "json" (default), "xml", or "properties" is expected. Set to null for relying metadata extraction patterns instead. Only applicable when the ${OUTPUT} token is part of the command.
    
    Parameters:
    
    metadataOutputFormat - format of the metadata output file
    
    Since:
    
    2.8.0
  - parseDocument
```
public List<ImporterDocument> parseDocument(ImporterDocument doc,
                                            Writer output)
                                     throws DocumentParserException
```
    Description copied from interface: IDocumentParser
    
    Parses a document.
    
    Specified by:
    
    parseDocument in interface IDocumentParser
    
    Parameters:
    
    doc - importer document to parse
    
    output - where to store extracted or modified content of the supplied document
    
    Returns:
    
    a list of first-level embedded documents, if any
    
    Throws:
    
    DocumentParserException - problem parsing document
  - loadFromXML
```
public void loadFromXML(Reader in)
                 throws IOException
```
    Specified by:
    
    loadFromXML in interface IXMLConfigurable
    
    Throws:
    
    IOException
  - saveToXML
```
public void saveToXML(Writer out)
               throws IOException
```
    Specified by:
    
    saveToXML in interface IXMLConfigurable
    
    Throws:
    
    IOException
  - equals
```
public boolean equals(Object other)
```
    Overrides:
    
    equals in class Object
  - hashCode
```
public int hashCode()
```
    Overrides:
    
    hashCode in class Object
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object

Class ExternalParser

XML configuration usage:

Usage example:

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

TOKEN_INPUT

TOKEN_OUTPUT

TOKEN_INPUT_META

TOKEN_OUTPUT_META

TOKEN_REFERENCE

META_FORMAT_JSON

META_FORMAT_XML

META_FORMAT_PROPERTIES

Constructor Detail

ExternalParser

Method Detail

getCommand

setCommand

getTempDir

setTempDir

getMetadataExtractionPatterns

setMetadataExtractionPatterns

addMetadataExtractionPatterns

addMetadataExtractionPattern

addMetadataExtractionPattern

addMetadataExtractionPattern

addMetadataExtractionPattern

addMetadataExtractionPatterns

setMetadataExtractionPatterns

getEnvironmentVariables

setEnvironmentVariables

addEnvironmentVariables

addEnvironmentVariable

getMetadataInputFormat

setMetadataInputFormat

getMetadataOutputFormat

setMetadataOutputFormat

parseDocument

loadFromXML

saveToXML

equals

hashCode

toString