com.norconex.committer.solr

Class SolrCommitter

java.lang.Object

com.norconex.committer.core3.AbstractCommitter

com.norconex.committer.core3.batch.AbstractBatchCommitter

com.norconex.committer.solr.SolrCommitter

All Implemented Interfaces:

IBatchConsumer, ICommitter, IXMLConfigurable, AutoCloseable

public class SolrCommitter
extends AbstractBatchCommitter

Commits documents to Apache Solr.

Solr Client:

As of 2.4.0, it is possible to specify which type of Solr Client to use. The expected configuration value of "solrURL" is influenced by the client type chosen. Default client type is HttpSolrClient. The clients are:

HttpSolrClient

For direct access to a single Solr node. Ideal for local development. Needs a Solr URL. Default client.

LBHttpSolrClient

Simple load-balancing as an alternative to an external load balancer. Needs two or more Solr node URLs (comma-separated).

ConcurrentUpdateSolrClient

Optimized for mass upload on a single node. Not best for queries. Needs a Solr URL.

CloudSolrClient

For use with a SolrCloud cluster. Needs a comma-separated list of Zookeeper hosts.

Http2SolrClient

Same as HttpSolrClient but for HTTP/2 support. Marked as experimental by Apache.

LBHttp2SolrClient

Same as LBHttpSolrClient but for HTTP/2 support. Marked as experimental by Apache.

ConcurrentUpdateHttp2SolrClient

Same as LBHttpSolrClient but for HTTP/2 support. Marked as experimental by Apache.

Authentication

Basic authentication is supported for password-protected Solr installations.

Password encryption:

Passwords can be encrypted using EncryptionUtil (or command-line "encrypt.bat" or "encrypt.sh" if those are available to you). In order for the password to be decrypted properly, you need to specify the encryption key used to encrypt it. The key can obtained from a few supported locations. The combination of the password key "value" and "source" is used to properly locate the key. The supported sources are:

key	The actual encryption key.
file	Path to a file containing the encryption key.
environment	Name of an environment variable containing the key.
property	Name of a JVM system property containing the key.

Restricting committer to specific documents

Optionally apply a committer only to certain type of documents. Documents are restricted based on their metadata field names and values. This option can be used to perform document routing when you have multiple committers defined.

Field mappings

By default, this abstract class applies field mappings for metadata fields, but leaves the document reference and content (input stream) for concrete implementations to handle. In other words, they only apply to a committer request metadata. Field mappings are performed on committer requests before upserts and deletes are actually performed.

XML configuration usage:

<committer
    class="com.norconex.committer.solr.SolrCommitter">
  <solrClientType>
    (See class documentation for options. Default: HttpSolrClient.)
  </solrClientType>
  <solrURL>(URL to Solr)</solrURL>
  <solrUpdateURLParams>
    <param
        name="(parameter name)">
      (parameter value)
    </param>
    <!-- multiple param tags allowed -->
  </solrUpdateURLParams>
  <solrCommitDisabled>[false|true]</solrCommitDisabled>
  <!-- Use the following if authentication is required. -->
  <credentials>
    <username>(the username)</username>
    <password>(the optionally encrypted password)</password>
    <passwordKey>
      <value>(The actual password encryption key or a reference to it.)</value>
      <source>[key|file|environment|property]</source>
      <size>(Size in bits of encryption key. Default is 128.)</size>
    </passwordKey>
  </credentials>
  <sourceIdField>
    (Optional document field name containing the value that will be stored
    in Solr target ID field. Default is the document reference.)
  </sourceIdField>
  <targetIdField>
    (Optional name of Solr field where to store a document unique
    identifier (sourceIdField).  If not specified, default is "id".)
  </targetIdField>
  <targetContentField>
    (Optional Solr field name to store document content/body.
    Default is "content".)
  </targetContentField>
  <!-- multiple "restrictTo" tags allowed (only one needs to match) -->
  <restrictTo>
    <fieldMatcher
        method="[basic|csv|wildcard|regex]"
        ignoreCase="[false|true]"
        ignoreDiacritic="[false|true]"
        partial="[false|true]">
      (field-matching expression)
    </fieldMatcher>
    <valueMatcher
        method="[basic|csv|wildcard|regex]"
        ignoreCase="[false|true]"
        ignoreDiacritic="[false|true]"
        partial="[false|true]">
      (value-matching expression)
    </valueMatcher>
  </restrictTo>
  <fieldMappings>
    <!-- Add as many field mappings as needed -->
    <mapping
        fromField="(source field name)"
        toField="(target field name)"/>
  </fieldMappings>
  <!-- Settings for default queue implementation ("class" is optional): -->
  <queue
      class="com.norconex.committer.core3.batch.queue.impl.FSQueue">
    <batchSize>
      (Optional number of documents queued after which we process a batch.
       Default is 20.)
    </batchSize>
    <maxPerFolder>
      (Optional maximum number of files or directories that can be queued
       in a single folder before a new one gets created. Default is 500.)
    </maxPerFolder>
    <commitLeftoversOnInit>
      (Optionally force to commit any leftover documents from a previous
       execution. E.g., prematurely ended.  Default is "false").
    </commitLeftoversOnInit>
    <onCommitFailure>
      <splitBatch>[OFF|HALF|ONE]</splitBatch>
      <maxRetries>(Max retries upon commit failures. Default is 0.)</maxRetries>
      <retryDelay>
        (Delay in milliseconds between retries. Default is 0.)
      </retryDelay>
      <ignoreErrors>
        [false|true]
        (When true, non-critical exceptions when interacting with the target

         repository won't be thrown to try continue the execution with other

         files to be committed. Instead, errors will be logged.
         In both cases the failing batch/files are moved to an
         "error" folder. Other types of exceptions may still be thrown.)
      </ignoreErrors>
    </onCommitFailure>
  </queue>
</committer>

XML configuration entries expecting millisecond durations can be provided in human-readable format (English only), as per DurationParser (e.g., "5 minutes and 30 seconds" or "5m30s").

Author:

Pascal Essiembre

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_SOLR_CONTENT_FIELD Default Solr content field
static String	DEFAULT_SOLR_ID_FIELD Default Solr ID field

Constructor Summary

Constructors

Constructor and Description
SolrCommitter() Constructor.

Method Summary

Modifier and Type	Method and Description
protected void	addSolrDeleteRequest(org.apache.solr.client.solrj.request.UpdateRequest solrBatchRequest, DeleteRequest committerRequest)
protected void	addSolrUpsertRequest(org.apache.solr.client.solrj.request.UpdateRequest solrBatchRequest, UpsertRequest committerRequest)
protected org.apache.solr.common.SolrInputDocument	buildSolrDocument(Properties fields)
protected void	closeBatchCommitter()
protected void	commitBatch(Iterator<ICommitterRequest> it)
boolean	equals(Object other)
Credentials	getCredentials() Gets Solr authentication credentials.
SolrClientType	getSolrClientType() Gets the Solr client type.
String	getSolrURL() Gets the Solr URL.
String	getSourceIdField() Gets the document field name containing the value to be stored in Solr ID field.
String	getTargetContentField() Gets the name of the Solr field where content will be stored.
String	getTargetIdField() Gets the name of the Solr field where to store a document unique identifier (sourceIdField).
String	getUpdateUrlParam(String name) Gets a URL parameter value by its parameter name.
Set<String>	getUpdateUrlParamNames() Gets the update URL parameter names.
int	hashCode()
protected void	initBatchCommitter()
boolean	isSolrCommitDisabled() Gets whether to send an explicit commit request at the end of every batch, or let the server auto-commit.
protected void	loadBatchCommitterFromXML(XML xml)
protected void	pushSolrRequest(org.apache.solr.client.solrj.request.UpdateRequest solrBatchRequest)
protected void	saveBatchCommitterToXML(XML xml)
void	setCredentials(Credentials credentials) Sets Solr authentication credentials.
void	setSolrClientType(SolrClientType solrClientType) Sets the Solr client type.
void	setSolrCommitDisabled(boolean solrCommitDisabled) Sets whether to send an explicit commit request at the end of every batch, or let the server auto-commit.
void	setSolrURL(String solrURL) Sets the Solr URL.
void	setSourceIdField(String sourceIdField) Sets the document field name containing the value to be stored in Solr ID field.
void	setTargetContentField(String targetContentField) Sets the name of the Solr field where content will be stored.
void	setTargetIdField(String targetIdField) Sets the name of the Solr field where to store a document unique identifier (sourceIdField).
void	setUpdateUrlParam(String name, String value) Sets URL parameters to be added on Solr HTTP calls.
String	toString()

Methods inherited from class com.norconex.committer.core3.batch.AbstractBatchCommitter

consume, doClean, doClose, doDelete, doInit, doUpsert, getCommitterQueue, loadCommitterFromXML, saveCommitterToXML, setCommitterQueue

Methods inherited from class com.norconex.committer.core3.AbstractCommitter

accept, addRestriction, addRestrictions, applyFieldMappings, clean, clearFieldMappings, clearRestrictions, close, delete, fireDebug, fireDebug, fireError, fireError, fireInfo, fireInfo, getCommitterContext, getFieldMappings, getRestrictions, init, loadFromXML, removeFieldMapping, removeRestriction, removeRestriction, saveToXML, setFieldMapping, setFieldMappings, upsert

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.norconex.commons.lang.xml.IXMLConfigurable

loadFromXML, saveToXML

Field Detail

DEFAULT_SOLR_ID_FIELD

public static final String DEFAULT_SOLR_ID_FIELD

Default Solr ID field

See Also:

Constant Field Values

DEFAULT_SOLR_CONTENT_FIELD

public static final String DEFAULT_SOLR_CONTENT_FIELD

Default Solr content field

See Also:

Constant Field Values

Constructor Detail

SolrCommitter

public SolrCommitter()

Constructor.

Method Detail

getSolrClientType

public SolrClientType getSolrClientType()

Gets the Solr client type.

Returns:

solr client type

setSolrClientType

public void setSolrClientType(SolrClientType solrClientType)

Sets the Solr client type.

Parameters:

solrClientType - solr client type

getSolrURL

public String getSolrURL()

Gets the Solr URL.

Returns:

Solr URL

setSolrURL

public void setSolrURL(String solrURL)

Sets the Solr URL.

Parameters:

solrURL - solrURL

setUpdateUrlParam

public void setUpdateUrlParam(String name,
                              String value)

Sets URL parameters to be added on Solr HTTP calls.

Parameters:

name - parameter name

value - parameter value

getUpdateUrlParam

public String getUpdateUrlParam(String name)

Gets a URL parameter value by its parameter name.

Parameters:

name - parameter name

Returns:

parameter value

getUpdateUrlParamNames

public Set<String> getUpdateUrlParamNames()

Gets the update URL parameter names.

Returns:

parameter names

setSolrCommitDisabled

public void setSolrCommitDisabled(boolean solrCommitDisabled)

Sets whether to send an explicit commit request at the end of every batch, or let the server auto-commit.

Parameters:

solrCommitDisabled - true if sending Solr commit is disabled

isSolrCommitDisabled

public boolean isSolrCommitDisabled()

Gets whether to send an explicit commit request at the end of every batch, or let the server auto-commit.

Returns:

true if sending Solr commit is disabled.

getCredentials

public Credentials getCredentials()

Gets Solr authentication credentials.

Returns:

credentials

setCredentials

public void setCredentials(Credentials credentials)

Sets Solr authentication credentials.

Parameters:

credentials - the credentials

getTargetContentField

public String getTargetContentField()

Gets the name of the Solr field where content will be stored. Default is "content".

Returns:

field name

setTargetContentField

public void setTargetContentField(String targetContentField)

Sets the name of the Solr field where content will be stored. Specifying a null value will disable storing the content.

Parameters:

targetContentField - field name

getSourceIdField

public String getSourceIdField()

Gets the document field name containing the value to be stored in Solr ID field. Default is not a field, but rather the document reference.

Returns:

name of field containing id value

setSourceIdField

public void setSourceIdField(String sourceIdField)

Sets the document field name containing the value to be stored in Solr ID field. Set null to use the document reference instead of a field (default).

Parameters:

sourceIdField - name of field containing id value, or null

getTargetIdField

public String getTargetIdField()

Gets the name of the Solr field where to store a document unique identifier (sourceIdField). Default is "id".

Returns:

name of Solr ID field

setTargetIdField

public void setTargetIdField(String targetIdField)

Sets the name of the Solr field where to store a document unique identifier (sourceIdField). If not specified, default is "id".

Parameters:

targetIdField - name of Solr ID field

initBatchCommitter

protected void initBatchCommitter()
                           throws CommitterException

Overrides:

initBatchCommitter in class AbstractBatchCommitter

Throws:

CommitterException

commitBatch

protected void commitBatch(Iterator<ICommitterRequest> it)
                    throws CommitterException

Specified by:

commitBatch in class AbstractBatchCommitter

Throws:

CommitterException

closeBatchCommitter

protected void closeBatchCommitter()
                            throws CommitterException

Overrides:

closeBatchCommitter in class AbstractBatchCommitter

Throws:

CommitterException

pushSolrRequest

protected void pushSolrRequest(org.apache.solr.client.solrj.request.UpdateRequest solrBatchRequest)
                        throws org.apache.solr.client.solrj.SolrServerException,
                               IOException,
                               CommitterException

Throws:

org.apache.solr.client.solrj.SolrServerException

IOException

CommitterException

addSolrUpsertRequest

protected void addSolrUpsertRequest(org.apache.solr.client.solrj.request.UpdateRequest solrBatchRequest,
                                    UpsertRequest committerRequest)
                             throws CommitterException

Throws:

CommitterException

addSolrDeleteRequest

protected void addSolrDeleteRequest(org.apache.solr.client.solrj.request.UpdateRequest solrBatchRequest,
                                    DeleteRequest committerRequest)

buildSolrDocument

protected org.apache.solr.common.SolrInputDocument buildSolrDocument(Properties fields)

loadBatchCommitterFromXML

protected void loadBatchCommitterFromXML(XML xml)

Specified by:

loadBatchCommitterFromXML in class AbstractBatchCommitter

saveBatchCommitterToXML

protected void saveBatchCommitterToXML(XML xml)

Specified by:

saveBatchCommitterToXML in class AbstractBatchCommitter

equals

public boolean equals(Object other)

Overrides:

equals in class AbstractBatchCommitter

hashCode

public int hashCode()

Overrides:

hashCode in class AbstractBatchCommitter

toString

public String toString()

Overrides:

toString in class AbstractBatchCommitter