com.norconex.committer.solr

Class SolrCommitter

java.lang.Object

com.norconex.committer.core.AbstractCommitter

com.norconex.committer.core.AbstractFileQueueCommitter

com.norconex.committer.core.AbstractBatchCommitter

com.norconex.committer.core.AbstractMappedCommitter

com.norconex.committer.solr.SolrCommitter

All Implemented Interfaces:

ICommitter, IXMLConfigurable

public class SolrCommitter
extends AbstractMappedCommitter

Commits documents to Apache Solr.

Solr Client:

As of 2.4.0, it is possible to specify which type of Solr Client to use. The expected configuration value of "solrURL" is influenced by the client type chosen. Default client type is HttpSolrClient. The clients are:

HttpSolrClient

For direct access to a single Solr node. Ideal for local development. Needs a Solr URL. Default client.

LBHttpSolrClient

Simple load-balancing as an alternative to an external load balancer. Needs two or more Solr node URLs (comma-separated).

ConcurrentUpdateSolrClient

Optimized for mass upload on a single node. Not best for queries. Needs a Solr URL.

CloudSolrClient

For use with a SolrCloud cluster. Needs a comma-separated list of Zookeeper hosts.

Http2SolrClient

Same as HttpSolrClient but for HTTP/2 support. Marked as experimental by Apache.

LBHttp2SolrClient

Same as LBHttpSolrClient but for HTTP/2 support. Marked as experimental by Apache.

ConcurrentUpdateHttp2SolrClient

Same as LBHttpSolrClient but for HTTP/2 support. Marked as experimental by Apache.

Authentication

Since 2.4.0, BASIC authentication is supported for password-protected Solr installations. The password can optionally be encrypted using EncryptionUtil (or command-line "encrypt.bat" or "encrypt.sh"). In order for the password to be decrypted properly, you need to specify the encryption key used to encrypt it. The key can be stored in a few supported locations and a combination of passwordKey and passwordKeySource must be specified to properly locate the key. The supported sources are:

passwordKeySource	passwordKey
key	The actual encryption key.
file	Path to a file containing the encryption key.
environment	Name of an environment variable containing the key.
property	Name of a JVM system property containing the key.

As of 2.2.1, XML configuration entries expecting millisecond durations can be provided in human-readable format (English only), as per DurationParser (e.g., "5 minutes and 30 seconds" or "5m30s").

XML configuration usage:

  <committer class="com.norconex.committer.solr.SolrCommitter">
      <solrClientType>
         (See class documentation for options. Default: HttpSolrClient.)
      </solrClientType>
      <solrURL>(URL to Solr)</solrURL>
      <solrUpdateURLParams>
         <param name="(parameter name)">(parameter value)</param>
         <-- multiple param tags allowed -->
      </solrUpdateURLParams>
      <solrCommitDisabled>[false|true]</solrCommitDisabled>

      <!-- Use the following if BASIC authentication is required. -->
      <username>(Optional user name)</username>
      <password>(Optional user password)</password>
      <!-- Use the following if password is encrypted. -->
      <passwordKey>(the encryption key or a reference to it)</passwordKey>
      <passwordKeySource>[key|file|environment|property]</passwordKeySource>

      <sourceReferenceField keep="[false|true]">
         (Optional name of field that contains the document reference, when
         the default document reference is not used.  The reference value
         will be mapped to Solr "id" field, or the "targetReferenceField"
         specified.
         Once re-mapped, this metadata source field is
         deleted, unless "keep" is set to true.)
      </sourceReferenceField>
      <targetReferenceField>
         (Name of Solr target field where the store a document unique
         identifier (idSourceField).  If not specified, default is "id".)
      </targetReferenceField>
      <sourceContentField keep="[false|true]">
         (If you wish to use a metadata field to act as the document
         "content", you can specify that field here.  Default
         does not take a metadata field but rather the document content.
         Once re-mapped, the metadata source field is deleted,
         unless "keep" is set to true.)
      </sourceContentField>
      <targetContentField>
         (Solr target field name for a document content/body.
          Default is: content)
      </targetContentField>
      <commitBatchSize>
         (Maximum number of docs to send Solr at once. Will issue a Solr
          commit unless "solrCommitDisabled" is true)
      </commitBatchSize>
      <queueDir>(optional path where to queue files)</queueDir>
      <queueSize>(max queue size before sending to Solr)</queueSize>
      <maxRetries>(max retries upon commit failures)</maxRetries>
      <maxRetryWait>(max delay in milliseconds between retries)</maxRetryWait>
  </committer>
 

Author:

Pascal Essiembre

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_SOLR_CONTENT_FIELD Default Solr content field
static String	DEFAULT_SOLR_ID_FIELD Default Solr ID field

Fields inherited from class com.norconex.committer.core.AbstractBatchCommitter

DEFAULT_COMMIT_BATCH_SIZE

Fields inherited from class com.norconex.committer.core.AbstractFileQueueCommitter

DEFAULT_QUEUE_DIR, filesCommitting

Fields inherited from class com.norconex.committer.core.AbstractCommitter

DEFAULT_QUEUE_SIZE, queueSize

Constructor Summary

Constructors

Constructor and Description
SolrCommitter() Constructor.
SolrCommitter(org.apache.solr.client.solrj.SolrClient solrClient) Constructor.

Method Summary

Modifier and Type	Method and Description
protected void	commitBatch(List<ICommitOperation> batch)
boolean	equals(Object obj)
String	getPassword() Gets the password.
EncryptionKey	getPasswordKey() Gets the password encryption key.
SolrClientType	getSolrClientType() Gets the Solr client type.
String	getSolrURL() Gets the Solr URL.
String	getUpdateUrlParam(String name) Gets a URL parameter value by its parameter name.
Set<String>	getUpdateUrlParamNames() Gets the update URL parameter names.
String	getUsername() Gets the username.
int	hashCode()
boolean	isSolrCommitDisabled() Gets whether to send an explicit commit request at the end of every batch, or let the server auto-commit.
protected void	loadFromXml(XMLConfiguration xml)
protected void	saveToXML(XMLStreamWriter writer)
void	setPassword(String password) Sets the password.
void	setPasswordKey(EncryptionKey passwordKey) Sets the password encryption key.
void	setSolrClientType(SolrClientType solrClientType) Sets the Solr client type.
void	setSolrCommitDisabled(boolean solrCommitDisabled) Sets whether to send an explicit commit request at the end of every batch, or let the server auto-commit.
void	setSolrURL(String solrURL) Sets the Solr URL.
void	setUpdateUrlParam(String name, String value) Sets URL parameters to be added on Solr HTTP calls.
void	setUsername(String username) Sets the username.
protected org.apache.solr.client.solrj.request.UpdateRequest	solrAddRequest(IAddOperation op)
protected org.apache.solr.client.solrj.request.UpdateRequest	solrDeleteRequest(IDeleteOperation op)
String	toString()

Methods inherited from class com.norconex.committer.core.AbstractMappedCommitter

getSourceContentField, getSourceReferenceField, getTargetContentField, getTargetReferenceField, isKeepSourceContentField, isKeepSourceReferenceField, loadFromXML, prepareCommitAddition, saveToXML, setKeepSourceContentField, setKeepSourceReferenceField, setSourceContentField, setSourceReferenceField, setTargetContentField, setTargetReferenceField

Methods inherited from class com.norconex.committer.core.AbstractBatchCommitter

commitAddition, commitComplete, commitDeletion, getCommitBatchSize, getMaxRetries, getMaxRetryWait, setCommitBatchSize, setMaxRetries, setMaxRetryWait

Methods inherited from class com.norconex.committer.core.AbstractFileQueueCommitter

commit, getInitialQueueDocCount, getQueueDir, prepareCommitDeletion, queueAddition, queueRemoval, setQueueDir

Methods inherited from class com.norconex.committer.core.AbstractCommitter

add, getQueueSize, remove, setQueueSize

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_SOLR_ID_FIELD

public static final String DEFAULT_SOLR_ID_FIELD

Default Solr ID field

See Also:

Constant Field Values

DEFAULT_SOLR_CONTENT_FIELD

public static final String DEFAULT_SOLR_CONTENT_FIELD

Default Solr content field

See Also:

Constant Field Values

Constructor Detail

SolrCommitter

public SolrCommitter()

Constructor.

SolrCommitter

public SolrCommitter(org.apache.solr.client.solrj.SolrClient solrClient)

Constructor.

Parameters:

solrClient - SolrClient to use

Method Detail

getSolrClientType

public SolrClientType getSolrClientType()

Gets the Solr client type.

Returns:

solr client type

Since:

2.4.0

setSolrClientType

public void setSolrClientType(SolrClientType solrClientType)

Sets the Solr client type.

Parameters:

solrClientType - solr client type

Since:

2.4.0

getSolrURL

public String getSolrURL()

Gets the Solr URL.

Returns:

Solr URL

setSolrURL

public void setSolrURL(String solrURL)

Sets the Solr URL.

Parameters:

solrURL - solrURL

setUpdateUrlParam

public void setUpdateUrlParam(String name,
                              String value)

Sets URL parameters to be added on Solr HTTP calls.

Parameters:

name - parameter name

value - parameter value

getUpdateUrlParam

public String getUpdateUrlParam(String name)

Gets a URL parameter value by its parameter name.

Parameters:

name - parameter name

Returns:

parameter value

getUpdateUrlParamNames

public Set<String> getUpdateUrlParamNames()

Gets the update URL parameter names.

Returns:

parameter names

setSolrCommitDisabled

public void setSolrCommitDisabled(boolean solrCommitDisabled)

Sets whether to send an explicit commit request at the end of every batch, or let the server auto-commit.

Parameters:

solrCommitDisabled - true if sending Solr commit is disabled

Since:

2.2.0

isSolrCommitDisabled

public boolean isSolrCommitDisabled()

Gets whether to send an explicit commit request at the end of every batch, or let the server auto-commit.

Returns:

true if sending Solr commit is disabled.

Since:

2.2.0

getUsername

public String getUsername()

Gets the username.

Returns:

username the username

Since:

2.4.0

setUsername

public void setUsername(String username)

Sets the username.

Parameters:

username - the username

Since:

2.4.0

getPassword

public String getPassword()

Gets the password.

Returns:

password the password

Since:

2.4.0

setPassword

public void setPassword(String password)

Sets the password.

Parameters:

password - the password

Since:

2.4.0

getPasswordKey

public EncryptionKey getPasswordKey()

Gets the password encryption key.

Returns:

the password key or null if the password is not encrypted.

Since:

2.4.0

See Also:

EncryptionUtil

setPasswordKey

public void setPasswordKey(EncryptionKey passwordKey)

Sets the password encryption key. Only required when the password is encrypted.

Parameters:

passwordKey - password key

Since:

2.4.0

See Also:

EncryptionUtil

commitBatch

protected void commitBatch(List<ICommitOperation> batch)

Specified by:

commitBatch in class AbstractBatchCommitter

solrAddRequest

protected org.apache.solr.client.solrj.request.UpdateRequest solrAddRequest(IAddOperation op)

solrDeleteRequest

protected org.apache.solr.client.solrj.request.UpdateRequest solrDeleteRequest(IDeleteOperation op)

saveToXML

protected void saveToXML(XMLStreamWriter writer)
                  throws XMLStreamException

Specified by:

saveToXML in class AbstractMappedCommitter

Throws:

XMLStreamException

loadFromXml

protected void loadFromXml(XMLConfiguration xml)

Specified by:

loadFromXml in class AbstractMappedCommitter

hashCode

public int hashCode()

Overrides:

hashCode in class AbstractMappedCommitter

equals

public boolean equals(Object obj)

Overrides:

equals in class AbstractMappedCommitter

toString

public String toString()

Overrides:

toString in class AbstractMappedCommitter