CloudSearchCommitter (Norconex Committer CloudSearch 1.4.0 API)

java.lang.Object
- com.norconex.committer.core.AbstractCommitter
- - com.norconex.committer.core.AbstractFileQueueCommitter
  - - com.norconex.committer.core.AbstractBatchCommitter
    - - com.norconex.committer.core.AbstractMappedCommitter
      - com.norconex.committer.cloudsearch.CloudSearchCommitter

All Implemented Interfaces:: ICommitter, IXMLConfigurable

public class CloudSearchCommitter
extends AbstractMappedCommitter

Commits documents to Amazon CloudSearch.

Authentication:

An access key and security key are required to connect to interact with CloudSearch. For enhanced security, it is best to use one of the methods described in DefaultAWSCredentialsProviderChain for setting them (environment variables, system properties, profile file, etc). Do not explicitly set "accessKey" and "secretKey" on this class if you want to rely on safer methods.

CloudSearch ID limitations:

As of this writing, CloudSearch has a 128 characters limitation on its "id" field. In addition, certain characters are not allowed. By default, an error will result from trying to submit documents with an invalid ID. As of 1.3.0, you can get around this by setting setFixBadIds(boolean) to true. It will truncate references that are too long and append a hash code to it representing the truncated part. It will also convert invalid characters to underscore. This approach is not 100% collision-free (uniqueness), but it should safely cover the vast majority of cases.

If you want to keep the original (non-truncated) URL, make sure you set AbstractMappedCommitter.setKeepSourceReferenceField(boolean) to true.

Password encryption in XML configuration:

As of 1.4.0, it is possible to specify proxy settings, and optionally, have the supplied password encrypted using EncryptionUtil or encrypt/decrypt scripts package with this library. In order for the password to be decrypted properly by the crawler, you need to specify the encryption key used to encrypt it. The key can be stored in a few supported locations and a combination of proxyPasswordKey and proxyPasswordKeySource must be specified to properly locate the key. The supported sources are:

`proxyPasswordKeySource`	`proxyPasswordKey`
`key`	The actual encryption key.
`file`	Path to a file containing the encryption key.
`environment`	Name of an environment variable containing the key.
`property`	Name of a JVM system property containing the key.

XML configuration usage:

  <committer class="com.norconex.committer.cloudsearch.CloudSearchCommitter">
  
      <!-- Mandatory: -->
      <serviceEndpoint>(CloudSearch service endpoint)</serviceEndpoint>
      
      <!-- Mandatory if not configured elsewhere: -->
      <accessKey>
         (Optional CloudSearch access key. Will be taken from environment 
          when blank.)
      </accessKey>
      <secretKey>
         (Optional CloudSearch secret key. Will be taken from environment
          when blank.)
      </secretKey>
      
      <!-- Optional settings: -->
      <fixBadIds>
         [false|true](Forces references to fit into a CloudSearch id field.)
      </fixBadIds>
      <signingRegion>(CloudSearch signing region)</signingRegion>
      <proxyHost>...</proxyHost>
      <proxyPort>...</proxyPort>
      <proxyUsername>...</proxyUsername>
      <proxyPassword>...</proxyPassword>
      <!-- Use the following if password is encrypted. -->
      <proxyPasswordKey>(the encryption key or a reference to it)</proxyPasswordKey>
      <proxyPasswordKeySource>[key|file|environment|property]</proxyPasswordKeySource>
      
      <sourceReferenceField keep="[false|true]">
         (Optional name of field that contains the document reference, when 
         the default document reference is not used.  The reference value
         will be mapped to CloudSearch "id" field, which is mandatory.
         Once re-mapped, this metadata source field is 
         deleted, unless "keep" is set to true.)
      </sourceReferenceField>
      <sourceContentField keep="[false|true]">
         (If you wish to use a metadata field to act as the document 
         "content", you can specify that field here.  Default 
         does not take a metadata field but rather the document content.
         Once re-mapped, the metadata source field is deleted,
         unless "keep" is set to true.)
      </sourceContentField>
      <targetContentField>
         (CloudSearch target field name for a document content/body.
          Default is: content)
      </targetContentField>
      <commitBatchSize>
          (Max number of docs to send CloudSearch at once. If you experience
           memory problems, lower this number.  Default is 100.)
      </commitBatchSize>
      <queueDir>(Optional path where to queue files)</queueDir>
      <queueSize>
          (Max queue size before committing. Default is 1000.)
      </queueSize>
      <maxRetries>
          (Max retries upon commit failures. Default is 0.)
      </maxRetries>
      <maxRetryWait>
          (Max delay in milliseconds between retries. Default is 0.)
      </maxRetryWait>
  </committer>

XML configuration entries expecting millisecond durations can be provided in human-readable format (English only), as per DurationParser (e.g., "5 minutes and 30 seconds" or "5m30s").

Author:: El-Hebri Khiari, Pascal Essiembre

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`COULDSEARCH_ID_FIELD` CloudSearch mandatory ID field
`static String`	`DEFAULT_COULDSEARCH_CONTENT_FIELD` Default CloudSearch content field
`static Pattern`	`FIELD_PATTERN` CouldSearch mandatory field pattern.

Fields inherited from class com.norconex.committer.core.AbstractBatchCommitter
DEFAULT_COMMIT_BATCH_SIZE

Fields inherited from class com.norconex.committer.core.AbstractFileQueueCommitter
DEFAULT_QUEUE_DIR, filesCommitting

Fields inherited from class com.norconex.committer.core.AbstractCommitter
DEFAULT_QUEUE_SIZE, queueSize

Constructor Summary

Constructors
Constructor and Description
`CloudSearchCommitter()`
`CloudSearchCommitter(String serviceEndpoint)`
`CloudSearchCommitter(String serviceEndpoint, String signingRegion)`

Method Summary

All Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`protected com.amazonaws.ClientConfiguration`	`buildClientConfiguration()`
`protected void`	`commitBatch(List<ICommitOperation> batch)`
`boolean`	`equals(Object obj)`
`String`	`getAccessKey()` Gets the CloudSearch access key.
`String`	`getDocumentEndpoint()` Deprecated. Since 1.2.0, use `setServiceEndpoint(String)`
`String`	`getProxyHost()` Gets the proxy host.
`String`	`getProxyPassword()` Gets the proxy password.
`EncryptionKey`	`getProxyPasswordKey()` Gets the proxy password encryption key.
`int`	`getProxyPort()` Gets the proxy port.
`String`	`getProxyUsername()` Gets the proxy username.
`String`	`getSecretKey()` Gets the CloudSearch secret key.
`String`	`getServiceEndpoint()` Gets AWS service endpoint.
`String`	`getSigningRegion()` Gets the AWS signing region.
`int`	`hashCode()`
`boolean`	`isFixBadIds()` Gets whether to fix IDs that are too long for CloudSearch ID limitation (128 characters max).
`protected void`	`loadFromXml(XMLConfiguration xml)`
`protected void`	`saveToXML(XMLStreamWriter writer)`
`void`	`setAccessKey(String accessKey)` Sets the CloudSearch access key.
`void`	`setDocumentEndpoint(String documentEndpoint)` Deprecated. Since 1.2.0, use `getServiceEndpoint()`
`void`	`setFixBadIds(boolean fixBadIds)` Sets whether to fix IDs that are too long for CloudSearch ID limitation (128 characters max).
`void`	`setProxyHost(String proxyHost)` Sets the proxy host.
`void`	`setProxyPassword(String proxyPassword)` Sets the proxy password.
`void`	`setProxyPasswordKey(EncryptionKey proxyPasswordKey)` Sets the proxy password encryption key.
`void`	`setProxyPort(int proxyPort)` Sets the proxy port.
`void`	`setProxyUsername(String proxyUsername)` Sets the proxy username
`void`	`setSecretKey(String secretKey)` Sets the CloudSearch secret key.
`void`	`setServiceEndpoint(String serviceEndpoint)` Sets AWS service endpoint.
`void`	`setSigningRegion(String signingRegion)` Gets the AWS signing region.
`void`	`setTargetReferenceField(String targetReferenceField)` This method is not supported and will throw an `UnsupportedOperationException` if invoked.
`String`	`toString()`

Methods inherited from class com.norconex.committer.core.AbstractMappedCommitter
getSourceContentField, getSourceReferenceField, getTargetContentField, getTargetReferenceField, isKeepSourceContentField, isKeepSourceReferenceField, loadFromXML, prepareCommitAddition, saveToXML, setKeepSourceContentField, setKeepSourceReferenceField, setSourceContentField, setSourceReferenceField, setTargetContentField

Methods inherited from class com.norconex.committer.core.AbstractBatchCommitter
commitAddition, commitComplete, commitDeletion, getCommitBatchSize, getMaxRetries, getMaxRetryWait, setCommitBatchSize, setMaxRetries, setMaxRetryWait

Methods inherited from class com.norconex.committer.core.AbstractFileQueueCommitter
commit, getInitialQueueDocCount, getQueueDir, prepareCommitDeletion, queueAddition, queueRemoval, setQueueDir

Methods inherited from class com.norconex.committer.core.AbstractCommitter
add, getQueueSize, remove, setQueueSize

Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

- Field Detail
  - FIELD_PATTERN
```
public static final Pattern FIELD_PATTERN
```
    CouldSearch mandatory field pattern. Characters not matching the pattern will be replaced by an underscore.
  - COULDSEARCH_ID_FIELD
```
public static final String COULDSEARCH_ID_FIELD
```
    CloudSearch mandatory ID field
    
    See Also:
    
    Constant Field Values
  - DEFAULT_COULDSEARCH_CONTENT_FIELD
```
public static final String DEFAULT_COULDSEARCH_CONTENT_FIELD
```
    Default CloudSearch content field
    
    See Also:
    
    Constant Field Values
- Constructor Detail
  - CloudSearchCommitter
```
public CloudSearchCommitter()
```
  - CloudSearchCommitter
```
public CloudSearchCommitter(String serviceEndpoint)
```
  - CloudSearchCommitter
```
public CloudSearchCommitter(String serviceEndpoint,
                            String signingRegion)
```
- Method Detail
  - getServiceEndpoint
```
public String getServiceEndpoint()
```
    Gets AWS service endpoint.
    
    Returns:
    
    AWS service endpoing
    
    Since:
    
    1.2.0.
  - setServiceEndpoint
```
public void setServiceEndpoint(String serviceEndpoint)
```
    Sets AWS service endpoint.
    
    Parameters:
    
    serviceEndpoint - AWS service endpoing
    
    Since:
    
    1.2.0.
  - getSigningRegion
```
public String getSigningRegion()
```
    Gets the AWS signing region.
    
    Returns:
    
    the AWS signing region
    
    Since:
    
    1.2.0.
  - setSigningRegion
```
public void setSigningRegion(String signingRegion)
```
    Gets the AWS signing region.
    
    Parameters:
    
    signingRegion - the AWS signing region
    
    Since:
    
    1.2.0.
  - getDocumentEndpoint
```
@Deprecated
public String getDocumentEndpoint()
```
    Deprecated. Since 1.2.0, use setServiceEndpoint(String)
    
    Gets the CloudSearch document endpoint.
    
    Returns:
    
    document endpoint
  - setDocumentEndpoint
```
@Deprecated
public void setDocumentEndpoint(String documentEndpoint)
```
    Deprecated. Since 1.2.0, use getServiceEndpoint()
    
    Sets the CloudSearch document endpoint.
    
    Parameters:
    
    documentEndpoint - document endpoint
  - getAccessKey
```
public String getAccessKey()
```
    Gets the CloudSearch access key. If null, the access key will be obtained from the environment, as detailed in DefaultAWSCredentialsProviderChain.
    
    Returns:
    
    the access key
  - setAccessKey
```
public void setAccessKey(String accessKey)
```
    Sets the CloudSearch access key. If null, the access key will be obtained from the environment, as detailed in DefaultAWSCredentialsProviderChain.
    
    Parameters:
    
    accessKey - the access key
  - getSecretKey
```
public String getSecretKey()
```
    Gets the CloudSearch secret key. If null, the secret key will be obtained from the environment, as detailed in DefaultAWSCredentialsProviderChain.
    
    Returns:
    
    the secret key
  - setSecretKey
```
public void setSecretKey(String secretKey)
```
    Sets the CloudSearch secret key. If null, the secret key will be obtained from the environment, as detailed in DefaultAWSCredentialsProviderChain.
    
    Parameters:
    
    secretKey - the secret key
  - setTargetReferenceField
```
public void setTargetReferenceField(String targetReferenceField)
```
    This method is not supported and will throw an UnsupportedOperationException if invoked. With CloudSearch, the target field for a document unique id is always "id".
    
    Overrides:
    
    setTargetReferenceField in class AbstractMappedCommitter
    
    Parameters:
    
    targetReferenceField - the target field
  - isFixBadIds
```
public boolean isFixBadIds()
```
    Gets whether to fix IDs that are too long for CloudSearch ID limitation (128 characters max). If true, long IDs will be truncated and a hash code representing the truncated part will be appended.
    
    Returns:
    
    true to fix IDs that are too long
    
    Since:
    
    1.3.0
  - setFixBadIds
```
public void setFixBadIds(boolean fixBadIds)
```
    Sets whether to fix IDs that are too long for CloudSearch ID limitation (128 characters max). If true, long IDs will be truncated and a hash code representing the truncated part will be appended.
    
    Parameters:
    
    fixBadIds - true to fix IDs that are too long
    
    Since:
    
    1.3.0
  - getProxyHost
```
public String getProxyHost()
```
    Gets the proxy host.
    
    Returns:
    
    proxy host
    
    Since:
    
    1.4.0
  - setProxyHost
```
public void setProxyHost(String proxyHost)
```
    Sets the proxy host.
    
    Parameters:
    
    proxyHost - proxy host
    
    Since:
    
    1.4.0
  - getProxyPort
```
public int getProxyPort()
```
    Gets the proxy port.
    
    Returns:
    
    proxy port
    
    Since:
    
    1.4.0
  - setProxyPort
```
public void setProxyPort(int proxyPort)
```
    Sets the proxy port.
    
    Parameters:
    
    proxyPort - proxy port
    
    Since:
    
    1.4.0
  - getProxyUsername
```
public String getProxyUsername()
```
    Gets the proxy username.
    
    Returns:
    
    proxy username
    
    Since:
    
    1.4.0
  - setProxyUsername
```
public void setProxyUsername(String proxyUsername)
```
    Sets the proxy username
    
    Parameters:
    
    proxyUsername - proxy username
    
    Since:
    
    1.4.0
  - getProxyPassword
```
public String getProxyPassword()
```
    Gets the proxy password.
    
    Returns:
    
    proxy password
    
    Since:
    
    1.4.0
  - setProxyPassword
```
public void setProxyPassword(String proxyPassword)
```
    Sets the proxy password.
    
    Parameters:
    
    proxyPassword - proxy password
    
    Since:
    
    1.4.0
  - getProxyPasswordKey
```
public EncryptionKey getProxyPasswordKey()
```
    Gets the proxy password encryption key.
    
    Returns:
    
    the password key or null if the password is not encrypted.
    
    Since:
    
    1.4.0
    
    See Also:
    
    EncryptionUtil
  - setProxyPasswordKey
```
public void setProxyPasswordKey(EncryptionKey proxyPasswordKey)
```
    Sets the proxy password encryption key. Only required when the password is encrypted.
    
    Parameters:
    
    proxyPasswordKey - password key
    
    Since:
    
    1.4.0
    
    See Also:
    
    EncryptionUtil
  - commitBatch
```
protected void commitBatch(List<ICommitOperation> batch)
```
    Specified by:
    
    commitBatch in class AbstractBatchCommitter
  - buildClientConfiguration
```
protected com.amazonaws.ClientConfiguration buildClientConfiguration()
```
  - saveToXML
```
protected void saveToXML(XMLStreamWriter writer)
                  throws XMLStreamException
```
    Specified by:
    
    saveToXML in class AbstractMappedCommitter
    
    Throws:
    
    XMLStreamException
  - loadFromXml
```
protected void loadFromXml(XMLConfiguration xml)
```
    Specified by:
    
    loadFromXml in class AbstractMappedCommitter
  - hashCode
```
public int hashCode()
```
    Overrides:
    
    hashCode in class AbstractMappedCommitter
  - equals
```
public boolean equals(Object obj)
```
    Overrides:
    
    equals in class AbstractMappedCommitter
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class AbstractMappedCommitter

Class CloudSearchCommitter

Authentication:

CloudSearch ID limitations:

Password encryption in XML configuration:

XML configuration usage:

Field Summary

Fields inherited from class com.norconex.committer.core.AbstractBatchCommitter

Fields inherited from class com.norconex.committer.core.AbstractFileQueueCommitter

Fields inherited from class com.norconex.committer.core.AbstractCommitter

Constructor Summary

Method Summary

Methods inherited from class com.norconex.committer.core.AbstractMappedCommitter

Methods inherited from class com.norconex.committer.core.AbstractBatchCommitter

Methods inherited from class com.norconex.committer.core.AbstractFileQueueCommitter

Methods inherited from class com.norconex.committer.core.AbstractCommitter

Methods inherited from class java.lang.Object

Field Detail

FIELD_PATTERN

COULDSEARCH_ID_FIELD

DEFAULT_COULDSEARCH_CONTENT_FIELD

Constructor Detail

CloudSearchCommitter

CloudSearchCommitter

CloudSearchCommitter

Method Detail

getServiceEndpoint

setServiceEndpoint

getSigningRegion

setSigningRegion

getDocumentEndpoint

setDocumentEndpoint

getAccessKey

setAccessKey

getSecretKey

setSecretKey

setTargetReferenceField

isFixBadIds

setFixBadIds

getProxyHost

setProxyHost

getProxyPort

setProxyPort

getProxyUsername

setProxyUsername

getProxyPassword

setProxyPassword

getProxyPasswordKey

setProxyPasswordKey

commitBatch

buildClientConfiguration

saveToXML

loadFromXml

hashCode

equals

toString