public abstract class AbstractBatchCommitter extends AbstractCommitter implements IXMLConfigurable, IBatchConsumer
A base implementation for doing batch commits. Uses an internal queue for storing update/addition requests and deletion requests. It sends the queued data to the remote target every time a given queue threshold has been reached. Unless otherwise stated, both additions and deletions count towards that threshold.
This class also provides batch-related events:
COMMITTER_BATCH_BEGIN
,
COMMITTER_BATCH_END
, and
COMMITTER_BATCH_ERROR
.
The default queue is FSQueue
(file-system queue).
Optionally apply a committer only to certain type of documents. Documents are restricted based on their metadata field names and values. This option can be used to perform document routing when you have multiple committers defined.
By default, this abstract class applies field mappings for metadata fields, but leaves the document reference and content (input stream) for concrete implementations to handle. In other words, they only apply to a committer request metadata. Field mappings are performed on committer requests before upserts and deletes are actually performed.
Subclasses inherits this IXMLConfigurable
configuration:
<!-- multiple "restrictTo" tags allowed (only one needs to match) -->
<restrictTo>
<fieldMatcher
method="[basic|csv|wildcard|regex]"
ignoreCase="[false|true]"
ignoreDiacritic="[false|true]"
partial="[false|true]">
(field-matching expression)
</fieldMatcher>
<valueMatcher
method="[basic|csv|wildcard|regex]"
ignoreCase="[false|true]"
ignoreDiacritic="[false|true]"
partial="[false|true]">
(value-matching expression)
</valueMatcher>
</restrictTo>
<fieldMappings>
<!-- Add as many field mappings as needed -->
<mapping
fromField="(source field name)"
toField="(target field name)"/>
</fieldMappings>
<!-- Settings for default queue implementation ("class" is optional): -->
<queue
class="com.norconex.committer.core3.batch.queue.impl.FSQueue">
<batchSize>
(Optional number of documents queued after which we process a batch.
Default is 20.)
</batchSize>
<maxPerFolder>
(Optional maximum number of files or directories that can be queued
in a single folder before a new one gets created. Default is 500.)
</maxPerFolder>
<commitLeftoversOnInit>
(Optionally force to commit any leftover documents from a previous
execution. E.g., prematurely ended. Default is "false").
</commitLeftoversOnInit>
<onCommitFailure>
<splitBatch>[OFF|HALF|ONE]</splitBatch>
<maxRetries>(Max retries upon commit failures. Default is 0.)</maxRetries>
<retryDelay>
(Delay in milliseconds between retries. Default is 0.)
</retryDelay>
<ignoreErrors>
[false|true]
(When true, non-critical exceptions when interacting with the target
repository won't be thrown to try continue the execution with other
files to be committed. Instead, errors will be logged.
In both cases the failing batch/files are moved to an
"error" folder. Other types of exceptions may still be thrown.)
</ignoreErrors>
</onCommitFailure>
</queue>
Constructor and Description |
---|
AbstractBatchCommitter() |
Modifier and Type | Method and Description |
---|---|
protected void |
closeBatchCommitter()
Subclasses can perform additional closing logic by overriding this
method.
|
protected abstract void |
commitBatch(Iterator<ICommitterRequest> it) |
void |
consume(Iterator<ICommitterRequest> it) |
protected void |
doClean() |
protected void |
doClose()
Subclasses can perform additional closing logic by overriding this
method.
|
protected void |
doDelete(DeleteRequest deleteRequest) |
protected void |
doInit()
Subclasses can perform additional initialization by overriding this
method.
|
protected void |
doUpsert(UpsertRequest upsertRequest) |
boolean |
equals(Object other) |
ICommitterQueue |
getCommitterQueue() |
int |
hashCode() |
protected void |
initBatchCommitter()
Subclasses can perform additional initialization by overriding this
method.
|
protected abstract void |
loadBatchCommitterFromXML(XML xml) |
void |
loadCommitterFromXML(XML xml) |
protected abstract void |
saveBatchCommitterToXML(XML xml) |
void |
saveCommitterToXML(XML xml) |
void |
setCommitterQueue(ICommitterQueue queue) |
String |
toString() |
accept, addRestriction, addRestrictions, applyFieldMappings, clean, clearFieldMappings, clearRestrictions, close, delete, fireDebug, fireDebug, fireError, fireError, fireInfo, fireInfo, getCommitterContext, getFieldMappings, getRestrictions, init, loadFromXML, removeFieldMapping, removeRestriction, removeRestriction, saveToXML, setFieldMapping, setFieldMappings, upsert
clone, finalize, getClass, notify, notifyAll, wait, wait, wait
loadFromXML, saveToXML
protected final void doInit() throws CommitterException
AbstractCommitter
CommitterContext
will be initialized when invoking
AbstractCommitter.getCommitterContext()
doInit
in class AbstractCommitter
CommitterException
- error initializingprotected void doUpsert(UpsertRequest upsertRequest) throws CommitterException
doUpsert
in class AbstractCommitter
CommitterException
protected void doDelete(DeleteRequest deleteRequest) throws CommitterException
doDelete
in class AbstractCommitter
CommitterException
protected void doClose() throws CommitterException
AbstractCommitter
doClose
in class AbstractCommitter
CommitterException
- error closing committerprotected void doClean() throws CommitterException
doClean
in class AbstractCommitter
CommitterException
public void consume(Iterator<ICommitterRequest> it) throws CommitterException
consume
in interface IBatchConsumer
CommitterException
public final void loadCommitterFromXML(XML xml)
loadCommitterFromXML
in class AbstractCommitter
public final void saveCommitterToXML(XML xml)
saveCommitterToXML
in class AbstractCommitter
protected abstract void loadBatchCommitterFromXML(XML xml)
protected abstract void saveBatchCommitterToXML(XML xml)
public ICommitterQueue getCommitterQueue()
public void setCommitterQueue(ICommitterQueue queue)
protected void initBatchCommitter() throws CommitterException
AbstractCommitter.getCommitterContext()
and getCommitterQueue()
,
respectively.CommitterException
- error initializingprotected abstract void commitBatch(Iterator<ICommitterRequest> it) throws CommitterException
CommitterException
protected void closeBatchCommitter() throws CommitterException
CommitterException
- error closing committerpublic boolean equals(Object other)
equals
in class AbstractCommitter
public int hashCode()
hashCode
in class AbstractCommitter
public String toString()
toString
in class AbstractCommitter
Copyright © 2009–2022 Norconex Inc.. All rights reserved.