public class FSQueue extends Object implements ICommitterQueue, IXMLConfigurable
File System queue. Queues committer requests as zip files and deletes successfully committed zip files.
The top-level queue directory is the one defined in the
CommitterContext
initialization argument or the system
temporary directory if null
. A "queue" sub-folder will be
created for queued requests, while an "error" one will also be created
for failed batches.
Before moving a failed batch to the "error" folder you have the opportunity to have this queue tell the Committer to retry the batch. This can be particularly useful if you experience issues such as:
In addition to specifying how many time to retry and how long to wait between each attempts, you can also break the batch size into smaller chunks. The "splitBatch" setting offers the following options:
When "splitBatch" is not OFF and used in combination with a non-zero "maxRetry", the later then represents how many times to retry each new smaller batches created.
<queue
class="com.norconex.committer.core3.batch.queue.impl.FSQueue">
<batchSize>
(Optional number of documents queued after which we process a batch.
Default is 20.)
</batchSize>
<maxPerFolder>
(Optional maximum number of files or directories that can be queued
in a single folder before a new one gets created. Default is 500.)
</maxPerFolder>
<commitLeftoversOnInit>
(Optionally force to commit any leftover documents from a previous
execution. E.g., prematurely ended. Default is "false").
</commitLeftoversOnInit>
<onCommitFailure>
<splitBatch>[OFF|HALF|ONE]</splitBatch>
<maxRetries>(Max retries upon commit failures. Default is 0.)</maxRetries>
<retryDelay>
(Delay in milliseconds between retries. Default is 0.)
</retryDelay>
<ignoreErrors>
[false|true]
(When true, non-critical exceptions when interacting with the target
repository won't be thrown to try continue the execution with other
files to be committed. Instead, errors will be logged.
In both cases the failing batch/files are moved to an
"error" folder. Other types of exceptions may still be thrown.)
</ignoreErrors>
</onCommitFailure>
</queue>
Modifier and Type | Class and Description |
---|---|
static class |
FSQueue.SplitBatch |
Modifier and Type | Field and Description |
---|---|
static int |
DEFAULT_BATCH_SIZE |
static int |
DEFAULT_MAX_PER_FOLDER |
Constructor and Description |
---|
FSQueue() |
Modifier and Type | Method and Description |
---|---|
void |
clean()
Cleans any persisted information specific to this queue.
|
void |
close() |
boolean |
equals(Object other) |
IBatchConsumer |
getBatchConsumer() |
int |
getBatchSize()
Gets the number of documents to be queued in a batch on disk before
consuming that batch.
|
int |
getMaxPerFolder()
Gets the maximum number of files to be queued on disk in a given folders.
|
int |
getMaxRetries() |
long |
getRetryDelay() |
FSQueue.SplitBatch |
getSplitBatch() |
int |
hashCode() |
void |
init(CommitterContext committerContext,
IBatchConsumer batchConsumer) |
boolean |
isCommitLeftoversOnInit() |
boolean |
isIgnoreErrors() |
void |
loadFromXML(XML xml) |
void |
queue(ICommitterRequest request) |
void |
saveToXML(XML xml) |
void |
setBatchSize(int batchSize)
Sets the number of documents to be queued in a batch on disk before
consuming that batch.
|
void |
setCommitLeftoversOnInit(boolean commitLeftoversOnInit) |
void |
setIgnoreErrors(boolean ignoreErrors) |
void |
setMaxPerFolder(int maxPerFolder)
Sets the maximum number of files to be queued on disk in a given folders.
|
void |
setMaxRetries(int maxRetries) |
void |
setRetryDelay(long retryDelay) |
void |
setSplitBatch(FSQueue.SplitBatch splitBatch) |
String |
toString() |
public static final int DEFAULT_BATCH_SIZE
public static final int DEFAULT_MAX_PER_FOLDER
public void init(CommitterContext committerContext, IBatchConsumer batchConsumer) throws CommitterQueueException
init
in interface ICommitterQueue
CommitterQueueException
public IBatchConsumer getBatchConsumer()
public int getBatchSize()
public void setBatchSize(int batchSize)
batchSize
- the batch sizepublic int getMaxPerFolder()
public void setMaxPerFolder(int maxPerFolder)
maxPerFolder
- number of files queued per directorypublic boolean isCommitLeftoversOnInit()
public void setCommitLeftoversOnInit(boolean commitLeftoversOnInit)
public int getMaxRetries()
public void setMaxRetries(int maxRetries)
public long getRetryDelay()
public void setRetryDelay(long retryDelay)
public FSQueue.SplitBatch getSplitBatch()
public void setSplitBatch(FSQueue.SplitBatch splitBatch)
public boolean isIgnoreErrors()
public void setIgnoreErrors(boolean ignoreErrors)
public void queue(ICommitterRequest request) throws CommitterQueueException
queue
in interface ICommitterQueue
CommitterQueueException
public void loadFromXML(XML xml)
loadFromXML
in interface IXMLConfigurable
public void saveToXML(XML xml)
saveToXML
in interface IXMLConfigurable
public void close() throws CommitterQueueException
close
in interface ICommitterQueue
close
in interface AutoCloseable
CommitterQueueException
public void clean() throws CommitterQueueException
ICommitterQueue
clean
in interface ICommitterQueue
CommitterQueueException
- could not clean queueCopyright © 2009–2022 Norconex Inc.. All rights reserved.