public class MVStoreCrawlDataStore extends AbstractCrawlDataStore
ICrawlDataStore
implementation.Constructor and Description |
---|
MVStoreCrawlDataStore(String path,
boolean resume) |
MVStoreCrawlDataStore(String path,
boolean resume,
MVStoreConfig mvStoreConfig) |
Modifier and Type | Method and Description |
---|---|
void |
close()
Closes a database connection.
|
int |
getActiveCount()
Gets the number of active references (currently being processed).
|
ICrawlData |
getCached(String cacheReference)
Gets the cached reference from previous time crawler was run
(e.g.
|
Iterator<ICrawlData> |
getCacheIterator()
Gets the cache iterator.
|
ICrawlData |
getProcessed(String reference)
Gets an already processed reference from the current crawl session.
|
int |
getProcessedCount()
Gets the number of references processed.
|
int |
getQueueSize()
Gets the size of the reference queue (number of
references left to process).
|
boolean |
isActive(String reference)
Whether the given reference is currently being processed (i.e.
|
boolean |
isCacheEmpty()
Whether there are any references the the cache from a previous crawler
run.
|
boolean |
isProcessed(String reference)
Whether the given reference has been processed.
|
boolean |
isQueued(String reference)
Whether the given reference is in the queue or not
(waiting to be processed).
|
boolean |
isQueueEmpty()
Whether there are any references to process in the queue.
|
ICrawlData |
nextQueued()
Returns the next reference to be processed from the queue and marks it as
being "active" (i.e.
|
void |
processed(ICrawlData crawlData)
Marks this reference as processed.
|
void |
queue(ICrawlData crawlData)
Queues a reference for future processing.
|
public MVStoreCrawlDataStore(String path, boolean resume)
public MVStoreCrawlDataStore(String path, boolean resume, MVStoreConfig mvStoreConfig)
public void queue(ICrawlData crawlData)
ICrawlDataStore
Queues a reference for future processing.
crawlData
- the reference to eventually be processedpublic boolean isQueueEmpty()
ICrawlDataStore
true
if the queue is emptypublic int getQueueSize()
ICrawlDataStore
public boolean isQueued(String reference)
ICrawlDataStore
reference
- the referencetrue
if the reference is in the queuepublic ICrawlData nextQueued()
ICrawlDataStore
public boolean isActive(String reference)
ICrawlDataStore
reference
- the referencetrue
if activepublic int getActiveCount()
ICrawlDataStore
public ICrawlData getCached(String cacheReference)
ICrawlDataStore
cacheReference
- reference cached from previous runpublic boolean isCacheEmpty()
ICrawlDataStore
true
if the cache is emptypublic ICrawlData getProcessed(String reference)
ICrawlDataStore
reference
- reference to getpublic void processed(ICrawlData crawlData)
ICrawlDataStore
crawlData
- processed referencepublic boolean isProcessed(String reference)
ICrawlDataStore
reference
- the referencetrue
if processedpublic int getProcessedCount()
ICrawlDataStore
public Iterator<ICrawlData> getCacheIterator()
ICrawlDataStore
public void close()
ICrawlDataStore
Copyright © 2014–2021 Norconex Inc.. All rights reserved.