com.norconex.collector.fs.crawler

Class FilesystemCrawler

java.lang.Object

com.norconex.jef4.job.AbstractResumableJob

com.norconex.collector.core.crawler.AbstractCrawler

com.norconex.collector.fs.crawler.FilesystemCrawler

All Implemented Interfaces:

com.norconex.collector.core.crawler.ICrawler, IJob

public class FilesystemCrawler
extends com.norconex.collector.core.crawler.AbstractCrawler

The Filesystem Crawler.

Author:

Pascal Essiembre

Nested Class Summary

Nested classes/interfaces inherited from class com.norconex.collector.core.crawler.AbstractCrawler

com.norconex.collector.core.crawler.AbstractCrawler.CopyIfNullBeanUtilsBean

Constructor Summary

Constructors

Constructor and Description
FilesystemCrawler(FilesystemCrawlerConfig crawlerConfig) Constructor.

Method Summary

Modifier and Type	Method and Description
protected void	cleanupExecution(JobStatusUpdater statusUpdater, JobSuite suite, com.norconex.collector.core.data.store.ICrawlDataStore refStore)
protected com.norconex.collector.core.data.BaseCrawlData	createEmbeddedCrawlData(String embeddedReference, com.norconex.collector.core.data.ICrawlData parentCrawlData)
protected void	executeCommitterPipeline(com.norconex.collector.core.crawler.ICrawler crawler, ImporterDocument doc, com.norconex.collector.core.data.store.ICrawlDataStore crawlDataStore, com.norconex.collector.core.data.BaseCrawlData crawlData, com.norconex.collector.core.data.BaseCrawlData cachedCrawlData)
protected ImporterResponse	executeImporterPipeline(com.norconex.collector.core.pipeline.importer.ImporterPipelineContext importerContext)
protected void	executeQueuePipeline(com.norconex.collector.core.data.ICrawlData crawlData, com.norconex.collector.core.data.store.ICrawlDataStore crawlDataStore)
FilesystemCrawlerConfig	getCrawlerConfig()
org.apache.commons.vfs2.FileSystemManager	getFileManager()
protected void	markReferenceVariationsAsProcessed(com.norconex.collector.core.data.BaseCrawlData crawlData, com.norconex.collector.core.data.store.ICrawlDataStore crawlDataStore)
protected void	prepareExecution(JobStatusUpdater statusUpdater, JobSuite suite, com.norconex.collector.core.data.store.ICrawlDataStore crawlDataStore, boolean resume)
protected ImporterDocument	wrapDocument(com.norconex.collector.core.data.ICrawlData crawlData, ImporterDocument document)

Methods inherited from class com.norconex.collector.core.crawler.AbstractCrawler

beforeFinalizeDocumentProcessing, createCrawlDataStore, deleteCacheOrphans, execute, fireCrawlerEvent, getBaseDownloadDir, getCrawlerDownloadDir, getCrawlerEventManager, getId, getImporter, getStreamFactory, handleOrphans, initCrawlData, isMaxDocuments, isStopped, processNextReference, processReferences, reprocessCacheOrphans, resumeExecution, startExecution, stop

Methods inherited from class com.norconex.jef4.job.AbstractResumableJob

execute

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.norconex.jef4.job.IJob

execute

Constructor Detail

FilesystemCrawler

public FilesystemCrawler(FilesystemCrawlerConfig crawlerConfig)

Constructor.

Parameters:

crawlerConfig - HTTP crawler configuration

Method Detail

getCrawlerConfig

public FilesystemCrawlerConfig getCrawlerConfig()

Specified by:

getCrawlerConfig in interface com.norconex.collector.core.crawler.ICrawler

Overrides:

getCrawlerConfig in class com.norconex.collector.core.crawler.AbstractCrawler

getFileManager

public org.apache.commons.vfs2.FileSystemManager getFileManager()

Returns:

the fileManager

prepareExecution

protected void prepareExecution(JobStatusUpdater statusUpdater,
                                JobSuite suite,
                                com.norconex.collector.core.data.store.ICrawlDataStore crawlDataStore,
                                boolean resume)

Specified by:

prepareExecution in class com.norconex.collector.core.crawler.AbstractCrawler

executeQueuePipeline

protected void executeQueuePipeline(com.norconex.collector.core.data.ICrawlData crawlData,
                                    com.norconex.collector.core.data.store.ICrawlDataStore crawlDataStore)

Specified by:

executeQueuePipeline in class com.norconex.collector.core.crawler.AbstractCrawler

wrapDocument

protected ImporterDocument wrapDocument(com.norconex.collector.core.data.ICrawlData crawlData,
                                        ImporterDocument document)

Specified by:

wrapDocument in class com.norconex.collector.core.crawler.AbstractCrawler

executeImporterPipeline

protected ImporterResponse executeImporterPipeline(com.norconex.collector.core.pipeline.importer.ImporterPipelineContext importerContext)

Specified by:

executeImporterPipeline in class com.norconex.collector.core.crawler.AbstractCrawler

createEmbeddedCrawlData

protected com.norconex.collector.core.data.BaseCrawlData createEmbeddedCrawlData(String embeddedReference,
                                                                                 com.norconex.collector.core.data.ICrawlData parentCrawlData)

Specified by:

createEmbeddedCrawlData in class com.norconex.collector.core.crawler.AbstractCrawler

executeCommitterPipeline

protected void executeCommitterPipeline(com.norconex.collector.core.crawler.ICrawler crawler,
                                        ImporterDocument doc,
                                        com.norconex.collector.core.data.store.ICrawlDataStore crawlDataStore,
                                        com.norconex.collector.core.data.BaseCrawlData crawlData,
                                        com.norconex.collector.core.data.BaseCrawlData cachedCrawlData)

Specified by:

executeCommitterPipeline in class com.norconex.collector.core.crawler.AbstractCrawler

markReferenceVariationsAsProcessed

protected void markReferenceVariationsAsProcessed(com.norconex.collector.core.data.BaseCrawlData crawlData,
                                                  com.norconex.collector.core.data.store.ICrawlDataStore crawlDataStore)

Specified by:

markReferenceVariationsAsProcessed in class com.norconex.collector.core.crawler.AbstractCrawler

cleanupExecution

protected void cleanupExecution(JobStatusUpdater statusUpdater,
                                JobSuite suite,
                                com.norconex.collector.core.data.store.ICrawlDataStore refStore)

Specified by:

cleanupExecution in class com.norconex.collector.core.crawler.AbstractCrawler