Package com.norconex.collector.fs.crawler

Class FilesystemCrawler

java.lang.Object

com.norconex.collector.core.crawler.Crawler

com.norconex.collector.fs.crawler.FilesystemCrawler

public class FilesystemCrawler
extends Crawler

The Filesystem Crawler.

Author:

Pascal Essiembre

Nested Class Summary

Nested classes/interfaces inherited from class com.norconex.collector.core.crawler.Crawler

Crawler.ReferenceProcessStatus

Constructor Summary

Constructors

Constructor	Description
FilesystemCrawler(FilesystemCrawlerConfig crawlerConfig, Collector collector)	Constructor.

Method Summary

Modifier and Type	Method	Description
protected void	afterCrawlerExecution()
protected void	beforeCrawlerExecution(boolean resume)
protected CrawlDocInfo	createChildDocInfo(String embeddedReference, CrawlDocInfo parentCrawlData)
protected void	executeCommitterPipeline(Crawler crawler, CrawlDoc doc)
protected ImporterResponse	executeImporterPipeline(ImporterPipelineContext importerContext)
protected void	executeQueuePipeline(CrawlDocInfo crawlData)
FilesystemCrawlerConfig	getCrawlerConfig()
org.apache.commons.vfs2.FileSystemManager	getFileManager()
protected void	markReferenceVariationsAsProcessed(CrawlDocInfo crawlData)

Methods inherited from class com.norconex.collector.core.crawler.Crawler

beforeFinalizeDocumentProcessing, clean, deleteCacheOrphans, destroyCrawler, doExecute, exportDataStore, getCollector, getCommitterService, getCrawlDocInfoType, getDataStoreEngine, getDocInfoService, getDownloadDir, getEventManager, getId, getImporter, getMonitor, getStreamFactory, getTempDir, getWorkDir, handleOrphans, importDataStore, initCrawlDoc, initCrawler, isMaxDocuments, isQueueInitialized, isStopped, processNextReference, processReferences, reprocessCacheOrphans, start, stop, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

FilesystemCrawler

public FilesystemCrawler(FilesystemCrawlerConfig crawlerConfig,
 Collector collector)

Constructor.

Parameters:

crawlerConfig - HTTP crawler configuration

Method Details

getCrawlerConfig

public FilesystemCrawlerConfig getCrawlerConfig()

Overrides:

getCrawlerConfig in class Crawler

getFileManager

public org.apache.commons.vfs2.FileSystemManager getFileManager()

Returns:

the fileManager

beforeCrawlerExecution

protected void beforeCrawlerExecution(boolean resume)

Specified by:

beforeCrawlerExecution in class Crawler

afterCrawlerExecution

protected void afterCrawlerExecution()

Specified by:

afterCrawlerExecution in class Crawler

executeQueuePipeline

protected void executeQueuePipeline(CrawlDocInfo crawlData)

Specified by:

executeQueuePipeline in class Crawler

executeImporterPipeline

protected ImporterResponse executeImporterPipeline(ImporterPipelineContext importerContext)

Specified by:

executeImporterPipeline in class Crawler

createChildDocInfo

protected CrawlDocInfo createChildDocInfo(String embeddedReference,
 CrawlDocInfo parentCrawlData)

Specified by:

createChildDocInfo in class Crawler

executeCommitterPipeline

protected void executeCommitterPipeline(Crawler crawler,
 CrawlDoc doc)

Specified by:

executeCommitterPipeline in class Crawler

markReferenceVariationsAsProcessed

protected void markReferenceVariationsAsProcessed(CrawlDocInfo crawlData)

Specified by:

markReferenceVariationsAsProcessed in class Crawler