com.norconex.collector.fs.crawler

Class FilesystemCrawlerConfig

java.lang.Object

com.norconex.collector.core.crawler.AbstractCrawlerConfig

com.norconex.collector.fs.crawler.FilesystemCrawlerConfig

All Implemented Interfaces:

com.norconex.collector.core.crawler.ICrawlerConfig, IXMLConfigurable

public class FilesystemCrawlerConfig
extends com.norconex.collector.core.crawler.AbstractCrawlerConfig

Filesystem Crawler configuration.

Author:

Pascal Essiembre

Nested Class Summary

Nested classes/interfaces inherited from interface com.norconex.collector.core.crawler.ICrawlerConfig

com.norconex.collector.core.crawler.ICrawlerConfig.OrphansStrategy

Constructor Summary

Constructors

Constructor and Description
FilesystemCrawlerConfig()

Method Summary

Modifier and Type	Method and Description
boolean	equals(Object other)
IFileDocumentFetcher	getDocumentFetcher() Gets the document fetcher.
com.norconex.collector.core.checksum.IMetadataChecksummer	getMetadataChecksummer()
IFileMetadataFetcher	getMetadataFetcher() Gets the document metadata fetcher.
IFilesystemOptionsProvider	getOptionsProvider() Gets the file system options provider.
String[]	getPathsFiles()
IFileDocumentProcessor[]	getPostImportProcessors()
IFileDocumentProcessor[]	getPreImportProcessors()
String[]	getStartPaths()
IStartPathsProvider[]	getStartPathsProviders() Gets the providers of paths used as starting points for crawling.
int	hashCode()
boolean	isKeepDownloads()
protected void	loadCrawlerConfigFromXML(XMLConfiguration xml)
protected void	saveCrawlerConfigToXML(Writer out)
void	setDocumentFetcher(IFileDocumentFetcher documentFetcher) Sets the document fetcher.
void	setKeepDownloads(boolean keepDownloads)
void	setMetadataChecksummer(com.norconex.collector.core.checksum.IMetadataChecksummer metadataChecksummer)
void	setMetadataFetcher(IFileMetadataFetcher metadataFetcher) Sets the document metadata fetcher.
void	setOptionsProvider(IFilesystemOptionsProvider filesystemOptionsProvider) Sets the file system options provider.
void	setPathsFiles(String[] pathsFiles)
void	setPostImportProcessors(IFileDocumentProcessor[] filePostProcessors)
void	setPreImportProcessors(IFileDocumentProcessor[] filePostProcessors)
void	setStartPaths(String[] startPaths)
void	setStartPathsProviders(IStartPathsProvider... startPathsProviders) Sets the providers of paths used as starting points for crawling.
String	toString()

Methods inherited from class com.norconex.collector.core.crawler.AbstractCrawlerConfig

defaultIfEmpty, getCommitter, getCrawlDataStoreFactory, getCrawlerListeners, getDocumentChecksummer, getDocumentFilters, getId, getImporterConfig, getMaxDocuments, getMetadataFilters, getNumThreads, getOrphansStrategy, getReferenceFilters, getSpoiledReferenceStrategizer, getStopOnExceptions, getWorkDir, loadFromXML, saveToXML, setCommitter, setCrawlDataStoreFactory, setCrawlerListeners, setDocumentChecksummer, setDocumentFilters, setId, setImporterConfig, setMaxDocuments, setMetadataFilters, setNumThreads, setOrphansStrategy, setReferenceFilters, setSpoiledReferenceStrategizer, setStopOnExceptions, setWorkDir, writeArray, writeObject, writeObject

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

FilesystemCrawlerConfig

public FilesystemCrawlerConfig()

Method Detail

getStartPaths

public String[] getStartPaths()

setStartPaths

public void setStartPaths(String[] startPaths)

getPathsFiles

public String[] getPathsFiles()

setPathsFiles

public void setPathsFiles(String[] pathsFiles)

getStartPathsProviders

public IStartPathsProvider[] getStartPathsProviders()

Gets the providers of paths used as starting points for crawling. Use this approach over other methods when paths need to be provided dynamicaly at launch time. Paths obtained by a provider are combined with start paths provided through other methods.

Returns:

a start paths provider

Since:

2.7.0

setStartPathsProviders

public void setStartPathsProviders(IStartPathsProvider... startPathsProviders)

Sets the providers of paths used as starting points for crawling. Use this approach over other methods when paths need to be provided dynamicaly at launch time. Paths obtained by a provider are combined with start paths provided through other methods.

Parameters:

startPathsProviders - start paths provider

Since:

2.7.0

isKeepDownloads

public boolean isKeepDownloads()

setKeepDownloads

public void setKeepDownloads(boolean keepDownloads)

getOptionsProvider

public IFilesystemOptionsProvider getOptionsProvider()

Gets the file system options provider. Default is GenericFilesystemOptionsProvider.

Returns:

file system options provider

Since:

2.7.0

setOptionsProvider

public void setOptionsProvider(IFilesystemOptionsProvider filesystemOptionsProvider)

Sets the file system options provider. Cannot be null.

Parameters:

filesystemOptionsProvider - file system options provider

Since:

2.7.0

getMetadataFetcher

public IFileMetadataFetcher getMetadataFetcher()

Gets the document metadata fetcher. Default is GenericFileMetadataFetcher.

Returns:

metadata fetcher

Since:

2.7.0

setMetadataFetcher

public void setMetadataFetcher(IFileMetadataFetcher metadataFetcher)

Sets the document metadata fetcher. Cannot be null.

Parameters:

metadataFetcher - metadata fetcher

Since:

2.7.0

getMetadataChecksummer

public com.norconex.collector.core.checksum.IMetadataChecksummer getMetadataChecksummer()

setMetadataChecksummer

public void setMetadataChecksummer(com.norconex.collector.core.checksum.IMetadataChecksummer metadataChecksummer)

getDocumentFetcher

public IFileDocumentFetcher getDocumentFetcher()

Gets the document fetcher. Default is GenericFileDocumentFetcher.

Returns:

document fetcher

Since:

2.7.0

setDocumentFetcher

public void setDocumentFetcher(IFileDocumentFetcher documentFetcher)

Sets the document fetcher. Cannot be null.

Parameters:

documentFetcher - document fetcher

Since:

2.7.0

getPreImportProcessors

public IFileDocumentProcessor[] getPreImportProcessors()

setPreImportProcessors

public void setPreImportProcessors(IFileDocumentProcessor[] filePostProcessors)

getPostImportProcessors

public IFileDocumentProcessor[] getPostImportProcessors()

setPostImportProcessors

public void setPostImportProcessors(IFileDocumentProcessor[] filePostProcessors)

saveCrawlerConfigToXML

protected void saveCrawlerConfigToXML(Writer out)
                               throws IOException

Specified by:

saveCrawlerConfigToXML in class com.norconex.collector.core.crawler.AbstractCrawlerConfig

Throws:

IOException

loadCrawlerConfigFromXML

protected void loadCrawlerConfigFromXML(XMLConfiguration xml)
                                 throws IOException

Specified by:

loadCrawlerConfigFromXML in class com.norconex.collector.core.crawler.AbstractCrawlerConfig

Throws:

IOException

equals

public boolean equals(Object other)

Overrides:

equals in class com.norconex.collector.core.crawler.AbstractCrawlerConfig

hashCode

public int hashCode()

Overrides:

hashCode in class com.norconex.collector.core.crawler.AbstractCrawlerConfig

toString

public String toString()

Overrides:

toString in class com.norconex.collector.core.crawler.AbstractCrawlerConfig