Package com.norconex.collector.fs.crawler

Class FilesystemCrawlerConfig

java.lang.Object
com.norconex.collector.core.crawler.CrawlerConfig
com.norconex.collector.fs.crawler.FilesystemCrawlerConfig
All Implemented Interfaces:
IXMLConfigurable
public class FilesystemCrawlerConfig extends CrawlerConfig
Filesystem Crawler configuration.
Author:
Pascal Essiembre

  • Constructor Details

    • FilesystemCrawlerConfig

      public FilesystemCrawlerConfig()

  • Method Details

    • getStartPaths

      public String[] getStartPaths()

    • setStartPaths

      public void setStartPaths(String[] startPaths)

    • getPathsFiles

      public String[] getPathsFiles()

    • setPathsFiles

      public void setPathsFiles(String[] pathsFiles)

    • getStartPathsProviders

      public IStartPathsProvider[] getStartPathsProviders()
      Gets the providers of paths used as starting points for crawling. Use this approach over other methods when paths need to be provided dynamicaly at launch time. Paths obtained by a provider are combined with start paths provided through other methods.
      Returns:
      a start paths provider
      Since:
      2.7.0

    • setStartPathsProviders

      public void setStartPathsProviders(IStartPathsProvider... startPathsProviders)
      Sets the providers of paths used as starting points for crawling. Use this approach over other methods when paths need to be provided dynamicaly at launch time. Paths obtained by a provider are combined with start paths provided through other methods.
      Parameters:
      startPathsProviders - start paths provider
      Since:
      2.7.0

    • isKeepDownloads

      public boolean isKeepDownloads()

    • setKeepDownloads

      public void setKeepDownloads(boolean keepDownloads)

    • getOptionsProvider

      public IFilesystemOptionsProvider getOptionsProvider()
      Gets the file system options provider. Default is GenericFilesystemOptionsProvider.
      Returns:
      file system options provider
      Since:
      2.7.0

    • setOptionsProvider

      public void setOptionsProvider(IFilesystemOptionsProvider filesystemOptionsProvider)
      Sets the file system options provider. Cannot be null.
      Parameters:
      filesystemOptionsProvider - file system options provider
      Since:
      2.7.0

    • getMetadataFetcher

      public IFileMetadataFetcher getMetadataFetcher()
      Gets the document metadata fetcher. Default is GenericFileMetadataFetcher.
      Returns:
      metadata fetcher
      Since:
      2.7.0

    • setMetadataFetcher

      public void setMetadataFetcher(IFileMetadataFetcher metadataFetcher)
      Sets the document metadata fetcher. Cannot be null.
      Parameters:
      metadataFetcher - metadata fetcher
      Since:
      2.7.0

    • getMetadataChecksummer

      public IMetadataChecksummer getMetadataChecksummer()
      Overrides:
      getMetadataChecksummer in class CrawlerConfig

    • setMetadataChecksummer

      public void setMetadataChecksummer(IMetadataChecksummer metadataChecksummer)
      Overrides:
      setMetadataChecksummer in class CrawlerConfig

    • getDocumentFetcher

      public IFileDocumentFetcher getDocumentFetcher()
      Gets the document fetcher. Default is GenericFileDocumentFetcher.
      Returns:
      document fetcher
      Since:
      2.7.0

    • setDocumentFetcher

      public void setDocumentFetcher(IFileDocumentFetcher documentFetcher)
      Sets the document fetcher. Cannot be null.
      Parameters:
      documentFetcher - document fetcher
      Since:
      2.7.0

    • getPreImportProcessors

      public IFileDocumentProcessor[] getPreImportProcessors()

    • setPreImportProcessors

      public void setPreImportProcessors(IFileDocumentProcessor[] filePostProcessors)

    • getPostImportProcessors

      public IFileDocumentProcessor[] getPostImportProcessors()

    • setPostImportProcessors

      public void setPostImportProcessors(IFileDocumentProcessor[] filePostProcessors)

    • saveCrawlerConfigToXML

      protected void saveCrawlerConfigToXML(XML xml)
      Specified by:
      saveCrawlerConfigToXML in class CrawlerConfig

    • loadCrawlerConfigFromXML

      protected void loadCrawlerConfigFromXML(XML xml)
      Specified by:
      loadCrawlerConfigFromXML in class CrawlerConfig

    • equals

      public boolean equals(Object other)
      Overrides:
      equals in class CrawlerConfig

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class CrawlerConfig

    • toString

      public String toString()
      Overrides:
      toString in class CrawlerConfig