Norconex Filesystem Collector

Open-Source Enterprise Filesystem Crawler

Getting Started Download 2.9.0

Crawl Filesystems

Norconex Filesystem Collector is a flexible crawler for collecting, parsing, and manipulating data ranging from local hard drives to network locations into various data repositories such as search engines.


Features

Norconex Filesystem Collector shares common features with other Norconex Collectors. Find out about those here. The following is a non exhaustive list of features supported by the Norconex Filesystem Collector:

  • Multi-threaded
  • Extract text out of many file formats (HTML, PDF, Word, etc.)
  • Extract metadata associated with documents.
  • Language detection.
  • OCR support on images and PDFs.
  • Translation support.
  • Dynamic title generation.
  • Allow easy text transformation and metadata manipulation.
  • Filters unwanted document.
  • Detects modified and deleted documents.
  • Easy to add your own features.
  • Can be used command-line or embedded in your Java application.
  • Can treat embedded documents as distinct documents.
  • Can split a formatted document into multiple documents.
  • Can store crawled URLs in different database engines.
  • Fires crawler event types for custom listeners.
  • Date parsers/formatters to match your source/target repository dates.
  • Can create hierarchical fields.
  • Supports scripting languages for manipulating documents.
  • Reference XML/HTML elements using simple DOM tree navigation.
  • Extract document ACL from SMB/CIFS file systems.
  • Many others.

Supported Filesystems

  • Local files
  • BZIP2, GZIP
  • FTP, FTPS
  • HDFS
  • HTTP, HTTPS
  • RAM
  • RES
  • SFTP
  • Temp
  • WebDAV
  • Zip, Jar and Tar
  • SMB/CIFS

Latest news

Norconex HTTP Collector 3.0.0 snapshots available
2020-09-07
Development builds of upcoming version 3 now available to experiment with. More...

opensource.norconex.com
2020-09-07
All Norconex open-source projects are now grouped under the same domain. More...

Norconex HTTP and FileSystem Collectors 2.9.0 released
2019-12-22
New URL normalization rules, support for CMIS protocol, ACL extraction from more sources, etc. More...

Norconex Collector Core 1.10.0 released
2019-12-22
Unmanaged logs, max parallel crawlers, etc. More...

Norconex Importer 2.10.0 released
2019-12-22
New FieldReportTagger, Tika upgrade, fixes, etc. More...