Norconex File System Crawler

Open-Source Enterprise Crawler (AKA Norconex Filesystem Collector)

Getting Started Download

Crawl File Systems

Use Norconex flexible open-source enterprise file system crawler for collecting, parsing, and manipulating data ranging from local hard drives to network locations into various data repositories such as search engines.


Features

Norconex File System Crawler shares common features with other Norconex Crawlers. Find out about those here. The following is a non exhaustive list of features supported by the Norconex File System Crawler:

  • Multi-threaded
  • Extract text out of many file formats (HTML, PDF, Word, etc.)
  • Extract metadata associated with documents.
  • Language detection.
  • OCR support on images and PDFs.
  • Translation support.
  • Dynamic title generation.
  • Allow easy text transformation and metadata manipulation.
  • Filters unwanted document.
  • Detects modified and deleted documents.
  • Easy to add your own features.
  • Can be used command-line or embedded in your Java application.
  • Can treat embedded documents as distinct documents.
  • Can split a formatted document into multiple documents.
  • Can store crawled URLs in different database engines.
  • Fires crawler event types for custom listeners.
  • Date parsers/formatters to match your source/target repository dates.
  • Can create hierarchical fields.
  • Supports scripting languages for manipulating documents.
  • Reference XML/HTML elements using simple DOM tree navigation.
  • Extract document ACL from SMB/CIFS file systems.
  • Many others.

Supported Filesystems

  • Local files
  • BZIP2, GZIP
  • FTP, FTPS
  • HDFS
  • HTTP, HTTPS
  • RAM
  • RES
  • SFTP
  • Temp
  • WebDAV
  • Zip, Jar and Tar
  • SMB/CIFS

Latest news

Norconex Web Crawler 3.0.0 Released!
2022-01-05
The new major release of Norconex HTTP Collector it is finally here. Check out what's new. More...

Norconex HTTP Collector 3.0.0 Release Candidate 1
2021-10-10
Is this the last pre-release? Put it to the test and let us know! Includes applicable release candidates of core dependencies as well. More...

Norconex HTTP Collector 3.0.0 Milestone 2
2021-07-28
3.0.0 second milestone release. Includes applicable milestone releases of core dependencies as well. More...

Norconex HTTP Collector 3.0.0 Milestone 1
2021-03-01
A step closer to final release. Available with milestone releases of core dependencies as well. More...

Norconex HTTP Collector 3.0.0 snapshots available
2020-09-07
Development builds of upcoming version 3 now available to experiment with. More...