Crawl File Systems
Use Norconex flexible open-source enterprise file system crawler
for collecting, parsing, and manipulating data ranging from local hard
drives to network locations into various data repositories such as
search engines.
Features
Norconex File System Crawler shares common features with other Norconex
Crawlers. Find out about those
here.
The following is a non exhaustive list of features supported by
the Norconex File System Crawler:
- Multi-threaded
- Extract text out of many file formats (HTML, PDF, Word, etc.)
- Extract metadata associated with documents.
- Language detection.
- OCR support on images and PDFs.
- Translation support.
- Dynamic title generation.
- Allow easy text transformation and metadata manipulation.
- Filters unwanted document.
- Detects modified and deleted documents.
- Easy to add your own features.
- Can be used command-line or embedded in your Java application.
- Can treat embedded documents as distinct documents.
- Can split a formatted document into multiple documents.
- Can store crawled URLs in different database engines.
- Fires crawler event types for custom listeners.
- Date parsers/formatters to match your source/target repository dates.
- Can create hierarchical fields.
- Supports scripting languages for manipulating documents.
- Reference XML/HTML elements using simple DOM tree navigation.
- Extract document ACL from SMB/CIFS file systems.
- Many others.
Supported Filesystems
- Local files
- BZIP2, GZIP
- FTP, FTPS
- HDFS
- HTTP, HTTPS
- RAM
- RES
- SFTP
- Temp
- WebDAV
- Zip, Jar and Tar
- SMB/CIFS