Norconex Filesystem Collector is a flexible crawler for collecting,
parsing, and manipulating data ranging from local hard drives to network
locations into various data repositories such as search engines.
Norconex Filesystem Collector shares common features with other Norconex
Collectors. Find out about those
The following is a non exhaustive list of features supported by
the Norconex Filesystem Collector:
- Extract text out of many file formats (HTML, PDF, Word, etc.)
- Extract metadata associated with documents.
- Language detection.
- OCR support on images and PDFs.
- Translation support.
- Dynamic title generation.
- Allow easy text transformation and metadata manipulation.
- Filters unwanted document.
- Detects modified and deleted documents.
- Easy to add your own features.
- Can be used command-line or embedded in your Java application.
- Can treat embedded documents as distinct documents.
- Can split a formatted document into multiple documents.
- Can store crawled URLs in different database engines.
- Fires crawler event types for custom listeners.
- Date parsers/formatters to match your source/target repository dates.
- Can create hierarchical fields.
- Supports scripting languages for manipulating documents.
- Reference XML/HTML elements using simple DOM tree navigation.
- Extract document ACL from SMB/CIFS file systems.
- Many others.
- Local files
- BZIP2, GZIP
- FTP, FTPS
- HTTP, HTTPS
- Zip, Jar and Tar