Norconex Crawler V4 is here, in beta. A rewritten crawler with a free visual configurator — give it a try and tell us what you think. Try the beta →
Norconex Web Crawler

Config Starter

Fill-in the following form to generate a basic configuration file for Web Crawler version 3. All fields are optional. You can manually add missing values from the generated configuration file afterwards.

Collect
Crawling will start at this URL.
Enter parts of URLs you do not want to crawl. One per field.
Remove
Add
Import
Increase focus on content by eliminating text from navigation elements.
Fields you want to send to your Committer. For each, you can specify one or more source fields (comma-separated) to be renamed (mapped) to the target one. Leave blank to send all fields as they are discovered.
Target field
Source document field(s)
Remove
Add
Commit
Select one or more Committers or none if you want to add one yourself.
For more, check out the list of Committers.