Getting Started

Command Line Usage

(Replace collector-http.sh with collector-http.bat on Windows)

Usage: collector-http.sh [-hv] [COMMAND]
 
Options:
  -h, -help      Show usage help message and exit
  -v, -version   Show the Collector version and exit
 
Commands:
  help          Displays help information about the specified command
  start         Start the Collector
  stop          Stop the Collector
  configcheck   Validate configuration file syntax
  configrender  Render effective configuration
  clean         Clean the Collector crawling history (to start fresh)
  storeexport   Export crawl store to specified directory
  storeimport   Import crawl store from specified files
 
Examples:
 
  Start the Collector:
 
    collector-http.sh start -config=/path/to/config.xml
 
  Stop the Collector:
 
    collector-http.sh stop -config=/path/to/config.xml
 
  Get usage help on "check" command:
 
    collector-http.sh help configcheck

The above collector-http.sh (or collector-http.bat on Windows) startup script is found in the root directory of your installation (where you extracted the ZIP you downloaded). Refer to the Configuration pages for documentation on all available options. Refer to ConfigurationLoader Javadoc for details on the optional variables file.

Working Command-Line Examples

You may prefer to start with a working example and modify it to fit your need. Two such examples are provided in the examples for testing your Norconex Web Crawler installation. The first one represents a very basic configuration and the second shows how you can share configuration fragments.

To run the tests, go to the root directory where you uncompressed the distribution ZIP and run the following in a command-line console.

(Replace collector-http.sh with collector-http.bat on Windows)

# Minimal test:
collector-http.sh start -config=examples/minimum/minimum-config.xml
 
# Complex test:
collector-http.sh start -config=examples/complex/complex-config.xml

You need Internet access to run the examples. Modify the example files if you wish to run them on a private network.

Java Integration (for programmers)

If you are using Maven, simply add the project dependency to your pom.xml. If you are not using Maven, you can add all JAR files found in your installation "lib" folder to your application classpath. Configure the HttpCollector by passing it a HttpCollectorConfig object.

You can build the configuration object entirely in java, or by loading an existing XML configuration file using com.norconex.commons.lang.config.ConfigurationLoader.

XML-based configuration

Path configFile = Paths.get("/path/to/your/xml/config/file");
HttpCollectorConfig collectorConfig = new HttpCollectorConfig();
new ConfigurationLoader().loadFromXML(configFile, collectorConfig);

Java-based configuration

HttpCollectorConfig collectorConfig = new HttpCollectorConfig();
collectorConfig.setId("MyHttpCollector");
...
HttpCrawlerConfig crawlerConfig = new HttpCrawlerConfig();
crawlerConfig.setId("MyHttpCrawler");
crawlerConfig.setStartURLs("http://example1.com", "http://example2.com");
...
collectorConfig.setCrawlerConfigs(crawlerConfig);

Start the collector

HttpCollector collector = new HttpCollector(collectorConfig);
collector.start();

Refer to the Web Crawler Javadoc for more documentation or the Configuration page for XML configuration options.

Extend the Web Crawler

You can create your own feature or modify existing ones. Create a new Java project in your favorite IDE. If you are not using Maven, add to your classpath all the files contained in the lib folder of the Web Crawler installation. You can make command-line testing easier by configuring your project to have its binary output directory to be the classes folder of the installation directory. Automatically, code created and stored under classes will be picked up by the Web Crawler when you run it.