com.norconex.collector.http

Class HttpCollector

java.lang.Object

com.norconex.collector.core.AbstractCollector

com.norconex.collector.http.HttpCollector

All Implemented Interfaces:

ICollector, IJobSuiteFactory

public class HttpCollector
extends AbstractCollector

Main application class. In order to use it properly, you must first configure it, either by providing a populated instance of HttpCollectorConfig, or by XML configuration, loaded using CollectorConfigLoader. Instances of this class can hold several crawler, running at once. This is convenient when there are configuration setting to be shared amongst crawlers. When you have many crawler jobs defined that have nothing in common, it may be best to configure and run them separately, to facilitate troubleshooting. There is no set rules for this, experimenting with your target sites will help you.

Author:

Pascal Essiembre

Constructor Summary

Constructors

Constructor and Description
HttpCollector() Creates a non-configured HTTP collector.
HttpCollector(HttpCollectorConfig collectorConfig) Creates and configure an HTTP Collector with the provided configuration.

Method Summary

Modifier and Type	Method and Description
protected ICrawler	createCrawler(ICrawlerConfig config)
HttpCollectorConfig	getCollectorConfig()
static void	main(String[] args) Invokes the HTTP Collector from the command line.

Methods inherited from class com.norconex.collector.core.AbstractCollector

createJobSuite, getCrawlers, getId, getJobSuite, getState, setCrawlers, start, stop

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HttpCollector

public HttpCollector()

Creates a non-configured HTTP collector.

HttpCollector

public HttpCollector(HttpCollectorConfig collectorConfig)

Creates and configure an HTTP Collector with the provided configuration.

Parameters:

collectorConfig - HTTP Collector configuration

Method Detail

main

public static void main(String[] args)

Invokes the HTTP Collector from the command line.

Parameters:

args - Invoke it once without any arguments to get a list of command-line options.

getCollectorConfig

public HttpCollectorConfig getCollectorConfig()

Specified by:

getCollectorConfig in interface ICollector

Overrides:

getCollectorConfig in class AbstractCollector

createCrawler

protected ICrawler createCrawler(ICrawlerConfig config)

Specified by:

createCrawler in class AbstractCollector