2.x Release Notes

2.9.1 Maintenance release Release date 2021-10-18 Download

New	New "validExitCodes" on PhantomJSDocumentFetcher to support other than 0 (now default).	#729
New	StandardSiteMapResolver can now optionally filter entries that are older than a given date.	#718
New	StandardSiteMapResolver can now optionally throw exception instead of logging errors.	#718
Updated	Robots.txt redirects are now being followed.	#726
Updated	Extracted links are no longer extracted/kept by default for documents having reached the max depth. To keep former behavior, use the new method HttpCrawlerConfig#keepMaxDepthLinks(...).	#498
Updated	Generic canonical link detector now supports non-standard canonical URL definition using meta "property" instead of "rel".	#697
Updated	More permissive gzip detection for sitemap.xml files.	#687
Updated	Can now decide to re-process headers as part of document fetching when metadata fetcher fails.	#655
Updated	Redirects are now ignored when part of a bad HTTP response when "skipOnBadStatus" is "true".	#695
Updated	Metadata fetcher exceptions are swallowed and logged when "skipOnBadStatus" is "true".	#696
Updated	Unsupported content encoding form Content-Type HTTP headers now triggers a detection attempt of both encoding and content type based on document content.
Updated	Norconex JEF 4.1.3, Norconex Commons Lang 1.15.2, Norconex Collector Core 1.10.1.
Fixed	Fixed sitemap.xml URL entries not being extracted when "loc" is from a sitemap extension namespace (e.g., "image:loc").	#761
Fixed	Fixed sitemap.xml URL entries not being extracted when they also contain images.	#758
Fixed	Fixed RecrawlableResolver settings not always being respected.	#741
Fixed	Fixed redirect targets sometimes being processed again when already processed normally (not through a redirect).	#741
Fixed	Fixed NullPointerException when resolving sitemaps.	#738
Fixed	Fixed authFormParams XML configuration not being loaded properly.	#730
Fixed	Fixed occasional FileNotFoundException with PhantomJS when the downloaded file was not yet written even if PhantomJS has already returned.
Fixed	Fixed REJECTED_REDIRECTED event not being fired for the redirect source URL when the target URL was previously encountered.	#678
Fixed	Unsupported content encoding form Content-Type HTTP headers no longer rejects a document.

2.9.0 Feature release Release date 2019-12-22 Download

New	From Collector Core update, added "unmanaged" attribute to "logsDir" configuration option to prevent the collector from managing its own file-based logging.	#593
New	Added new normalization rules to GenericURLNormalizer: removeQueryString, lowerCase, lowerCasePath, lowerCaseQuery, lowerCaseQueryParameterNames, lowerCaseQueryParameterValues.	#479
New	Added "includeSubdomains" to start URL group to consider subdomains as being the same as any URL domain when "stayOnDomain" is true.	#563
New	GenericLinkExtractor now supports using "selector-syntax" to identify portions of an HTML document to include or exclude from URL extraction.
Updated	Switch default cookie specs from DEFAULT to STANDARD and can now be configured.	#525
Updated	Dependency updates: Norconex Collector Core 1.10.0, Norconex Commons Lang 1.15.1.
Updated	Stateful support added for auth sessions (e.g., NTLM).	#526
Fixed	Fixed GenericCanonicalLinkDetector not handling properly HTTP Response "Link" Header with multiple values.	#646
Fixed	Fixed GenericRecrawlableResolver large frequency units resulting in those minimum frequencies not being respected.	#597
Fixed	Fix quoted mediatype from content type.	#578
Fixed	Fixed exceptions in URLCrawlScopeStrategy#isInScope(...) ending the crawler execution.	#540
Fixed	Discovered canonical URLs are now rejected if they are out of scope (not matching the stayOnXXX directives).	#527
Fixed	Fixed good redirected URLs being committed twice in same crawl session.	#533
Fixed	TikaLinkExtractor now extracts URL with spaces. It should also support other URLs typically considered "bad".	#628

2.8.1 Maintenance release Release date 2018-08-17 Download

Updated	Dependency updates: Norconex Collector Core 1.9.1, Norconex Commons Lang 1.15.0.
Updated	GenericDocumentFetcher now logs documents it could not fetch as INFO.	#475
Fixed	Fixed XMLFeedLinkExtractor sometimes splitting URLs.	#487
Fixed	Fixed invalid validation error when XML configuration for PhantomJSDocumentFetcher does not contain "screenshotStorageDiskDir".	#473
Fixed	Fixed NullPointerException when using PhantomJSDocumentFetcher without specifying any "screenshotDimensions".
Fixed	Added validation support for new "authFormParams" option in GenericHttpClientFactory.	#434
Fixed	Fixed PhantomJSDocumentFetcher sending invalid scheme to PhantomJS proxy (when used with HttpClientProxyCollectorListener).	#437

2.8.0 Feature release Release date 2017-11-26 Download

New	New FeaturedImageProcessor class to extract the "main" image from a web page.
New	New screenshot storage options for PhantomJSDocumentFetcher, like specifying image format, store inline (base64), and exact size.
New	New "authFormParams" option on GenericHttpClientFactory for adding arbitrary parameter to authentication forms.
New	New metadata field "collector.redirect-trail" for storing all redirect source URLs with the target URL metadata.	#397
New	New "authPreemptive" configuration option on GenericHttpClientFactory to use preemptive authentication with BASIC authentication.	#420
New	GenericLinkExtractor can now limit link extraction to specified portion of text only.
New	It is now possible to store as metadata URLs that are not "in-scope" with a new flag "keepOutOfScopeLinks".	#360
New	New GenericHttpClientFactory#removeRequestHeader(String) method.
New	New "resourceTimeout" option on PhantomJSDocumentFetcher to prevent individual page items to hang PhantomJS for a long time.	#383
Updated	Now handles circular references between a non-canonical page pointing to a URL redirecting back to that non-canonical page. When this loop is encountered, it will now process the non-canonical page (used to reject both).	#162
Updated	GenericLinkExtractor now extracts unquoted URLs.	#423
Updated	*.http.doc.IHttpDocumentProcessor now deprecated in favor of *.http.processor.IHttpDocumentProcessor.
Updated	Crawler event REJECTED_CANONICAL now deprecated in favor of REJECTED_NONCANONICAL.
Updated	RegexLinkExtractor now supports replacement values for regex matches.
Updated	GenericURLNormalizer XML configuration can now accepts empty normalizations tag to blank default normalizations.
Updated	Dependency updates: Norconex Collector Core 1.9.0, Norconex Commons Lang 1.14.0.
Updated	PhantomJSDocumentFetcher will now try to write a document for processing even if its status from PhantomJS is not "success" (for cases where valid content could be extracted nonetheless).	#383
Updated	Blank start URLs are now silently ignored.
Fixed	Fixed blank URLs extracted by RegexLinkExtractorin throwing NullPointerException.	#422
Fixed	Fixed blank lines in start URL seed files throwing NullPointerException.	#396
Fixed	Orphans being reprocessed no longer triggers the "RecrawlableResolverStage".	#390
Fixed	Fixed improper saving as XML of GenericDelayResolver.
Fixed	Fixed GenericDelayResolver not taking into account ranges that overlap into the next cycle (e.g. 22:00 to 5:00).
Fixed	Fixed setting proxy realm having no effect on GenericHttpClientFactory.
Fixed	Fixed invalid validation error for "tempDir" element on StandardSitemapResolverFactory.
Fixed	Fixed PhantomJSDocumentFetcher sometimes throwing a NullPointerException when content-type could not be established.	#408
Removed	Removed these deprecated classes: HtmlLinkExtractor, HttpMetadataChecksummer, and deprecated methods: HttpCrawler#getUrlsFiles, HttpCrawler#setUrlsFiles.

2.7.1 Bugfix release Release date 2017-05-26 Download

Updated	Dependency updates: Norconex Collector Core 1.8.2.
Fixed	Fixed "caseSensitive" flag sometimes having no effect in GenericRecrawlableResolver.

2.7.0 Feature release Release date 2017-04-26 Download

New	New PhantomJSDocumentFetcher which relies on an external installation of PhantomJS to fetch documents. This provides a way to crawl web pages with JavaScript-generated content.	#95
New	New XMLFeedLinkExtractor to extract links out of RSS or Atom feeds.	#319
New	New RegexLinkExtractor to extract links out of any text file using regular expressions.	#236
New	Added schema-based XML configuration validation which can be trigged on command prompt with this new flag: -k or --checkcfg
New	Can now provide start URLs dynamically with new IStartURLsProvider.	#331
New	Added "removeTrailingHash" to GenericURLNormalizer.	#331
New	New "detectContentType" and "detectCharset" options on GenericDocumentFetcher for ignoring the content type and character encoding obtained from the HTTP response headers and detect them instead.
New	New features from dependency updates. Collector Core: ICollectorLifeCycleListener. Importer: MergeTagger, ExternalTransformer.
New	MongoCrawlDataStoreFactory now accepts encrypted passwords.
New	Now distributed with utility scripts.
Updated	XML configuration entries expecting millisecond durations can now be provided in human-readable format (e.g., "5 minutes and 30 seconds" or "5m30s").
Updated	Dependency updates: Norconex Collection Core 1.8.0, Norconex Commons Lang 1.13.0, Jetty 9.2.20.
Updated	Crawler events REJECTED_FILTER, REJECTED_BAD_STATUS, REJECTED_IMPORT, REJECTED_ERROR, REJECTED_ROBOTS_TXT, and REJECTED_CANONICAL, REJECTED_REDIRECTED are now DEBUG in log4j.properties.
Updated	Modified Javadoc to include an XML usage example for all XML-configurable classes.
Fixed	Fixed OverlappingFileLockException when loading the a sitemap store more than once per JVM instance.	#336
Fixed	Fixed NullPointerException from GenericLinkExtractor when encountering HTML files with an empty href in their base tag.
Fixed	Fixed minor errors in writing IXMLConfigurable classes to XML.
Removed	Removed MapDB and Apache Derby crawlstore dependencies/implementations which were deprecated in previous versions.

2.6.2 Maintenance release Release date 2017-01-06 Download

New	New "sslProtocols" configuration option on GenericHttpClientFactory to specify which SSL/TLS protocols are supported (e.g., SSLv3, TLSv1, TLSv1.1, TLSv1.2). Useful to enable TLSv1.2 on Java 7 when encountering SSL handshake problems on site supporting TLSv1.2 only.
Fixed	Corrected documentation in collector-http-config-reference.xml to indicate default orphan strategy is now PROCESS.

2.6.1 Maintenance release Release date 2016-12-14 Download

New	New "addDomainTrailingSlash" normalization rule for GenericURLNormalizer.
Updated	Dependency updates: Norconex Commons Lang 1.12.3, JJ2000 5.3, Norconex Collection Core 1.7.0.
Fixed	Fixed canonical URLs defined in content with single quote in them being truncated.	#111
Fixed	Fixed StandardRobotsTxtProivder wrongfully rejecting documents when encountering a blank "Disallow:" directive.
Fixed	Fixed GenericLinkExtractor wrongfully resolving relative URLs when an HTML page had a "base href" being a relative page.

2.6.0 Feature release Release date 2016-08-25 Download

New	New normalization rule for GenericURLNormalizer: removeTrailingSlash.	#290
New	Can now specify "notFoundStatusCodes" on GenericMetadataFetcher.
Updated	Specifying an empty "path" tag in XML config or, setting a null or empty string array on StandardSitemapResolverFactory#setSitemapPaths(...) method will now prevent trying to locate sitemaps using default locations and will strictly rely on sitemap URLs specified as start URLs or found in robots.txt (if enabled).	#292
Updated	GenericLinkExtractor no longer extracts URL from HTML/XML comments by default. To re-enable this behavior a new "setCommentsEnabled(boolean)" method has been added.	#291
Updated	Normalization rule "addTrailingSlash" in GenericURLNormalizer has been renamed "addDirectoryTrailingSlash".	#290
Updated	SitemapStore now uses MVStore instead of MapDB for storing the list of processed sitemaps.
Updated	Referrer data is now always stored for GenericLinkExtractor (default) and TikaLinkExtractor.
Updated	Redirects encountered by IHttpMetadataFetcher implementations are now followed by default.	#281
Updated	Dependency updates: Norconex Collector Core 1.6.0.
Updated	Renamed constant HttpMetadata.COLLECTOR_REFERNCED_URLS to HttpMetadata.COLLECTOR_REFERENCED_URLS (was misspelled).
Updated	API break: method signature changed for IHttpMetadataFetcher from Properties fetchHTTPHeaders(HttpClient httpClient, String url) to HttpFetchResponse fetchHTTPHeaders( HttpClient httpClient, String url, Properties headers)
Updated	The subject for the crawler event HttpCrawlerEvent.DOCUMENT_METADATA_FETCHED is now an instance of HttpFetchResponse.
Updated	Extracted canonical URLs now have their referrer reference stored with them.
Updated	Extracted URLs are now stored in crawl store.
Updated	Now using MVStore for sitemap store (instead of MapDB).
Fixed	Fixed GenericRecrawlableResolver#MinFrequency() constructor with arguments not setting patterns correctly.	#289
Fixed	Fixed documents wrongfully being considered orphans when referrer was skipped for being unmodified or premature, or was in temporary error, before its URLs could be extracted. This could cause valid documents to be deleted/ignored (depending on orphan strategy used). Now those "child" URLs will be queued for processing as if they were extracted from referrer page.	#278
Fixed	Canonical URLs extracted are now normalized before being compared to their containing page URL, which were already normalized.

2.5.1 Maintenance release Release date 2016-07-27 Download

Updated	Updated references to test pages URLs to now be "https" in "minimum" and "complex" example configurations.	#252
Updated	Dependency updates: Joda Time 2.9.4, Apache HttpClient 4.5.2, JSoup 1.8.3.

2.5.0 Feature release Release date 2016-06-03 Download

New	Now supports sitemap "lastmod" and "changefreq", and also allows for specifying custom minimum recrawl frequency for documents. Default implementation is GenericRecrawlableResolver (implements new IRecrawlableResolver).
New	New ReferenceDelayResolver that introduces delays between downloads based on matching document reference (URL) patterns.	#246
Updated	Document "crawl date" and content type can be added to the crawl data and will be stored in the crawl data store (affects all ICrawlDataStoreFactory implementations).
Updated	MVStore is now the default URL crawl store.
Updated	Dependency updates: Norconex Collector Core 1.5.0.
Updated	StandardSitemapResolver now parses sitemaps with content type application/gzip (in addition to application/x-gzip).
Fixed	Fixed "stayOnDomain" being true not being honored for extracted URLs with no scheme (//www.example.com).	#243
Fixed	Fixed AbstractDelay wrongfully computing seconds when logging debug statement.

2.4.0 Feature release Release date 2016-02-28 Download

New	New configurable IRedirectURLProvider with default implementation GenericRedirectURLProvider now better handling redirect character encoding and offering encoding options.	#199
New	HTTP authentication password and proxy password can now be encrypted when using GenericHttpClientFactory.	#178
Updated	The "log4j.properties" file has been moved from classes to the installation root directory.
Updated	GenericLinkExtractor now only supports these URI schemes: http, https, ftp. It is possible to overwrite these default with #setSchemes(String[]).	#212
Updated	GenericLinkExtractor now attempts by default to detect the character encoding of documents before extracting links and referrer information. In addition, it is now possible to explicitly set which charset to use via the #setCharset(String) method.	#194
Updated	Derby unit tests are now much faster after settings its system durability for testing to "test".
Updated	Added more logging to FORM-based authentication to help troubleshoot authentication issues.
Updated	Dependency updates: Apache HttpClient 4.5.1, Norconex Collector Core 1.4.0, Joda Time 2.9.2.
Updated	Robot rules now escapes dots and other character having special meaning in regex, before converting robot rules to regex.	#200
Updated	GenericLinkExtractor now only strips the inside of script tags in HTML pages before extracting links, so that the src links can be followed if configured to do so.	#232
Fixed	GenericLinkExtractor is now reading characters as opposed to bytes to fix breaking some characters.	#219
Fixed	Robots.txt "Allow" directive is now supported properly as defined by Google bot usage.	#213
Fixed	URLStatusCrawlerEventListener no longer prefixes file names with "null" when no file name prefix is explicitly set.	#206
Fixed	Fixed both GenericLinkExtractor and TikaLinkExtractor which were not able to extract link titles when keepReferrerData was true.	#204
Fixed	Text is now extracted when HTML is found in an anchor body (when keepReferrerData is true).	#204
Fixed	GenericHttpClientFactory#trustAllSSLCertificates now disables hostname verification and algorithm constraints.	#196
Fixed	Fixed "lenient" flag having no effect on "sitemapResolverFactory".	#193
Fixed	Fixed "base href" HTML tag not being recognized when resolving relative URLs in HTML pages.	#188
Fixed	When the "class" attribute is not specified in "sitemapResolverFactory" for XML config, the default resolver instance is now used (as opposed to null, which was causing some configs not to be loaded properly).	#187
Fixed	GenericHttpClientFactory#trustAllSSLCertificates will now disable SSL SNI Extension, and unable unsafe cert algorithms that are now disabled by default since Java 7.	#181

2.3.0 Feature release Release date 2015-11-06 Download

New	GenericHttpClientFactory now allows you to set HTTP request headers on every HTTP calls a crawler will make.	#175
New	New crawler configuration options: stayOnProtocol, stayOnDomain, and stayOnPort. These new settings can be applied as attributes to the startURLs tag in an XML configuration, or on the object returned by HttpCrawlerConfig#getURLCrawlScopeStrategy(). This addition affects or replace previous implementations discussed in #138 , #135 , #131 , #17 , and possibly others.	#144
New	It is now possible to specify one or more sitemap URLs as "start URLs". New HttpCrawlerConfig#[set|get]StartSitemapURLs(...) methods.	#128
New	GenericURLNormalizer now has the few normalizations by default, as described in its Javadoc.
New	New StandardSitemapFactory#setPaths(...) method to specify where to look for sitemap files for each URLs processed (relative to URL root).
Updated	The "sitemap" tag used to set the ISitemapResolverFactory implementation has been renamed to "sitemapResolverFactory" to avoid confusion with the "sitemap" tag that can now be set as a start URL.
Updated	URL normalization now always takes place by default, using GenericURLNormalizer. Can be turned off by either setting it to null in the crawler configuration, or invoking GenericURLNormalizer#setDisabled(true).	#145
Updated	URLs extracted from a document are now stored in "collector.referenced-urls" after they have been normalized.	#145
Updated	URL redirects are now logged as REJECTED_REDIRECTED (log level INFO).
Updated	HtmlLinkExtractor has been deprecated in favor of GenericLinkExtractor.
Updated	HttpCrawlerConfig#[set|get]UrlsFiles(...) has been deprecated in favor of HttpCrawlerConfig#[set|get]StartUrlsFiles(...)
Updated	StandardSitemapFactory#setLocations(...) now deprecated in favor of being able to specify sitemaps as start URLs.
Updated	ISitemapResolver#resolveSitemaps(...) as a new argument to specify whether the sitemap locations provided were defined as "start URLs" or not.
Updated	Now logs User-Agent upon startup (log level INFO).
Updated	Dependency updates: Norconex Collector Core 1.3.0, Norconex Commons Lang 1.8.0.
Updated	Added new event types loggers to log4j.properties.
Updated	Corrected typos and improved documentation.
Updated	Saved and loaded configuration-related classes are now equal. Methods equals/hashCode/toString for those classes are now implemented uniformly and where added where missing.
Fixed	Fixed some configuration classes not always being saved to XML properly or giving errors.
Fixed	Relative redirect URLs are now converted to absolute.	#169
Fixed	Fixed robots.txt being fetched before reference filters were executed. Robots.txt are no longer fetched for rejected reference.	#168
Fixed	GenericLinkExtractor now unescapes HTML entities in URLs.
Fixed	Fixed ClassCastException in HttpCrawlerRedirectStrategy when using an HTTP Proxy.	#167
Fixed	GenericCanonicalLinkDetector now supports links that are escaped (HTML-entities). They are now unescaped before they are processed.	#164
Fixed	Fixed circular redirect exception.	#146
Fixed	GenericLinkExtractor and TikaLinkExtractor now extract meta http-equiv refresh properly when "refresh" is without quotes or not lowercase.	#141
Fixed	Fixed duplicate commits when multiple URL redirects are pointing to the same target URL.	#135
Fixed	Fixed possible URISyntaxException in GenericURLNormalizer (fixed by updated Norconex Commons Lang URLNormalizer dependency).	#132

2.2.1 Bug fix release Release date 2015-08-07 Download

Updated	Dependency updates: Norconex Collector Core 1.2.1.
Fixed	Fixed NullPointerException occurring inconsistently when using "thread" scope in GenericDelayResolver.
Fixed	GenericDocumentFetcher now handles more special characters in URLs. Also saves itself as XML properly now. Created new unit tests for it.
Fixed	StandardRobotsTxtProvider now handles empty "Disallow: " properly."	#129
Fixed	Fixed NullPointerException when configuring an extractor tag in XML without specifying the class attribute (now defaults to HtmlLinkExtractor).

2.2.0 Feature release Release date 2015-07-22 Download

New	Added support for canonical links defined in both HTTP Headers or as a link tag in an HTML document head tag. Canonical links detection is always performed unless explicitly disabled.	#79
New	New URLStatusCrawlerEventListener class for producing reports of fetched URLs and their status. Useful for finding broken links or else.
New	Added three new configuration options to GenericHttpClientFactory to better deal with HTTP connectivity issues (like timeouts): "maxConnectionsPerRoute", "maxConnectionIdleTime", and "maxConnectionInactiveTime".	#118
New	New LastModifiedMetadataChecksummer that uses Last-Modified HTTP header value for checksum purposes, replacing HttpMetadataChecksummer as the default implementation. For choosing one or more fields of your choice to create a checksum, you can now use the new GenericMetadataChecksummer from the Collector Core dependency.
New	New CurrentDateTagger, DateMetadataFilter, NumericMetadataFilter, TextPatternTagger, GenericSpoiledReferenceStrategizer and more new features introduced by dependency upgrades.
New	New method GenericDocumentFetcher#setNotFoundStatusCodes(int...) to specify one or several custom "Not Found" HTTP codes. Default is 404.
Updated	GenericHttpClientFactory default maximum connection was increased from 20 to 200 and default maximum connections per route was increased from 2 to 20.	#118
Updated	New HttpFetchResponse class now passed to crawl event listeners after a document fetch instead of the IHttpDocumentFetcher used. This adds the ability to listen for specific HTTP response status code. As a consequence, IHttpDocumentFetcher now returns a HttpFetchResponse.
Updated	HttpMetadataChecksummer has been deprecated in favor of LastModifiedMetadataChecksummer.
Updated	HtmlLinkExtractor now supports specifying tags without an attribute for detecting URLs.
Updated	HtmlLinkExtractor now ignores whatever is found between "script" tags so that JavaScript-generated URLs can no longer cause trouble.	#119
Updated	Dependency updates: Norconex Collector Core 1.2.0, Joda Time 2.8.1, Apache HTTP Client 4.5, Jetty Webapp 9.2.11.v20150529, Apache Ant 1.9.5.
Updated	Jar manifest now includes implementation entries and specifications entries (matching Maven pom.xml).
Updated	Improved/fixed javadoc.
Updated	HttpCrawlState#NOT_FOUND was migrated to Norconex Collector Core CrawlState#NOT_FOUND.
Fixed	Fixed HTML documents being skipped when HtmlLinkExtractor found a URL of invalid format. Now a warning is thrown for each bad URLs instead and the document is processed anyway, and good URLs are extracted.	#119
Fixed	Fixed MongoDB stage/depth compound index.	#97
Fixed	Fixed MongoCrawlDataSerializer "referrerLinkText" link attribute/metadata having the same value as "referrerLinkTag".	#82
Fixed	HtmlLinkExtractor now decodes encoded ampersands present in URLs.	#88
Fixed	Both HtmlLinkExtractor and TikaLinkExtractor no longer extract empty href links. Added this use case to corresponding unit test.	#87
Fixed	HtmlLinkExtractor now strips leading spaces.
Fixed	Fixed "trustAllSSLCertificates" configuration option on GenericHttpClientFactory not being recognized in XML config.	#100
Fixed	Fixed exceptions thrown in StandardRobotsTxtProvider when robots.txt contained rules ending with ? or when the referrer URL was starting with a space.
Fixed	Several invalid characters are now supported in URLs (e.g., commas).
Fixed	Fixed GenericDelayResolver not saving in XML properly and fixed its javadoc as well, which did not mention how to pass the delay in XML config.
Fixed	TargetURLRedirectStrategy no longer throws an exception when redirects are disabled for a request.	#124
Fixed	Fixed "sitemapLocations" and "lenient" not being applied to StandardSitemapFactory.	#120

2.1.0 Feature release Release date 2015-04-08 Download

New	Several new features, updates and fixes were added by upgrading Norconex Collector Core (https://opensource.norconex.com/collectors/core/) and Norconex Importer (https://opensource.norconex.com/importer/) dependencies. Those include support for ORC, translation, a title generator, new content type parsing, and more. Refer to dependency release notes for more details.
New	New methods and configuration attribute to disable checksum creation in HttpMetadataChecksummer.
Updated	Sitemap resolving pipeline stage is now always invoked (but won't do anything if disabled).
Updated	Dependency updates: Norconex Collector Core 1.1.0, JUnit 4.12, Joda-Time 2.7, Apache HTTP Components 4.4, Jetty Webapp 9.2.10.v20150310, Fongo 1.6.2.
Updated	Added Sonatype repository to pom.xml for snapshot releases.
Updated	Updated several maven plugins and added SonarQube maven plugin.
Updated	Improvements on character encoding detection from HTTP headers.
Updated	log4j.properties from ./classes/ now properly loaded by collector-http.sh.	#59
Updated	Improved javadoc.
Updated	Added many unit tests for testing start vs resume vs stop vs deleted vs modified, in different JVM instances.
Updated	"minimum" and "complex" configuration examples now ignore sitemap.xml files.
Fixed	Fixed link extractor not fetching link text properly when keepReferrerData is true on HtmlLinkExtractor.	#56
Fixed	Robot meta data found in HTML pages will no longer be extracted if found within an HTML comment. Robot metadata detection is also more robust.	#60
Fixed	Fixed NPE in HttpImporterPipelineUtil#enhanceHTTPHeaders when content type from HTTP header is not defined.
Fixed	Fixed log4j log levels incorrectly ending with a semi-colon.

2.0.2 Bug fix release Release date 2015-02-04 Download

Fixed	Fixed the collector "stop" action having no effect.	#49
Fixed	Fixed crawl data wrongfully applied as metadata after the import phase.
Fixed	Fixed NullPointerException when sitemap support is disabled.
Fixed	Fixed incorrect deletion behavior for embedded orphan documents.
Updated	Improved log4j.properties logging options for crawler events.
Updated	Upgraded Norconex Collector Core dependency to 1.0.2.

2.0.1 Bug fix release Release date 2014-12-03 Download

Fixed	From collector-core-1.0.1: When keepDownloads is true, saved files and directories are now prefixed with "f." and "d." respectively to avoid collisions.	#44
Fixed	Fixed errors in example configuration files.

2.0.0 Major release. Release date 2014-11-26 Download

New	Upgraded Norconex Importer to version 2.0.0, which brings to Norconex HTTP Collector a lot of new features, such as: Document content splitting, splitting of embedded documents into individual documents, new taggers for language detection, changing character case, parsing and formatting dates, providing content statistics, and more.
New	Can now supplied a "pathsFile" as part of the startPaths, acting as a seed list.
New	New fast MVStore database implementation for URL database (from Norconex Collector Core).
New	New H2 database implementation for URL database (crawl data store).
New	Now keeps track of parent references (for embedded/split documents).
New	More unit tests, with the addition of an embedded Jetty Web server rendering test pages for some unit tests.
New	New JMX/MBean support added on crawlers.
New	IUrlExtractor is now ILinkExtractor and both their implementing classes (HtmlLinkExtractor and TikaLinkExtractor) now support also extracting a link title and text, and they also support the "nofollow" robot rule.	#23
New	It is now possible to configure multiple link extraction classes, each taking effect on particular URLs and/or content-types.
New	IHtmlLinkExtractor can be configured to use specified HTML tags and attributes to find URLs.
Updated	Now licensed under The Apache License, Version 2.0.
Updated	Replaced the configuration option "deleteOrphans(true|false)" with "orphansStrategy(DELETE|PROCESS|IGNORE)".
Updated	The collector now references document content as reusable InputStream with memory caching instead of relying only on files. This saves a great deal of disk I/O and improves performance in most cases.
Updated	Refactored to use the new Norconex Collector Core library. A significant portion of the Norconex HTTP Collector code has been moved to that core library. Some of the moved classes are (base package being com.norconex.collector.*): http.checksum.IHttpDocumentChecksummer to core.checksum.IDocumentChecksummer, http.checksum.IHttpHeadersChecksummer to core.checksum.IMetadataChecksummer, http.checksum.DefaultHttpDocumentChecksummer to core.checksum.impl.MD5DocumentChecksummer, http.filter.IURLFilter to core.filter.IReferenceFilter http.filter.IHttpHeadersFilter to core.filter.IMetadataFilter http.filter.IHttpDocumentFilter to core.filter.IDocumentFilter http.filter.impl.ExtensionURLFilter to core.filter.impl.ExtensionURLFilter http.filter.impl.RegexHeaderFilter to core.filter.impl.RegexMetadataFilter http.filter.impl.RegexURLFilter to core.filter.impl.RegexReferenceFilter
Updated	Amongst others, the following classes were renamed (within com.norconex.collector.http.*): checksum.impl.DefaultHttpHeadersChecksummer to checksum.impl.HttpMetadataChecksummer, client.impl.DefaultHttpClientFactory to client.impl.GenericHttpClientFactory, delay.impl.DefaultDelayResolver to delay.impl.GenericDelayResolver, fetch.impl.DefaultDocumentFetcher to fetch.impl.GenericDocumentFetcher, fetch.impl.SimpleHttpHeadersFetcher to fetch.impl.GenericHttpHeadersFetcher, robot.impl.DefaultRobotsMetaProvider to robot.impl.StandardRobotsMetaProvider, robot.impl.DefaultRobotsTxtProvider to robot.impl.StandardRobotsTxtProvider, sitemap.impl.DefaultSitemapResolver to sitemap.impl.StandardSitemapResolver, url.impl.DefaultURLExtractor to url.impl.GenericURLExtractor
Updated	Several references to "url" were changed to "reference".
Updated	New and more scalable crawler event model along with new listeners.
Updated	Refactored to use JEF 4.0.0 which makes the HTTP Collector easier to monitor.
Updated	Other libray upgrades: Norconex Committer to 2.0.0 and Norconex Commons Lang to 1.5.0.
Updated	Removed previously deprecated classes.
Updated	Crawled sitemap details now has its own store (no longer mixed with the crawl data store).
Updated	ISiteMapResolver now needs an ISiteMapResolverFactory.
Fixed	Sitemap resolution now stops when a stop request is issued.	#38
Fixed	Now checks if crawler is running before attempting to stop it.	#37

1.3.4 Maintenance release. Release date 2014-08-24 Download

New	MongoCrawlURLDatabase now supports user authentication.
Updated	Now requires Java 7 or higher.
Fixed	Fixed DefaultRobotsTxtProvider failing to parse some robots.txt patterns.	#36
Fixed	Fixed DefaultRobotsTxtProvider failing to parse some robots.txt patterns.	#36

1.3.3 Bug fix release. Release date 2014-08-07 Download

Updated	Upgraded JEF to 3.0.1 to fix stop action not working.
Fixed	Fixed NullPointerException in robots.txt resolution under some circonstances.

1.3.2 Bug fix release. Release date 2014-06-17 Download

Fixed	DefaultURLExtractor no longer treat empty href as being a URL ending with a double-quote.
Updated	Renamed HttpMetadata key "collector.http.dept" to "collector.http.depth" (typo fix).
Updated	Upgraded Norconex Commons Lang to 1.3.2
Updated	GenericURLNormallizer no longer rejects URLs with spaces in them. It now logs a warning instead (thanks to Norconex Commons Lang upgrade).

1.3.1 Bug fix release. Release date 2014-04-10 Download

Fixed

Header and document checksum value are no longer added by default to prevent the issue described in github ticket #24 . Instead, adding checksum is now an optional feature of DefaultHttpDocumentChecksummer and DefaultHttpHeadersChecksummer.

#24

1.3.0 Feature release. Release date 2014-03-24 Download

New	Now supports NTLM authentication. SPNEGO and Kerberos were also added but are experimental (see DefaultHttpClientFactory).
New	Can now specify character set of HTTP connections and authentication forms.
New	Can now set custom timeout values on HTTP connection-related activities.
New	New option to trust all SSL certificates of sites being crawled (see DefaultHttpClientFactory).
New	Can now specify a maximum number of HTTP connections for each crawler independently of configured number of threads (see DefaultHttpClientFactory).
New	DefaultHttpClientFactory introduces additional configuration options: proxy scheme, 'Expect: 100-continue' handshake, maximum HTTP redirects, local address, stale connection checks
New	HTTP header checksum and document checksum are now added to the document metadata as HttpMetadata#CHECKSUM_HEADER and HttpMetadata#CHECKSUM_DOC.
Updated	The empty sub-folders contained under the "download" folder are now periodically deleted. This speeds up directory scanning and increases performance on large crawls.
Updated	The userAgent is now a crawler configuration option (previously was an option of DefaultHttpClientInitializer )
Updated	API change: IRobotsTxtProvider#getRobotsTxt(...) method signature has changed to accept the User-Agent.
Updated	IHttpClientInitializer is now deprecated in favor of IHttpClientFactory, giving you more control over HttpClient creation.
Updated	API change: Methods previously accepting DefaultHttpClient instances now have their signature accepting parent interface HttpClient instead.
Updated	More logging to help resolve crawler issues with DEBUG log level.
Updated	HttpCrawler more lenient upon encountering some errors that were previous aborting entire execution.
Updated	Library upgrades. Updated default crawl url database (MapDB) to version 0.9.10, Norconex Commons Lang to 1.3.0, Norconex Committer to 1.2.0, Norconex Importer to 1.2.0, and Apache HttpClient to 4.3.2.
Fixed	Now ensures that robots.txt agent matching gives priority to the most specific match (as opposed to the first match). Sitemaps detected in robots.txt are also preserved for sitemap resolving.
Removed	Removed classes deprecated since 1.1.

1.2.0 Feature release. Release date 2014-01-11 Download

New	New optional Mongo URL Database implementation.
New	New TikaURLExtractor class providing an alternate IURLExtractor implementation based on Apache Tika HTMLParser.
New	New SegmentCountURLFilter class for filtering URLs having a specified number of segments (can check duplicate segments too).
New	New unit tests.
Updated	MapDB URL Database classes moved to its own "mapdb" package. DefaultCrawlURLDatabaseFactory still exists, but is just a pointer to MapDBCrawlURLDatabaseFactory.
Updated	Example configurations now point to Norconex test pages to ensure their stability.
Updated	Upgraded dependent libraries: Norconex Committer 1.1.0, Norconex Commons Lang 1.2.0, MapDB 0.9.8 and other thrid party libraries.
Updated	Improved Javadoc.

1.1.1 Bug fix release. Release date 2013-10-03 Download

Fixed	Fixed not being able to extract the "href" attribute when it starts a new line.	#18
Fixed	Fixed HTTP redirects not storing final target URL but the source URL instead.	#17
Updated	Upgraded dependent libraries to Norconex Importer 1.1.0 and Norconex Commons Lang 1.1.0.

1.1.0 Feature release. Release date 2013-08-21 Download

New	Crawler now fires additional events. Added "documentRobotsMetaRejected" and "documentImportRejected" methods to IHttpCrawlerEventListener.
New	DefaultCrawlURLDatabase now uses a MapDB-based implementation for faster performance. The derby implementation has been kept for those with a preference for it.
New	Now support sitemap.xml and sitemap index (plain or gzip). URLs from sitemaps will have the sitemap information as metadata.
New	BASIC and DIGEST authentication now supported.
New	Now supports in-page robot instructions. Via "ROBOTS" meta tag first, or "X-Robots-Tag" tag if present in HTTP header.
New	"ftp" protocol now supported.
New	It is now possible to specify the scope of each delay between URL download with DefaultDelayResolver (per crawler, site, or thread). "crawler" is default.
New	Javadoc Jar and Source Jar are now also deployed to Maven repository.
Updated	Deprecation of *.handler.* package. Classes have been moved to more intuitive packages.
Updated	IDelayResolver are no longer systematically synchronized (i.e. accessible only one thread a a time). This is a decision left up to each implementation.
Updated	Reduced the number of calls to the crawl database to improve performance (URL filtering but be successfully passed for a document to get queued for processing).
Updated	Upgraded to Norconex Importer from 1.0.0 to 1.0.2.
Updated	Upgraded to Norconex Commons Lang from 1.0.0 to 1.0.1.
Updated	Connection timeout now 30 seconds in DefaultHttpClientInitializer.
Updated	When deleting downloaded files, the text file from the importer output is now also deleted as URLs are being processed, greatly reducing the number of files on disk during the crawling process.
Updated	The application now prints the full path of an error file when an error occurs on startup (error-*.log).
Fixed	When reprocessed, orphan URLs are processed/filtered in case there has been a configuration change. For instance, if if the maxDepth was reduced, former URLs that were once good may now be invalid due to their depth being too big. Only "valid" URLs are now considered for detecting orphans (OK or UNMODIFIED statuses).	#16
Fixed	Fixed java.io.NotSerializableException when cloning HttpCrawlerConfig (when is used).

1.0.2 Bug fix release. Release date 2013-07-11 Download

Fixed

DefaultURLExtractor now handle URLs starting with ? properly and those prefixed with "URL=" (e.g. meta http-equiv="refresh")

1.0.1 Bug fix release. Release date 2013-06-26 Download

Fixed	Relative URLs starting with two slashes (//) are now extracted properly.	#15
Fixed	Replacements now working for GenericURLNormalizer.	#15

1.0.0 Open Source release Release date 2013-06-05 Download

New

Starting with this release, Norconex HTTP Collector is open-source under GPL.