Norconex Web Crawler

2.x Release Notes

Release History

Version Date Description
2.9.1 2021-10-18 Maintenance release
2.9.0 2019-12-22 Feature release
2.8.1 2018-08-17 Maintenance release
2.8.0 2017-11-26 Feature release
2.7.1 2017-05-26 Bugfix release
2.7.0 2017-04-26 Feature release
2.6.2 2017-01-06 Maintenance release
2.6.1 2016-12-14 Maintenance release
2.6.0 2016-08-25 Feature release
2.5.1 2016-07-27 Maintenance release
2.5.0 2016-06-03 Feature release
2.4.0 2016-02-28 Feature release
2.3.0 2015-11-06 Feature release
2.2.1 2015-08-07 Bug fix release
2.2.0 2015-07-22 Feature release
2.1.0 2015-04-08 Feature release
2.0.2 2015-02-04 Bug fix release
2.0.1 2014-12-03 Bug fix release
2.0.0 2014-11-26 Major release.
1.3.4 2014-08-24 Maintenance release.
1.3.3 2014-08-07 Bug fix release.
1.3.2 2014-06-17 Bug fix release.
1.3.1 2014-04-10 Bug fix release.
1.3.0 2014-03-24 Feature release.
1.2.0 2014-01-11 Feature release.
1.1.1 2013-10-03 Bug fix release.
1.1.0 2013-08-21 Feature release.
1.0.2 2013-07-11 Bug fix release.
1.0.1 2013-06-26 Bug fix release.
1.0.0 2013-06-05 Open Source release

2.9.1 Maintenance release Release date 2021-10-18 Download

New New "validExitCodes" on PhantomJSDocumentFetcher to support other than 0 (now default). #729
New StandardSiteMapResolver can now optionally filter entries that are older than a given date. #718
New StandardSiteMapResolver can now optionally throw exception instead of logging errors. #718
Updated Robots.txt redirects are now being followed. #726
Updated Extracted links are no longer extracted/kept by default for documents having reached the max depth. To keep former behavior, use the new method HttpCrawlerConfig#keepMaxDepthLinks(...). #498
Updated Generic canonical link detector now supports non-standard canonical URL definition using meta "property" instead of "rel". #697
Updated More permissive gzip detection for sitemap.xml files. #687
Updated Can now decide to re-process headers as part of document fetching when metadata fetcher fails. #655
Updated Redirects are now ignored when part of a bad HTTP response when "skipOnBadStatus" is "true". #695
Updated Metadata fetcher exceptions are swallowed and logged when "skipOnBadStatus" is "true". #696
Updated Unsupported content encoding form Content-Type HTTP headers now triggers a detection attempt of both encoding and content type based on document content.
Updated Norconex JEF 4.1.3, Norconex Commons Lang 1.15.2, Norconex Collector Core 1.10.1.
Fixed Fixed sitemap.xml URL entries not being extracted when "loc" is from a sitemap extension namespace (e.g., "image:loc"). #761
Fixed Fixed sitemap.xml URL entries not being extracted when they also contain images. #758
Fixed Fixed RecrawlableResolver settings not always being respected. #741
Fixed Fixed redirect targets sometimes being processed again when already processed normally (not through a redirect). #741
Fixed Fixed NullPointerException when resolving sitemaps. #738
Fixed Fixed authFormParams XML configuration not being loaded properly. #730
Fixed Fixed occasional FileNotFoundException with PhantomJS when the downloaded file was not yet written even if PhantomJS has already returned.
Fixed Fixed REJECTED_REDIRECTED event not being fired for the redirect source URL when the target URL was previously encountered. #678
Fixed Unsupported content encoding form Content-Type HTTP headers no longer rejects a document.

2.9.0 Feature release Release date 2019-12-22 Download

New From Collector Core update, added "unmanaged" attribute to "logsDir" configuration option to prevent the collector from managing its own file-based logging. #593
New Added new normalization rules to GenericURLNormalizer: removeQueryString, lowerCase, lowerCasePath, lowerCaseQuery, lowerCaseQueryParameterNames, lowerCaseQueryParameterValues. #479
New Added "includeSubdomains" to start URL group to consider subdomains as being the same as any URL domain when "stayOnDomain" is true. #563
New GenericLinkExtractor now supports using "selector-syntax" to identify portions of an HTML document to include or exclude from URL extraction.
Updated Switch default cookie specs from DEFAULT to STANDARD and can now be configured. #525
Updated Dependency updates: Norconex Collector Core 1.10.0, Norconex Commons Lang 1.15.1.
Updated Stateful support added for auth sessions (e.g., NTLM). #526
Fixed Fixed GenericCanonicalLinkDetector not handling properly HTTP Response "Link" Header with multiple values. #646
Fixed Fixed GenericRecrawlableResolver large frequency units resulting in those minimum frequencies not being respected. #597
Fixed Fix quoted mediatype from content type. #578
Fixed Fixed exceptions in URLCrawlScopeStrategy#isInScope(...) ending the crawler execution. #540
Fixed Discovered canonical URLs are now rejected if they are out of scope (not matching the stayOnXXX directives). #527
Fixed Fixed good redirected URLs being committed twice in same crawl session. #533
Fixed TikaLinkExtractor now extracts URL with spaces. It should also support other URLs typically considered "bad". #628

2.8.1 Maintenance release Release date 2018-08-17 Download

Updated Dependency updates: Norconex Collector Core 1.9.1, Norconex Commons Lang 1.15.0.
Updated GenericDocumentFetcher now logs documents it could not fetch as INFO. #475
Fixed Fixed XMLFeedLinkExtractor sometimes splitting URLs. #487
Fixed Fixed invalid validation error when XML configuration for PhantomJSDocumentFetcher does not contain "screenshotStorageDiskDir". #473
Fixed Fixed NullPointerException when using PhantomJSDocumentFetcher without specifying any "screenshotDimensions".
Fixed Added validation support for new "authFormParams" option in GenericHttpClientFactory. #434
Fixed Fixed PhantomJSDocumentFetcher sending invalid scheme to PhantomJS proxy (when used with HttpClientProxyCollectorListener). #437

2.8.0 Feature release Release date 2017-11-26 Download

New New FeaturedImageProcessor class to extract the "main" image from a web page.
New New screenshot storage options for PhantomJSDocumentFetcher, like specifying image format, store inline (base64), and exact size.
New New "authFormParams" option on GenericHttpClientFactory for adding arbitrary parameter to authentication forms.
New New metadata field "collector.redirect-trail" for storing all redirect source URLs with the target URL metadata. #397
New New "authPreemptive" configuration option on GenericHttpClientFactory to use preemptive authentication with BASIC authentication. #420
New GenericLinkExtractor can now limit link extraction to specified portion of text only.
New It is now possible to store as metadata URLs that are not "in-scope" with a new flag "keepOutOfScopeLinks". #360
New New GenericHttpClientFactory#removeRequestHeader(String) method.
New New "resourceTimeout" option on PhantomJSDocumentFetcher to prevent individual page items to hang PhantomJS for a long time. #383
Updated Now handles circular references between a non-canonical page pointing to a URL redirecting back to that non-canonical page. When this loop is encountered, it will now process the non-canonical page (used to reject both). #162
Updated GenericLinkExtractor now extracts unquoted URLs. #423
Updated *.http.doc.IHttpDocumentProcessor now deprecated in favor of *.http.processor.IHttpDocumentProcessor.
Updated Crawler event REJECTED_CANONICAL now deprecated in favor of REJECTED_NONCANONICAL.
Updated RegexLinkExtractor now supports replacement values for regex matches.
Updated GenericURLNormalizer XML configuration can now accepts empty normalizations tag to blank default normalizations.
Updated Dependency updates: Norconex Collector Core 1.9.0, Norconex Commons Lang 1.14.0.
Updated PhantomJSDocumentFetcher will now try to write a document for processing even if its status from PhantomJS is not "success" (for cases where valid content could be extracted nonetheless). #383
Updated Blank start URLs are now silently ignored.
Fixed Fixed blank URLs extracted by RegexLinkExtractorin throwing NullPointerException. #422
Fixed Fixed blank lines in start URL seed files throwing NullPointerException. #396
Fixed Orphans being reprocessed no longer triggers the "RecrawlableResolverStage". #390
Fixed Fixed improper saving as XML of GenericDelayResolver.
Fixed Fixed GenericDelayResolver not taking into account ranges that overlap into the next cycle (e.g. 22:00 to 5:00).
Fixed Fixed setting proxy realm having no effect on GenericHttpClientFactory.
Fixed Fixed invalid validation error for "tempDir" element on StandardSitemapResolverFactory.
Fixed Fixed PhantomJSDocumentFetcher sometimes throwing a NullPointerException when content-type could not be established. #408
Removed Removed these deprecated classes: HtmlLinkExtractor, HttpMetadataChecksummer, and deprecated methods: HttpCrawler#getUrlsFiles, HttpCrawler#setUrlsFiles.

2.7.1 Bugfix release Release date 2017-05-26 Download

Updated Dependency updates: Norconex Collector Core 1.8.2.
Fixed Fixed "caseSensitive" flag sometimes having no effect in GenericRecrawlableResolver.

2.7.0 Feature release Release date 2017-04-26 Download

New New PhantomJSDocumentFetcher which relies on an external installation of PhantomJS to fetch documents. This provides a way to crawl web pages with JavaScript-generated content. #95
New New XMLFeedLinkExtractor to extract links out of RSS or Atom feeds. #319
New New RegexLinkExtractor to extract links out of any text file using regular expressions. #236
New Added schema-based XML configuration validation which can be trigged on command prompt with this new flag: -k or --checkcfg
New Can now provide start URLs dynamically with new IStartURLsProvider. #331
New Added "removeTrailingHash" to GenericURLNormalizer. #331
New New "detectContentType" and "detectCharset" options on GenericDocumentFetcher for ignoring the content type and character encoding obtained from the HTTP response headers and detect them instead.
New New features from dependency updates. Collector Core: ICollectorLifeCycleListener. Importer: MergeTagger, ExternalTransformer.
New MongoCrawlDataStoreFactory now accepts encrypted passwords.
New Now distributed with utility scripts.
Updated XML configuration entries expecting millisecond durations can now be provided in human-readable format (e.g., "5 minutes and 30 seconds" or "5m30s").
Updated Dependency updates: Norconex Collection Core 1.8.0, Norconex Commons Lang 1.13.0, Jetty 9.2.20.
Updated Crawler events REJECTED_FILTER, REJECTED_BAD_STATUS, REJECTED_IMPORT, REJECTED_ERROR, REJECTED_ROBOTS_TXT, and REJECTED_CANONICAL, REJECTED_REDIRECTED are now DEBUG in log4j.properties.
Updated Modified Javadoc to include an XML usage example for all XML-configurable classes.
Fixed Fixed OverlappingFileLockException when loading the a sitemap store more than once per JVM instance. #336
Fixed Fixed NullPointerException from GenericLinkExtractor when encountering HTML files with an empty href in their base tag.
Fixed Fixed minor errors in writing IXMLConfigurable classes to XML.
Removed Removed MapDB and Apache Derby crawlstore dependencies/implementations which were deprecated in previous versions.

2.6.2 Maintenance release Release date 2017-01-06 Download

New New "sslProtocols" configuration option on GenericHttpClientFactory to specify which SSL/TLS protocols are supported (e.g., SSLv3, TLSv1, TLSv1.1, TLSv1.2). Useful to enable TLSv1.2 on Java 7 when encountering SSL handshake problems on site supporting TLSv1.2 only.
Fixed Corrected documentation in collector-http-config-reference.xml to indicate default orphan strategy is now PROCESS.

2.6.1 Maintenance release Release date 2016-12-14 Download

New New "addDomainTrailingSlash" normalization rule for GenericURLNormalizer.
Updated Dependency updates: Norconex Commons Lang 1.12.3, JJ2000 5.3, Norconex Collection Core 1.7.0.
Fixed Fixed canonical URLs defined in content with single quote in them being truncated. #111
Fixed Fixed StandardRobotsTxtProivder wrongfully rejecting documents when encountering a blank "Disallow:" directive.
Fixed Fixed GenericLinkExtractor wrongfully resolving relative URLs when an HTML page had a "base href" being a relative page.

2.6.0 Feature release Release date 2016-08-25 Download

New New normalization rule for GenericURLNormalizer: removeTrailingSlash. #290
New Can now specify "notFoundStatusCodes" on GenericMetadataFetcher.
Updated Specifying an empty "path" tag in XML config or, setting a null or empty string array on StandardSitemapResolverFactory#setSitemapPaths(...) method will now prevent trying to locate sitemaps using default locations and will strictly rely on sitemap URLs specified as start URLs or found in robots.txt (if enabled). #292
Updated GenericLinkExtractor no longer extracts URL from HTML/XML comments by default. To re-enable this behavior a new "setCommentsEnabled(boolean)" method has been added. #291
Updated Normalization rule "addTrailingSlash" in GenericURLNormalizer has been renamed "addDirectoryTrailingSlash". #290
Updated SitemapStore now uses MVStore instead of MapDB for storing the list of processed sitemaps.
Updated Referrer data is now always stored for GenericLinkExtractor (default) and TikaLinkExtractor.
Updated Redirects encountered by IHttpMetadataFetcher implementations are now followed by default. #281
Updated Dependency updates: Norconex Collector Core 1.6.0.
Updated Renamed constant HttpMetadata.COLLECTOR_REFERNCED_URLS to HttpMetadata.COLLECTOR_REFERENCED_URLS (was misspelled).
Updated API break: method signature changed for IHttpMetadataFetcher from Properties fetchHTTPHeaders(HttpClient httpClient, String url) to HttpFetchResponse fetchHTTPHeaders( HttpClient httpClient, String url, Properties headers)
Updated The subject for the crawler event HttpCrawlerEvent.DOCUMENT_METADATA_FETCHED is now an instance of HttpFetchResponse.
Updated Extracted canonical URLs now have their referrer reference stored with them.
Updated Extracted URLs are now stored in crawl store.
Updated Now using MVStore for sitemap store (instead of MapDB).
Fixed Fixed GenericRecrawlableResolver#MinFrequency() constructor with arguments not setting patterns correctly. #289
Fixed Fixed documents wrongfully being considered orphans when referrer was skipped for being unmodified or premature, or was in temporary error, before its URLs could be extracted. This could cause valid documents to be deleted/ignored (depending on orphan strategy used). Now those "child" URLs will be queued for processing as if they were extracted from referrer page. #278
Fixed Canonical URLs extracted are now normalized before being compared to their containing page URL, which were already normalized.

2.5.1 Maintenance release Release date 2016-07-27 Download

Updated Updated references to test pages URLs to now be "https" in "minimum" and "complex" example configurations. #252
Updated Dependency updates: Joda Time 2.9.4, Apache HttpClient 4.5.2, JSoup 1.8.3.

2.5.0 Feature release Release date 2016-06-03 Download

New Now supports sitemap "lastmod" and "changefreq", and also allows for specifying custom minimum recrawl frequency for documents. Default implementation is GenericRecrawlableResolver (implements new IRecrawlableResolver).
New New ReferenceDelayResolver that introduces delays between downloads based on matching document reference (URL) patterns. #246
Updated Document "crawl date" and content type can be added to the crawl data and will be stored in the crawl data store (affects all ICrawlDataStoreFactory implementations).
Updated MVStore is now the default URL crawl store.
Updated Dependency updates: Norconex Collector Core 1.5.0.
Updated StandardSitemapResolver now parses sitemaps with content type application/gzip (in addition to application/x-gzip).
Fixed Fixed "stayOnDomain" being true not being honored for extracted URLs with no scheme (//www.example.com). #243
Fixed Fixed AbstractDelay wrongfully computing seconds when logging debug statement.

2.4.0 Feature release Release date 2016-02-28 Download

New New configurable IRedirectURLProvider with default implementation GenericRedirectURLProvider now better handling redirect character encoding and offering encoding options. #199
New HTTP authentication password and proxy password can now be encrypted when using GenericHttpClientFactory. #178
Updated The "log4j.properties" file has been moved from classes to the installation root directory.
Updated GenericLinkExtractor now only supports these URI schemes: http, https, ftp. It is possible to overwrite these default with #setSchemes(String[]). #212
Updated GenericLinkExtractor now attempts by default to detect the character encoding of documents before extracting links and referrer information. In addition, it is now possible to explicitly set which charset to use via the #setCharset(String) method. #194
Updated Derby unit tests are now much faster after settings its system durability for testing to "test".
Updated Added more logging to FORM-based authentication to help troubleshoot authentication issues.
Updated Dependency updates: Apache HttpClient 4.5.1, Norconex Collector Core 1.4.0, Joda Time 2.9.2.
Updated Robot rules now escapes dots and other character having special meaning in regex, before converting robot rules to regex. #200
Updated GenericLinkExtractor now only strips the inside of script tags in HTML pages before extracting links, so that the src links can be followed if configured to do so. #232
Fixed GenericLinkExtractor is now reading characters as opposed to bytes to fix breaking some characters. #219
Fixed Robots.txt "Allow" directive is now supported properly as defined by Google bot usage. #213
Fixed URLStatusCrawlerEventListener no longer prefixes file names with "null" when no file name prefix is explicitly set. #206
Fixed Fixed both GenericLinkExtractor and TikaLinkExtractor which were not able to extract link titles when keepReferrerData was true. #204
Fixed Text is now extracted when HTML is found in an anchor body (when keepReferrerData is true). #204
Fixed GenericHttpClientFactory#trustAllSSLCertificates now disables hostname verification and algorithm constraints. #196
Fixed Fixed "lenient" flag having no effect on "sitemapResolverFactory". #193
Fixed Fixed "base href" HTML tag not being recognized when resolving relative URLs in HTML pages. #188
Fixed When the "class" attribute is not specified in "sitemapResolverFactory" for XML config, the default resolver instance is now used (as opposed to null, which was causing some configs not to be loaded properly). #187
Fixed GenericHttpClientFactory#trustAllSSLCertificates will now disable SSL SNI Extension, and unable unsafe cert algorithms that are now disabled by default since Java 7. #181

2.3.0 Feature release Release date 2015-11-06 Download

New GenericHttpClientFactory now allows you to set HTTP request headers on every HTTP calls a crawler will make. #175
New New crawler configuration options: stayOnProtocol, stayOnDomain, and stayOnPort. These new settings can be applied as attributes to the startURLs tag in an XML configuration, or on the object returned by HttpCrawlerConfig#getURLCrawlScopeStrategy(). This addition affects or replace previous implementations discussed in #138 , #135 , #131 , #17 , and possibly others. #144
New It is now possible to specify one or more sitemap URLs as "start URLs". New HttpCrawlerConfig#[set|get]StartSitemapURLs(...) methods. #128
New GenericURLNormalizer now has the few normalizations by default, as described in its Javadoc.
New New StandardSitemapFactory#setPaths(...) method to specify where to look for sitemap files for each URLs processed (relative to URL root).
Updated The "sitemap" tag used to set the ISitemapResolverFactory implementation has been renamed to "sitemapResolverFactory" to avoid confusion with the "sitemap" tag that can now be set as a start URL.
Updated URL normalization now always takes place by default, using GenericURLNormalizer. Can be turned off by either setting it to null in the crawler configuration, or invoking GenericURLNormalizer#setDisabled(true). #145
Updated URLs extracted from a document are now stored in "collector.referenced-urls" after they have been normalized. #145
Updated URL redirects are now logged as REJECTED_REDIRECTED (log level INFO).
Updated HtmlLinkExtractor has been deprecated in favor of GenericLinkExtractor.
Updated HttpCrawlerConfig#[set|get]UrlsFiles(...) has been deprecated in favor of HttpCrawlerConfig#[set|get]StartUrlsFiles(...)
Updated StandardSitemapFactory#setLocations(...) now deprecated in favor of being able to specify sitemaps as start URLs.
Updated ISitemapResolver#resolveSitemaps(...) as a new argument to specify whether the sitemap locations provided were defined as "start URLs" or not.
Updated Now logs User-Agent upon startup (log level INFO).
Updated Dependency updates: Norconex Collector Core 1.3.0, Norconex Commons Lang 1.8.0.
Updated Added new event types loggers to log4j.properties.
Updated Corrected typos and improved documentation.
Updated Saved and loaded configuration-related classes are now equal. Methods equals/hashCode/toString for those classes are now implemented uniformly and where added where missing.
Fixed Fixed some configuration classes not always being saved to XML properly or giving errors.
Fixed Relative redirect URLs are now converted to absolute. #169
Fixed Fixed robots.txt being fetched before reference filters were executed. Robots.txt are no longer fetched for rejected reference. #168
Fixed GenericLinkExtractor now unescapes HTML entities in URLs.
Fixed Fixed ClassCastException in HttpCrawlerRedirectStrategy when using an HTTP Proxy. #167
Fixed GenericCanonicalLinkDetector now supports links that are escaped (HTML-entities). They are now unescaped before they are processed. #164
Fixed Fixed circular redirect exception. #146
Fixed GenericLinkExtractor and TikaLinkExtractor now extract meta http-equiv refresh properly when "refresh" is without quotes or not lowercase. #141
Fixed Fixed duplicate commits when multiple URL redirects are pointing to the same target URL. #135
Fixed Fixed possible URISyntaxException in GenericURLNormalizer (fixed by updated Norconex Commons Lang URLNormalizer dependency). #132

2.2.1 Bug fix release Release date 2015-08-07 Download

Updated Dependency updates: Norconex Collector Core 1.2.1.
Fixed Fixed NullPointerException occurring inconsistently when using "thread" scope in GenericDelayResolver.
Fixed GenericDocumentFetcher now handles more special characters in URLs. Also saves itself as XML properly now. Created new unit tests for it.
Fixed StandardRobotsTxtProvider now handles empty "Disallow: " properly." #129
Fixed Fixed NullPointerException when configuring an extractor tag in XML without specifying the class attribute (now defaults to HtmlLinkExtractor).

2.2.0 Feature release Release date 2015-07-22 Download

New Added support for canonical links defined in both HTTP Headers or as a link tag in an HTML document head tag. Canonical links detection is always performed unless explicitly disabled. #79
New New URLStatusCrawlerEventListener class for producing reports of fetched URLs and their status. Useful for finding broken links or else.
New Added three new configuration options to GenericHttpClientFactory to better deal with HTTP connectivity issues (like timeouts): "maxConnectionsPerRoute", "maxConnectionIdleTime", and "maxConnectionInactiveTime". #118
New New LastModifiedMetadataChecksummer that uses Last-Modified HTTP header value for checksum purposes, replacing HttpMetadataChecksummer as the default implementation. For choosing one or more fields of your choice to create a checksum, you can now use the new GenericMetadataChecksummer from the Collector Core dependency.
New New CurrentDateTagger, DateMetadataFilter, NumericMetadataFilter, TextPatternTagger, GenericSpoiledReferenceStrategizer and more new features introduced by dependency upgrades.
New New method GenericDocumentFetcher#setNotFoundStatusCodes(int...) to specify one or several custom "Not Found" HTTP codes. Default is 404.
Updated GenericHttpClientFactory default maximum connection was increased from 20 to 200 and default maximum connections per route was increased from 2 to 20. #118
Updated New HttpFetchResponse class now passed to crawl event listeners after a document fetch instead of the IHttpDocumentFetcher used. This adds the ability to listen for specific HTTP response status code. As a consequence, IHttpDocumentFetcher now returns a HttpFetchResponse.
Updated HttpMetadataChecksummer has been deprecated in favor of LastModifiedMetadataChecksummer.
Updated HtmlLinkExtractor now supports specifying tags without an attribute for detecting URLs.
Updated HtmlLinkExtractor now ignores whatever is found between "script" tags so that JavaScript-generated URLs can no longer cause trouble. #119
Updated Dependency updates: Norconex Collector Core 1.2.0, Joda Time 2.8.1, Apache HTTP Client 4.5, Jetty Webapp 9.2.11.v20150529, Apache Ant 1.9.5.
Updated Jar manifest now includes implementation entries and specifications entries (matching Maven pom.xml).
Updated Improved/fixed javadoc.
Updated HttpCrawlState#NOT_FOUND was migrated to Norconex Collector Core CrawlState#NOT_FOUND.
Fixed Fixed HTML documents being skipped when HtmlLinkExtractor found a URL of invalid format. Now a warning is thrown for each bad URLs instead and the document is processed anyway, and good URLs are extracted. #119
Fixed Fixed MongoDB stage/depth compound index. #97
Fixed Fixed MongoCrawlDataSerializer "referrerLinkText" link attribute/metadata having the same value as "referrerLinkTag". #82
Fixed HtmlLinkExtractor now decodes encoded ampersands present in URLs. #88
Fixed Both HtmlLinkExtractor and TikaLinkExtractor no longer extract empty href links. Added this use case to corresponding unit test. #87
Fixed HtmlLinkExtractor now strips leading spaces.
Fixed Fixed "trustAllSSLCertificates" configuration option on GenericHttpClientFactory not being recognized in XML config. #100
Fixed Fixed exceptions thrown in StandardRobotsTxtProvider when robots.txt contained rules ending with ? or when the referrer URL was starting with a space.
Fixed Several invalid characters are now supported in URLs (e.g., commas).
Fixed Fixed GenericDelayResolver not saving in XML properly and fixed its javadoc as well, which did not mention how to pass the delay in XML config.
Fixed TargetURLRedirectStrategy no longer throws an exception when redirects are disabled for a request. #124
Fixed Fixed "sitemapLocations" and "lenient" not being applied to StandardSitemapFactory. #120

2.1.0 Feature release Release date 2015-04-08 Download

New Several new features, updates and fixes were added by upgrading Norconex Collector Core (https://opensource.norconex.com/collectors/core/) and Norconex Importer (https://opensource.norconex.com/importer/) dependencies. Those include support for ORC, translation, a title generator, new content type parsing, and more. Refer to dependency release notes for more details.
New New methods and configuration attribute to disable checksum creation in HttpMetadataChecksummer.
Updated Sitemap resolving pipeline stage is now always invoked (but won't do anything if disabled).
Updated Dependency updates: Norconex Collector Core 1.1.0, JUnit 4.12, Joda-Time 2.7, Apache HTTP Components 4.4, Jetty Webapp 9.2.10.v20150310, Fongo 1.6.2.
Updated Added Sonatype repository to pom.xml for snapshot releases.
Updated Updated several maven plugins and added SonarQube maven plugin.
Updated Improvements on character encoding detection from HTTP headers.
Updated log4j.properties from ./classes/ now properly loaded by collector-http.sh. #59
Updated Improved javadoc.
Updated Added many unit tests for testing start vs resume vs stop vs deleted vs modified, in different JVM instances.
Updated "minimum" and "complex" configuration examples now ignore sitemap.xml files.
Fixed Fixed link extractor not fetching link text properly when keepReferrerData is true on HtmlLinkExtractor. #56
Fixed Robot meta data found in HTML pages will no longer be extracted if found within an HTML comment. Robot metadata detection is also more robust. #60
Fixed Fixed NPE in HttpImporterPipelineUtil#enhanceHTTPHeaders when content type from HTTP header is not defined.
Fixed Fixed log4j log levels incorrectly ending with a semi-colon.

2.0.2 Bug fix release Release date 2015-02-04 Download

Fixed Fixed the collector "stop" action having no effect. #49
Fixed Fixed crawl data wrongfully applied as metadata after the import phase.
Fixed Fixed NullPointerException when sitemap support is disabled.
Fixed Fixed incorrect deletion behavior for embedded orphan documents.
Updated Improved log4j.properties logging options for crawler events.
Updated Upgraded Norconex Collector Core dependency to 1.0.2.

2.0.1 Bug fix release Release date 2014-12-03 Download

Fixed From collector-core-1.0.1: When keepDownloads is true, saved files and directories are now prefixed with "f." and "d." respectively to avoid collisions. #44
Fixed Fixed errors in example configuration files.

2.0.0 Major release. Release date 2014-11-26 Download

New Upgraded Norconex Importer to version 2.0.0, which brings to Norconex HTTP Collector a lot of new features, such as: Document content splitting, splitting of embedded documents into individual documents, new taggers for language detection, changing character case, parsing and formatting dates, providing content statistics, and more.
New Can now supplied a "pathsFile" as part of the startPaths, acting as a seed list.
New New fast MVStore database implementation for URL database (from Norconex Collector Core).
New New H2 database implementation for URL database (crawl data store).
New Now keeps track of parent references (for embedded/split documents).
New More unit tests, with the addition of an embedded Jetty Web server rendering test pages for some unit tests.
New New JMX/MBean support added on crawlers.
New IUrlExtractor is now ILinkExtractor and both their implementing classes (HtmlLinkExtractor and TikaLinkExtractor) now support also extracting a link title and text, and they also support the "nofollow" robot rule. #23
New It is now possible to configure multiple link extraction classes, each taking effect on particular URLs and/or content-types.
New IHtmlLinkExtractor can be configured to use specified HTML tags and attributes to find URLs.
Updated Now licensed under The Apache License, Version 2.0.
Updated Replaced the configuration option "deleteOrphans(true|false)" with "orphansStrategy(DELETE|PROCESS|IGNORE)".
Updated The collector now references document content as reusable InputStream with memory caching instead of relying only on files. This saves a great deal of disk I/O and improves performance in most cases.
Updated Refactored to use the new Norconex Collector Core library. A significant portion of the Norconex HTTP Collector code has been moved to that core library. Some of the moved classes are (base package being com.norconex.collector.*): http.checksum.IHttpDocumentChecksummer to core.checksum.IDocumentChecksummer, http.checksum.IHttpHeadersChecksummer to core.checksum.IMetadataChecksummer, http.checksum.DefaultHttpDocumentChecksummer to core.checksum.impl.MD5DocumentChecksummer, http.filter.IURLFilter to core.filter.IReferenceFilter http.filter.IHttpHeadersFilter to core.filter.IMetadataFilter http.filter.IHttpDocumentFilter to core.filter.IDocumentFilter http.filter.impl.ExtensionURLFilter to core.filter.impl.ExtensionURLFilter http.filter.impl.RegexHeaderFilter to core.filter.impl.RegexMetadataFilter http.filter.impl.RegexURLFilter to core.filter.impl.RegexReferenceFilter
Updated Amongst others, the following classes were renamed (within com.norconex.collector.http.*): checksum.impl.DefaultHttpHeadersChecksummer to checksum.impl.HttpMetadataChecksummer, client.impl.DefaultHttpClientFactory to client.impl.GenericHttpClientFactory, delay.impl.DefaultDelayResolver to delay.impl.GenericDelayResolver, fetch.impl.DefaultDocumentFetcher to fetch.impl.GenericDocumentFetcher, fetch.impl.SimpleHttpHeadersFetcher to fetch.impl.GenericHttpHeadersFetcher, robot.impl.DefaultRobotsMetaProvider to robot.impl.StandardRobotsMetaProvider, robot.impl.DefaultRobotsTxtProvider to robot.impl.StandardRobotsTxtProvider, sitemap.impl.DefaultSitemapResolver to sitemap.impl.StandardSitemapResolver, url.impl.DefaultURLExtractor to url.impl.GenericURLExtractor
Updated Several references to "url" were changed to "reference".
Updated New and more scalable crawler event model along with new listeners.
Updated Refactored to use JEF 4.0.0 which makes the HTTP Collector easier to monitor.
Updated Other libray upgrades: Norconex Committer to 2.0.0 and Norconex Commons Lang to 1.5.0.
Updated Removed previously deprecated classes.
Updated Crawled sitemap details now has its own store (no longer mixed with the crawl data store).
Updated ISiteMapResolver now needs an ISiteMapResolverFactory.
Fixed Sitemap resolution now stops when a stop request is issued. #38
Fixed Now checks if crawler is running before attempting to stop it. #37

1.3.4 Maintenance release. Release date 2014-08-24 Download

New MongoCrawlURLDatabase now supports user authentication.
Updated Now requires Java 7 or higher.
Fixed Fixed DefaultRobotsTxtProvider failing to parse some robots.txt patterns. #36
Fixed Fixed DefaultRobotsTxtProvider failing to parse some robots.txt patterns. #36

1.3.3 Bug fix release. Release date 2014-08-07 Download

Updated Upgraded JEF to 3.0.1 to fix stop action not working.
Fixed Fixed NullPointerException in robots.txt resolution under some circonstances.

1.3.2 Bug fix release. Release date 2014-06-17 Download

Fixed DefaultURLExtractor no longer treat empty href as being a URL ending with a double-quote.
Updated Renamed HttpMetadata key "collector.http.dept" to "collector.http.depth" (typo fix).
Updated Upgraded Norconex Commons Lang to 1.3.2
Updated GenericURLNormallizer no longer rejects URLs with spaces in them. It now logs a warning instead (thanks to Norconex Commons Lang upgrade).

1.3.1 Bug fix release. Release date 2014-04-10 Download

Fixed Header and document checksum value are no longer added by default to prevent the issue described in github ticket #24 . Instead, adding checksum is now an optional feature of DefaultHttpDocumentChecksummer and DefaultHttpHeadersChecksummer. #24

1.3.0 Feature release. Release date 2014-03-24 Download

New Now supports NTLM authentication. SPNEGO and Kerberos were also added but are experimental (see DefaultHttpClientFactory).
New Can now specify character set of HTTP connections and authentication forms.
New Can now set custom timeout values on HTTP connection-related activities.
New New option to trust all SSL certificates of sites being crawled (see DefaultHttpClientFactory).
New Can now specify a maximum number of HTTP connections for each crawler independently of configured number of threads (see DefaultHttpClientFactory).
New DefaultHttpClientFactory introduces additional configuration options: proxy scheme, 'Expect: 100-continue' handshake, maximum HTTP redirects, local address, stale connection checks
New HTTP header checksum and document checksum are now added to the document metadata as HttpMetadata#CHECKSUM_HEADER and HttpMetadata#CHECKSUM_DOC.
Updated The empty sub-folders contained under the "download" folder are now periodically deleted. This speeds up directory scanning and increases performance on large crawls.
Updated The userAgent is now a crawler configuration option (previously was an option of DefaultHttpClientInitializer )
Updated API change: IRobotsTxtProvider#getRobotsTxt(...) method signature has changed to accept the User-Agent.
Updated IHttpClientInitializer is now deprecated in favor of IHttpClientFactory, giving you more control over HttpClient creation.
Updated API change: Methods previously accepting DefaultHttpClient instances now have their signature accepting parent interface HttpClient instead.
Updated More logging to help resolve crawler issues with DEBUG log level.
Updated HttpCrawler more lenient upon encountering some errors that were previous aborting entire execution.
Updated Library upgrades. Updated default crawl url database (MapDB) to version 0.9.10, Norconex Commons Lang to 1.3.0, Norconex Committer to 1.2.0, Norconex Importer to 1.2.0, and Apache HttpClient to 4.3.2.
Fixed Now ensures that robots.txt agent matching gives priority to the most specific match (as opposed to the first match). Sitemaps detected in robots.txt are also preserved for sitemap resolving.
Removed Removed classes deprecated since 1.1.

1.2.0 Feature release. Release date 2014-01-11 Download

New New optional Mongo URL Database implementation.
New New TikaURLExtractor class providing an alternate IURLExtractor implementation based on Apache Tika HTMLParser.
New New SegmentCountURLFilter class for filtering URLs having a specified number of segments (can check duplicate segments too).
New New unit tests.
Updated MapDB URL Database classes moved to its own "mapdb" package. DefaultCrawlURLDatabaseFactory still exists, but is just a pointer to MapDBCrawlURLDatabaseFactory.
Updated Example configurations now point to Norconex test pages to ensure their stability.
Updated Upgraded dependent libraries: Norconex Committer 1.1.0, Norconex Commons Lang 1.2.0, MapDB 0.9.8 and other thrid party libraries.
Updated Improved Javadoc.

1.1.1 Bug fix release. Release date 2013-10-03 Download

Fixed Fixed not being able to extract the "href" attribute when it starts a new line. #18
Fixed Fixed HTTP redirects not storing final target URL but the source URL instead. #17
Updated Upgraded dependent libraries to Norconex Importer 1.1.0 and Norconex Commons Lang 1.1.0.

1.1.0 Feature release. Release date 2013-08-21 Download

New Crawler now fires additional events. Added "documentRobotsMetaRejected" and "documentImportRejected" methods to IHttpCrawlerEventListener.
New DefaultCrawlURLDatabase now uses a MapDB-based implementation for faster performance. The derby implementation has been kept for those with a preference for it.
New Now support sitemap.xml and sitemap index (plain or gzip). URLs from sitemaps will have the sitemap information as metadata.
New BASIC and DIGEST authentication now supported.
New Now supports in-page robot instructions. Via "ROBOTS" meta tag first, or "X-Robots-Tag" tag if present in HTTP header.
New "ftp" protocol now supported.
New It is now possible to specify the scope of each delay between URL download with DefaultDelayResolver (per crawler, site, or thread). "crawler" is default.
New Javadoc Jar and Source Jar are now also deployed to Maven repository.
Updated Deprecation of *.handler.* package. Classes have been moved to more intuitive packages.
Updated IDelayResolver are no longer systematically synchronized (i.e. accessible only one thread a a time). This is a decision left up to each implementation.
Updated Reduced the number of calls to the crawl database to improve performance (URL filtering but be successfully passed for a document to get queued for processing).
Updated Upgraded to Norconex Importer from 1.0.0 to 1.0.2.
Updated Upgraded to Norconex Commons Lang from 1.0.0 to 1.0.1.
Updated Connection timeout now 30 seconds in DefaultHttpClientInitializer.
Updated When deleting downloaded files, the text file from the importer output is now also deleted as URLs are being processed, greatly reducing the number of files on disk during the crawling process.
Updated The application now prints the full path of an error file when an error occurs on startup (error-*.log).
Fixed When reprocessed, orphan URLs are processed/filtered in case there has been a configuration change. For instance, if if the maxDepth was reduced, former URLs that were once good may now be invalid due to their depth being too big. Only "valid" URLs are now considered for detecting orphans (OK or UNMODIFIED statuses). #16
Fixed Fixed java.io.NotSerializableException when cloning HttpCrawlerConfig (when is used).

1.0.2 Bug fix release. Release date 2013-07-11 Download

Fixed DefaultURLExtractor now handle URLs starting with ? properly and those prefixed with "URL=" (e.g. meta http-equiv="refresh")

1.0.1 Bug fix release. Release date 2013-06-26 Download

Fixed Relative URLs starting with two slashes (//) are now extracted properly. #15
Fixed Replacements now working for GenericURLNormalizer. #15

1.0.0 Open Source release Release date 2013-06-05 Download

New Starting with this release, Norconex HTTP Collector is open-source under GPL.