com.norconex.collector.core.data.store.impl.mongo

Class AbstractMongoCrawlDataStoreFactory

java.lang.Object

com.norconex.collector.core.data.store.impl.mongo.AbstractMongoCrawlDataStoreFactory

All Implemented Interfaces:

ICrawlDataStoreFactory, IXMLConfigurable

public abstract class AbstractMongoCrawlDataStoreFactory
extends Object
implements ICrawlDataStoreFactory, IXMLConfigurable

Mongo implementation of ICrawlDataStore.

All the references are stored in a collection named 'references'. They go from the "QUEUED", "ACTIVE" and "PROCESSED" stages.

The cached references are stored in a separated collection named "cached".

As of 1.8.0, password can take a password that has been encrypted using EncryptionUtil (or command-line encrypt.[bat|sh]). In order for the password to be decrypted properly by the crawler, you need to specify the encryption key used to encrypt it. The key can be stored in a few supported locations and a combination of passwordKey and passwordKeySource must be specified to properly locate the key. The supported sources are:

passwordKeySource	passwordKey
key	The actual encryption key.
file	Path to a file containing the encryption key.
environment	Name of an environment variable containing the key.
property	Name of a JVM system property containing the key.

Implementing classes should contain the following XML configuration usage:

  <crawlDataStoreFactory class="(class name)">
      <host>(Optional Mongo server hostname. Default to localhost)</host>
      <port>(Optional Mongo port. Default to 27017)</port>
      <dbname>(Optional Mongo database name. Default to crawl id)</dbname>
      <username>(Optional user name)</username>
      <password>(Optional user password)</password>
      <cachedCollectionName>(Custom "cached" collection name)</cachedCollectionName>
      <referencesCollectionName>(Custom "references" collection name)</referencesCollectionName>
      <mechanism>(Optional authentication mechanism)</mechanism>
      <sslEnabled>[false|true]</sslEnabled>
      <sslInvalidHostNameAllowed>[false|true]</sslInvalidHostNameAllowed>

      <!-- Use the following if password is encrypted. -->
      <passwordKey>(the encryption key or a reference to it)</passwordKey>
      <passwordKeySource>[key|file|environment|property]</passwordKeySource>
  </crawlDataStoreFactory>
 

If "username" is not provided, no authentication will be attempted. The "username" must be a valid user that has the "readWrite" role over the database (set with "dbname").

Authentication mechanism

As of 1.8.1, it is now possible to specify the MongoDB authentication mechanism to use. The following are supported:

• MONGODB-CR
• SCRAM-SHA-1

When no mechanism is specified, the default mechanism will be the Challenge Response (MONGODB-CR) for MongoDB 2 and and SCRAM SHA1 (SCRAM-SHA-1) for MongoDB 3+. The following is an example forcing MONGODB-CR authentication:

      <username>joe_user</username>
      <password>joe_pwd</password>
      <mechanism>MONGODB-CR</mechanism>
 

As of 1.9.0, you can define your own collection names with setReferencesCollectionName(String) and setCachedCollectionName(String).

As of 1.10.0, you can enable SSL.

Author:

Pascal Essiembre

See Also:

BaseMongoSerializer

Constructor Summary

Constructors

Constructor and Description
AbstractMongoCrawlDataStoreFactory()

Method Summary

Modifier and Type	Method and Description
ICrawlDataStore	createCrawlDataStore(ICrawlerConfig config, boolean resume) Creates a new crawl data store.
protected abstract IMongoSerializer	createMongoSerializer()
boolean	equals(Object other)
String	getCachedCollectionName() Gets the cached collection name.
MongoConnectionDetails	getConnectionDetails()
String	getReferencesCollectionName() Gets the references collection name.
int	hashCode()
void	loadFromXML(Reader in)
void	saveToXML(Writer out)
void	setCachedCollectionName(String cachedCollectionName) Sets the cached collection name.
void	setReferencesCollectionName(String referencesCollectionName) Sets the references collection name.
String	toString()

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

AbstractMongoCrawlDataStoreFactory

public AbstractMongoCrawlDataStoreFactory()

Method Detail

createCrawlDataStore

public ICrawlDataStore createCrawlDataStore(ICrawlerConfig config,
                                            boolean resume)

Description copied from interface: ICrawlDataStoreFactory

Creates a new crawl data store.

Specified by:

createCrawlDataStore in interface ICrawlDataStoreFactory

Parameters:

config - crawler configuration

resume - whether the crawler was started or resumed

Returns:

new crawl data store

getConnectionDetails

public MongoConnectionDetails getConnectionDetails()

getReferencesCollectionName

public String getReferencesCollectionName()

Gets the references collection name. Defaults to "references".

Returns:

collection name

Since:

1.9.0

setReferencesCollectionName

public void setReferencesCollectionName(String referencesCollectionName)

Sets the references collection name.

Parameters:

referencesCollectionName - collection name

Since:

1.9.0

getCachedCollectionName

public String getCachedCollectionName()

Gets the cached collection name. Defaults to "cached".

Returns:

collection name

Since:

1.9.0

setCachedCollectionName

public void setCachedCollectionName(String cachedCollectionName)

Sets the cached collection name.

Parameters:

cachedCollectionName - collection name

Since:

1.9.0

createMongoSerializer

protected abstract IMongoSerializer createMongoSerializer()

loadFromXML

public void loadFromXML(Reader in)
                 throws IOException

Specified by:

loadFromXML in interface IXMLConfigurable

Throws:

IOException

saveToXML

public void saveToXML(Writer out)
               throws IOException

Specified by:

saveToXML in interface IXMLConfigurable

Throws:

IOException

equals

public boolean equals(Object other)

Overrides:

equals in class Object

hashCode

public int hashCode()

Overrides:

hashCode in class Object

toString

public String toString()

Overrides:

toString in class Object