GenericFilesystemOptionsProvider (Norconex Filesystem Collector 2.9.2-SNAPSHOT API)

java.lang.Object
- com.norconex.collector.fs.option.impl.GenericFilesystemOptionsProvider

All Implemented Interfaces:: IFilesystemOptionsProvider, IXMLConfigurable

public class GenericFilesystemOptionsProvider
extends Object
implements IFilesystemOptionsProvider, IXMLConfigurable

Generic implementation of IFilesystemOptionsProvider. This provider makes configurable many Commons VFS options. The same FileSystemOptions instance is returned regardless of the FileObject supplied. In many case, it is not necessary to configure any file system options.

Password encryption:

It is usually recommended to provide credentials here instead of doing so directly on the requested URL to prevent password from appearing in log entries. The authPassword can take a password that has been encrypted using EncryptionUtil. In order for the password to be decrypted properly by the crawler, you need to specify the encryption key used to encrypt it. The key can be stored in a few supported locations and a combination of [auth|proxy]PasswordKey and [auth|proxy]PasswordKeySource must be specified to properly locate the key. The supported sources are:

`[...]PasswordKeySource`	`[...]PasswordKey`
`key`	The actual encryption key.
`file`	Path to a file containing the encryption key.
`environment`	Name of an environment variable containing the key.
`property`	Name of a JVM system property containing the key.

Still, if you insist in having the password set on the URL, Apache Commons VFS offers a way to encrypt it there using their own EncryptUtil. More info under the "Naming" section here: http://commons.apache.org/proper/commons-vfs/filesystems.html

Available Options

You will not find methods or XML configuration tags for all options supported by various file systems. If you need to configure more advanced options for your file system, consider having your own implementation of IFilesystemOptionsProvider or extend this class and implement the buildOptions(FileSystemOptions) method to set additional options.

As of 2.9.0, CMIS-enabled Content Management Systems (CMS) are supported (Atom end-point). The start path can be specified as: cmis-atom:http://yourhost:port/path/to/atom. Optionally you can have a non-root starting path by adding the path name to the base URL, with an exclamation mark as a separator: cmis-atom:http://yourhost:port/path/to/atom!/MyFolder/MySubFolder. Start paths are assumed to be Atom URLs.

As of 2.7.0, XML configuration entries expecting millisecond durations can be provided in human-readable format (English only), as per DurationParser (e.g., "5 minutes and 30 seconds" or "5m30s").

XML configuration usage:

  <optionsProvider class="com.norconex.collector.fs.option.impl.GenericFilesystemOptionsProvider">

      <!-- Authentication (any file system) -->
      <authDomain>...</authDomain>
      <authUsername>...</authUsername>
      <authPassword>...</authPassword>
      <!-- Use the following if password is encrypted. -->
      <authPasswordKey>(the encryption key or a reference to it)</authPasswordKey>
      <authPasswordKeySource>[key|file|environment|property]</authPasswordKeySource>

      <!-- FTP (for FTPS options set ftpSecure to "true") -->
      <ftpSecure>[false|true]</ftpSecure>
      <ftpConnectTimeout>(milliseconds)</ftpConnectTimeout>
      <ftpControlEncoding>...</ftpControlEncoding>
      <ftpDataTimeout>(milliseconds)</ftpDataTimeout>
      <ftpDefaultDateFormat>...</ftpDefaultDateFormat>
      <ftpFileType>[ASCII|BINARY|LOCAL|EBCDIC]</ftpFileType>
      <ftpPassiveMode>[false|true]</ftpPassiveMode>
      <ftpRecentDateFormat>...</ftpRecentDateFormat>
      <ftpRemoteVerification>[false|true]</ftpRemoteVerification>
      <ftpServerLanguageCode>...</ftpServerLanguageCode>
      <ftpServerTimeZoneId>...</ftpServerTimeZoneId>
      <ftpShortMonthNames>(comma-separated list)</ftpShortMonthNames>
      <ftpSoTimeout>(milliseconds)</ftpSoTimeout>
      <ftpUserDirIsRoot>[false|true]</ftpUserDirIsRoot>

      <!-- HDFS -->
      <hdfsConfigName>...</hdfsConfigName>
      <hdfsConfigPath>...</hdfsConfigPath>
      <hdfsConfigURL>...</hdfsConfigURL>

      <!-- HTTP (for Webdav options, set httpWebdav to "true") -->
      <httpWebdav>[false|true]</httpWebdav>
      <httpConnectionTimeout>(milliseconds)</httpConnectionTimeout>
      <httpFollowRedirect>[false|true]</httpFollowRedirect>
      <httpMaxConnectionsPerHost>...</httpMaxConnectionsPerHost>
      <httpMaxTotalConnections>...</httpMaxTotalConnections>
      <httpPreemptiveAuth>[false|true]</httpPreemptiveAuth>
      <httpSoTimeout>(milliseconds)</httpSoTimeout>
      <httpUrlCharset>...</httpUrlCharset>
      <httpUserAgent>...</httpUserAgent>
      <httpWebdavCreatorName>...</httpWebdavCreatorName>
      <httpWebdavVersioning>[false|true]</httpWebdavVersioning>

      <!-- RAM -->
      <ramMaxSize>(number of bytes)</ramMaxSize>

      <!-- SFTP -->
      <sftpCompression>...</sftpCompression>
      <sftpFileNameEncoding>...</sftpFileNameEncoding>
      <sftpKnownHosts>...</sftpKnownHosts>
      <sftpPreferredAuthentications>...</sftpPreferredAuthentications>
      <sftpStrictHostKeyChecking>[no|yes|ask]</sftpStrictHostKeyChecking>
      <sftpTimeout>(milliseconds)</sftpTimeout>
      <sftpUserDirIsRoot>[false|true]</sftpUserDirIsRoot>

      <!-- CMIS -->
      <cmisRepositoryId>
          (Optional repository ID, defaults to first one found.)
      </cmisRepositoryId>
      <cmisXmlTargetField>
          (Optional target field name where to store the raw CMIS REST API
           XML. Default does not store the raw XML in a field.)
      </cmisXmlTargetField>

  </optionsProvider>

Usage example:

The following sets the FTP settings sometimes required to get directory listings on remote servers.

  <optionsProvider
      class="com.norconex.collector.fs.option.impl.GenericFilesystemOptionsProvider">
      <ftpPassiveMode>true</ftpPassiveMode>
      <ftpUserDirIsRoot>false</ftpUserDirIsRoot>
  </optionsProvider>

Since:: 2.7.0
Author:: Pascal Essiembre

Constructor Summary

Constructors
Constructor and Description

GenericFilesystemOptionsProvider()

Constructors
Constructor and Description
`GenericFilesystemOptionsProvider()`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected void`	`buildOptions(org.apache.commons.vfs2.FileSystemOptions opts)` For subclasses to apply more advanced configuration on file system options.
`boolean`	`equals(Object other)`
`String`	`getAuthDomain()` Gets the authentication domain.
`String`	`getAuthPassword()` Gets the authentication password.
`EncryptionKey`	`getAuthPasswordKey()` Gets the authentication password encryption key.
`String`	`getAuthUsername()` Gets the authentication username.
`String`	`getCmisRepositoryId()` Gets CMIS repository ID.
`String`	`getCmisXmlTargetField()` Sets the name of the field where the raw XML obtained from the CMIS REST API will be stored.
`org.apache.commons.vfs2.FileSystemOptions`	`getFilesystemOptions(org.apache.commons.vfs2.FileObject fileObject)` Provide file system options associated with a file object.
`Integer`	`getFtpConnectTimeout()`
`String`	`getFtpControlEncoding()`
`Integer`	`getFtpDataTimeout()`
`String`	`getFtpDefaultDateFormat()`
`org.apache.commons.vfs2.provider.ftp.FtpFileType`	`getFtpFileType()`
`String`	`getFtpRecentDateFormat()`
`String`	`getFtpServerLanguageCode()`
`String`	`getFtpServerTimeZoneId()`
`String[]`	`getFtpShortMonthNames()`
`Integer`	`getFtpSoTimeout()`
`String`	`getHdfsConfigName()`
`org.apache.hadoop.fs.Path`	`getHdfsConfigPath()`
`URL`	`getHdfsConfigURL()`
`int`	`getHttpConnectionTimeout()`
`int`	`getHttpMaxConnectionsPerHost()`
`int`	`getHttpMaxTotalConnections()`
`int`	`getHttpSoTimeout()`
`String`	`getHttpUrlCharset()`
`String`	`getHttpUserAgent()`
`String`	`getHttpWebdavCreatorName()`
`long`	`getRamMaxSize()`
`String`	`getSftpCompression()`
`String`	`getSftpFileNameEncoding()`
`File`	`getSftpKnownHosts()`
`String`	`getSftpPreferredAuthentications()`
`String`	`getSftpStrictHostKeyChecking()`
`int`	`getSftpTimeout()`
`int`	`hashCode()`
`boolean`	`isFtpPassiveMode()`
`boolean`	`isFtpRemoteVerification()`
`boolean`	`isFtpSecure()`
`boolean`	`isFtpUserDirIsRoot()`
`boolean`	`isHttpFollowRedirect()`
`boolean`	`isHttpPreemptiveAuth()`
`boolean`	`isHttpWebdav()`
`boolean`	`isHttpWebdavVersioning()`
`boolean`	`isSftpUserDirIsRoot()`
`void`	`loadFromXML(Reader in)`
`void`	`rebuildOptions()` Rebuilds file system options.
`void`	`saveToXML(Writer out)`
`void`	`setAuthDomain(String authDomain)` Sets the authentication domain.
`void`	`setAuthPassword(String authPassword)` Sets the authentication password.
`void`	`setAuthPasswordKey(EncryptionKey authPasswordKey)` Sets the authentication password encryption key.
`void`	`setAuthUsername(String authUsername)` Sets the authentication username.
`void`	`setCmisRepositoryId(String cmisRepositoryId)` Sets CMIS repository ID.
`void`	`setCmisXmlTargetField(String cmisXmlTargetField)` Sets the name of the field where the raw XML obtained from the CMIS REST API will be stored.
`void`	`setFtpConnectTimeout(Integer ftpConnectTimeout)`
`void`	`setFtpControlEncoding(String ftpControlEncoding)`
`void`	`setFtpDataTimeout(Integer ftpDataTimeout)`
`void`	`setFtpDefaultDateFormat(String ftpDefaultDateFormat)`
`void`	`setFtpFileType(org.apache.commons.vfs2.provider.ftp.FtpFileType ftpFileType)`
`void`	`setFtpPassiveMode(boolean ftpPassiveMode)`
`void`	`setFtpRecentDateFormat(String ftpRecentDateFormat)`
`void`	`setFtpRemoteVerification(boolean ftpRemoteVerification)`
`void`	`setFtpSecure(boolean ftpSecure)`
`void`	`setFtpServerLanguageCode(String ftpServerLanguageCode)`
`void`	`setFtpServerTimeZoneId(String ftpServerTimeZoneId)`
`void`	`setFtpShortMonthNames(String... ftpShortMonthNames)`
`void`	`setFtpSoTimeout(Integer ftpSoTimeout)`
`void`	`setFtpUserDirIsRoot(boolean ftpUserDirIsRoot)`
`void`	`setHdfsConfigName(String hdfsConfigName)`
`void`	`setHdfsConfigPath(org.apache.hadoop.fs.Path hdfsConfigPath)`
`void`	`setHdfsConfigURL(URL hdfsConfigURL)`
`void`	`setHttpConnectionTimeout(int httpConnectionTimeout)`
`void`	`setHttpFollowRedirect(boolean httpFollowRedirect)`
`void`	`setHttpMaxConnectionsPerHost(int httpMaxConnectionsPerHost)`
`void`	`setHttpMaxTotalConnections(int httpMaxTotalConnections)`
`void`	`setHttpPreemptiveAuth(boolean httpPreemptiveAuth)`
`void`	`setHttpSoTimeout(int httpSoTimeout)`
`void`	`setHttpUrlCharset(String httpUrlCharset)`
`void`	`setHttpUserAgent(String httpUserAgent)`
`void`	`setHttpWebdav(boolean httpWebdav)`
`void`	`setHttpWebdavCreatorName(String httpWebdavCreatorName)`
`void`	`setHttpWebdavVersioning(boolean httpWebdavVersioning)`
`void`	`setRamMaxSize(long ramMaxSize)`
`void`	`setSftpCompression(String sftpCompression)`
`void`	`setSftpFileNameEncoding(String sftpFileNameEncoding)`
`void`	`setSftpKnownHosts(File sftpKnownHosts)`
`void`	`setSftpPreferredAuthentications(String sftpPreferredAuthentications)`
`void`	`setSftpStrictHostKeyChecking(String sftpStrictHostKeyChecking)`
`void`	`setSftpTimeout(int sftpTimeout)`
`void`	`setSftpUserDirIsRoot(boolean sftpUserDirIsRoot)`
`String`	`toString()`

Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail
- GenericFilesystemOptionsProvider
```
public GenericFilesystemOptionsProvider()
```

Method Detail

rebuildOptions
```
public void rebuildOptions()
```
Rebuilds file system options. By default the options are reused. Calling this method asks this provider to rebuild the options before using them again.

buildOptions
```
protected void buildOptions(org.apache.commons.vfs2.FileSystemOptions opts)
```
For subclasses to apply more advanced configuration on file system options. Default implementation does nothing.

Parameters:

opts - file system options

getAuthDomain
```
public String getAuthDomain()
```
Gets the authentication domain.

Returns:

domain

setAuthDomain
```
public void setAuthDomain(String authDomain)
```
Sets the authentication domain.

Parameters:

authDomain - domain

getAuthUsername
```
public String getAuthUsername()
```
Gets the authentication username.

Returns:

username

setAuthUsername
```
public void setAuthUsername(String authUsername)
```
Sets the authentication username.

Parameters:

authUsername - username

getAuthPassword
```
public String getAuthPassword()
```
Gets the authentication password.

Returns:

the password

setAuthPassword
```
public void setAuthPassword(String authPassword)
```
Sets the authentication password.

Parameters:

authPassword - password

getAuthPasswordKey
```
public EncryptionKey getAuthPasswordKey()
```
Gets the authentication password encryption key.

Returns:

the password key or null if the password is not encrypted.

See Also:

EncryptionUtil

setAuthPasswordKey
```
public void setAuthPasswordKey(EncryptionKey authPasswordKey)
```
Sets the authentication password encryption key. Only required when the password is encrypted.

Parameters:

authPasswordKey - password key

See Also:

EncryptionUtil

getFtpConnectTimeout
```
public Integer getFtpConnectTimeout()
```

setFtpConnectTimeout

public void setFtpConnectTimeout(Integer ftpConnectTimeout)

getFtpControlEncoding
```
public String getFtpControlEncoding()
```

setFtpControlEncoding

public void setFtpControlEncoding(String ftpControlEncoding)

getFtpDataTimeout
```
public Integer getFtpDataTimeout()
```

setFtpDataTimeout

public void setFtpDataTimeout(Integer ftpDataTimeout)

getFtpDefaultDateFormat

public String getFtpDefaultDateFormat()

setFtpDefaultDateFormat

public void setFtpDefaultDateFormat(String ftpDefaultDateFormat)

getFtpFileType

public org.apache.commons.vfs2.provider.ftp.FtpFileType getFtpFileType()

setFtpFileType

public void setFtpFileType(org.apache.commons.vfs2.provider.ftp.FtpFileType ftpFileType)

isFtpPassiveMode
```
public boolean isFtpPassiveMode()
```

setFtpPassiveMode

public void setFtpPassiveMode(boolean ftpPassiveMode)

getFtpRecentDateFormat

public String getFtpRecentDateFormat()

setFtpRecentDateFormat

public void setFtpRecentDateFormat(String ftpRecentDateFormat)

isFtpRemoteVerification

public boolean isFtpRemoteVerification()

setFtpRemoteVerification

public void setFtpRemoteVerification(boolean ftpRemoteVerification)

getFtpServerLanguageCode

public String getFtpServerLanguageCode()

setFtpServerLanguageCode

public void setFtpServerLanguageCode(String ftpServerLanguageCode)

getFtpServerTimeZoneId

public String getFtpServerTimeZoneId()

setFtpServerTimeZoneId

public void setFtpServerTimeZoneId(String ftpServerTimeZoneId)

getFtpShortMonthNames

public String[] getFtpShortMonthNames()

setFtpShortMonthNames

public void setFtpShortMonthNames(String... ftpShortMonthNames)

getFtpSoTimeout
```
public Integer getFtpSoTimeout()
```

setFtpSoTimeout

public void setFtpSoTimeout(Integer ftpSoTimeout)

isFtpUserDirIsRoot
```
public boolean isFtpUserDirIsRoot()
```

setFtpUserDirIsRoot

public void setFtpUserDirIsRoot(boolean ftpUserDirIsRoot)

isFtpSecure
```
public boolean isFtpSecure()
```

setFtpSecure

public void setFtpSecure(boolean ftpSecure)

getHdfsConfigName
```
public String getHdfsConfigName()
```

setHdfsConfigName

public void setHdfsConfigName(String hdfsConfigName)

getHdfsConfigPath

public org.apache.hadoop.fs.Path getHdfsConfigPath()

setHdfsConfigPath

public void setHdfsConfigPath(org.apache.hadoop.fs.Path hdfsConfigPath)

getHdfsConfigURL
```
public URL getHdfsConfigURL()
```

setHdfsConfigURL

public void setHdfsConfigURL(URL hdfsConfigURL)

getHttpConnectionTimeout
```
public int getHttpConnectionTimeout()
```

setHttpConnectionTimeout

public void setHttpConnectionTimeout(int httpConnectionTimeout)

isHttpFollowRedirect
```
public boolean isHttpFollowRedirect()
```

setHttpFollowRedirect

public void setHttpFollowRedirect(boolean httpFollowRedirect)

getHttpMaxConnectionsPerHost

public int getHttpMaxConnectionsPerHost()

setHttpMaxConnectionsPerHost

public void setHttpMaxConnectionsPerHost(int httpMaxConnectionsPerHost)

getHttpMaxTotalConnections

public int getHttpMaxTotalConnections()

setHttpMaxTotalConnections

public void setHttpMaxTotalConnections(int httpMaxTotalConnections)

isHttpPreemptiveAuth
```
public boolean isHttpPreemptiveAuth()
```

setHttpPreemptiveAuth

public void setHttpPreemptiveAuth(boolean httpPreemptiveAuth)

getHttpSoTimeout
```
public int getHttpSoTimeout()
```

setHttpSoTimeout

public void setHttpSoTimeout(int httpSoTimeout)

getHttpUrlCharset
```
public String getHttpUrlCharset()
```

setHttpUrlCharset

public void setHttpUrlCharset(String httpUrlCharset)

getHttpUserAgent
```
public String getHttpUserAgent()
```

setHttpUserAgent

public void setHttpUserAgent(String httpUserAgent)

isHttpWebdav
```
public boolean isHttpWebdav()
```

setHttpWebdav

public void setHttpWebdav(boolean httpWebdav)

getHttpWebdavCreatorName

public String getHttpWebdavCreatorName()

setHttpWebdavCreatorName

public void setHttpWebdavCreatorName(String httpWebdavCreatorName)

isHttpWebdavVersioning

public boolean isHttpWebdavVersioning()

setHttpWebdavVersioning

public void setHttpWebdavVersioning(boolean httpWebdavVersioning)

getRamMaxSize
```
public long getRamMaxSize()
```

setRamMaxSize

public void setRamMaxSize(long ramMaxSize)

getSftpCompression
```
public String getSftpCompression()
```

setSftpCompression

public void setSftpCompression(String sftpCompression)

getSftpFileNameEncoding

public String getSftpFileNameEncoding()

setSftpFileNameEncoding

public void setSftpFileNameEncoding(String sftpFileNameEncoding)

getSftpKnownHosts
```
public File getSftpKnownHosts()
```

setSftpKnownHosts

public void setSftpKnownHosts(File sftpKnownHosts)

getSftpPreferredAuthentications

public String getSftpPreferredAuthentications()

setSftpPreferredAuthentications

public void setSftpPreferredAuthentications(String sftpPreferredAuthentications)

getSftpStrictHostKeyChecking

public String getSftpStrictHostKeyChecking()

setSftpStrictHostKeyChecking

public void setSftpStrictHostKeyChecking(String sftpStrictHostKeyChecking)

getSftpTimeout
```
public int getSftpTimeout()
```

setSftpTimeout

public void setSftpTimeout(int sftpTimeout)

isSftpUserDirIsRoot
```
public boolean isSftpUserDirIsRoot()
```

setSftpUserDirIsRoot

public void setSftpUserDirIsRoot(boolean sftpUserDirIsRoot)

getCmisRepositoryId
```
public String getCmisRepositoryId()
```
Gets CMIS repository ID.

Returns:

repository id

Since:

2.9.0

setCmisRepositoryId
```
public void setCmisRepositoryId(String cmisRepositoryId)
```
Sets CMIS repository ID.

Parameters:

cmisRepositoryId - repository ID

Since:

2.9.0

getCmisXmlTargetField
```
public String getCmisXmlTargetField()
```
Sets the name of the field where the raw XML obtained from the CMIS REST API will be stored. Default is null (does not store the raw XML).

Returns:

field name

Since:

2.9.0

setCmisXmlTargetField
```
public void setCmisXmlTargetField(String cmisXmlTargetField)
```
Sets the name of the field where the raw XML obtained from the CMIS REST API will be stored.

Parameters:

cmisXmlTargetField - target field

Since:

2.9.0

getFilesystemOptions
```
public org.apache.commons.vfs2.FileSystemOptions getFilesystemOptions(org.apache.commons.vfs2.FileObject fileObject)
```
Description copied from interface: IFilesystemOptionsProvider

Provide file system options associated with a file object.

Specified by:

getFilesystemOptions in interface IFilesystemOptionsProvider

Parameters:

fileObject - the file for which to get options

Returns:

options

loadFromXML
```
public void loadFromXML(Reader in)
                 throws IOException
```
Specified by:

loadFromXML in interface IXMLConfigurable

Throws:

IOException

saveToXML
```
public void saveToXML(Writer out)
               throws IOException
```
Specified by:

saveToXML in interface IXMLConfigurable

Throws:

IOException

equals
```
public boolean equals(Object other)
```
Overrides:

equals in class Object

hashCode
```
public int hashCode()
```
Overrides:

hashCode in class Object

toString
```
public String toString()
```
Overrides:

toString in class Object

Class GenericFilesystemOptionsProvider

Password encryption:

Available Options

XML configuration usage:

Usage example:

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

GenericFilesystemOptionsProvider

Method Detail

rebuildOptions

buildOptions

getAuthDomain

setAuthDomain

getAuthUsername

setAuthUsername

getAuthPassword

setAuthPassword

getAuthPasswordKey

setAuthPasswordKey

getFtpConnectTimeout