Package com.norconex.collector.fs.option.impl

Class GenericFilesystemOptionsProvider

java.lang.Object
com.norconex.collector.fs.option.impl.GenericFilesystemOptionsProvider
All Implemented Interfaces:
IFilesystemOptionsProvider, IXMLConfigurable, IXMLConfigurable
public class GenericFilesystemOptionsProvider extends Object implements IFilesystemOptionsProvider, IXMLConfigurable

Generic implementation of IFilesystemOptionsProvider. This provider makes configurable many Commons VFS options. The same FileSystemOptions instance is returned regardless of the FileObject supplied. In many case, it is not necessary to configure any file system options.

Password encryption:

It is usually recommended to provide credentials here instead of doing so directly on the requested URL to prevent password from appearing in log entries. The authPassword can take a password that has been encrypted using EncryptionUtil. In order for the password to be decrypted properly by the crawler, you need to specify the encryption key used to encrypt it. The key can be stored in a few supported locations and a combination of [auth|proxy]PasswordKey and [auth|proxy]PasswordKeySource must be specified to properly locate the key. The supported sources are:

Password key source options
[...]PasswordKeySource [...]PasswordKey
key The actual encryption key.
file Path to a file containing the encryption key.
environment Name of an environment variable containing the key.
property Name of a JVM system property containing the key.

Still, if you insist in having the password set on the URL, Apache Commons VFS offers a way to encrypt it there using their own EncryptUtil. More info under the "Naming" section here: http://commons.apache.org/proper/commons-vfs/filesystems.html

Available options

You will not find methods or XML configuration tags for all options supported by various file systems. If you need to configure more advanced options for your file system, consider having your own implementation of IFilesystemOptionsProvider or extend this class and implement the buildOptions(FileSystemOptions) method to set additional options.

As of 2.9.0, CMIS-enabled Content Management Systems (CMS) are supported (Atom end-point). The start path can be specified as: cmis-atom:http://yourhost:port/path/to/atom. Optionally you can have a non-root starting path by adding the path name to the base URL, with an exclamation mark as a separator: cmis-atom:http://yourhost:port/path/to/atom!/MyFolder/MySubFolder. Start paths are assumed to be Atom URLs.

As of 2.7.0, XML configuration entries expecting millisecond durations can be provided in human-readable format (English only), as per DurationParser (e.g., "5 minutes and 30 seconds" or "5m30s").

XML configuration usage:

  <optionsProvider class="com.norconex.collector.fs.option.impl.GenericFilesystemOptionsProvider">

      <!-- Authentication (any file system) -->
      <authDomain>...</authDomain>
      <authUsername>...</authUsername>
      <authPassword>...</authPassword>
      <!-- Use the following if password is encrypted. -->
      <authPasswordKey>(the encryption key or a reference to it)</authPasswordKey>
      <authPasswordKeySource>[key|file|environment|property]</authPasswordKeySource>

      <!-- FTP (for FTPS options set ftpSecure to "true") -->
      <ftpSecure>[false|true]</ftpSecure>
      <ftpConnectTimeout>(milliseconds)</ftpConnectTimeout>
      <ftpControlEncoding>...</ftpControlEncoding>
      <ftpDataTimeout>(milliseconds)</ftpDataTimeout>
      <ftpDefaultDateFormat>...</ftpDefaultDateFormat>
      <ftpFileType>[ASCII|BINARY|LOCAL|EBCDIC]</ftpFileType>
      <ftpPassiveMode>[false|true]</ftpPassiveMode>
      <ftpRecentDateFormat>...</ftpRecentDateFormat>
      <ftpRemoteVerification>[false|true]</ftpRemoteVerification>
      <ftpServerLanguageCode>...</ftpServerLanguageCode>
      <ftpServerTimeZoneId>...</ftpServerTimeZoneId>
      <ftpShortMonthNames>(comma-separated list)</ftpShortMonthNames>
      <ftpSoTimeout>(milliseconds)</ftpSoTimeout>
      <ftpUserDirIsRoot>[false|true]</ftpUserDirIsRoot>

      <!-- HDFS -->
      <hdfsConfigName>...</hdfsConfigName>
      <hdfsConfigPath>...</hdfsConfigPath>
      <hdfsConfigURL>...</hdfsConfigURL>

      <!-- HTTP (for Webdav options, set httpWebdav to "true") -->
      <httpWebdav>[false|true]</httpWebdav>
      <httpConnectionTimeout>(milliseconds)</httpConnectionTimeout>
      <httpFollowRedirect>[false|true]</httpFollowRedirect>
      <httpMaxConnectionsPerHost>...</httpMaxConnectionsPerHost>
      <httpMaxTotalConnections>...</httpMaxTotalConnections>
      <httpPreemptiveAuth>[false|true]</httpPreemptiveAuth>
      <httpSoTimeout>(milliseconds)</httpSoTimeout>
      <httpUrlCharset>...</httpUrlCharset>
      <httpUserAgent>...</httpUserAgent>
      <httpWebdavCreatorName>...</httpWebdavCreatorName>
      <httpWebdavVersioning>[false|true]</httpWebdavVersioning>

      <!-- RAM -->
      <ramMaxSize>(number of bytes)</ramMaxSize>

      <!-- SFTP -->
      <sftpCompression>...</sftpCompression>
      <sftpFileNameEncoding>...</sftpFileNameEncoding>
      <sftpKnownHosts>...</sftpKnownHosts>
      <sftpPreferredAuthentications>...</sftpPreferredAuthentications>
      <sftpStrictHostKeyChecking>[no|yes|ask]</sftpStrictHostKeyChecking>
      <sftpTimeout>(milliseconds)</sftpTimeout>
      <sftpUserDirIsRoot>[false|true]</sftpUserDirIsRoot>

      <!-- CMIS -->
      <cmisRepositoryId>
          (Optional repository ID, defaults to first one found.)
      </cmisRepositoryId>
      <cmisXmlTargetField>
          (Optional target field name where to store the raw CMIS REST API
           XML. Default does not store the raw XML in a field.)
      </cmisXmlTargetField>

  </optionsProvider>

Usage example:

The following sets the FTP settings sometimes required to get directory listings on remote servers. 

  <optionsProvider
      class="com.norconex.collector.fs.option.impl.GenericFilesystemOptionsProvider">
      <ftpPassiveMode>true</ftpPassiveMode>
      <ftpUserDirIsRoot>false</ftpUserDirIsRoot>
  </optionsProvider>
Since:
2.7.0
Author:
Pascal Essiembre

  • Constructor Details

    • GenericFilesystemOptionsProvider

      public GenericFilesystemOptionsProvider()

  • Method Details

    • rebuildOptions

      public void rebuildOptions()
      Rebuilds file system options. By default the options are reused. Calling this method asks this provider to rebuild the options before using them again.

    • buildOptions

      protected void buildOptions(org.apache.commons.vfs2.FileSystemOptions opts)
      For subclasses to apply more advanced configuration on file system options. Default implementation does nothing.
      Parameters:
      opts - file system options

    • getAuthDomain

      public String getAuthDomain()
      Gets the authentication domain.
      Returns:
      domain

    • setAuthDomain

      public void setAuthDomain(String authDomain)
      Sets the authentication domain.
      Parameters:
      authDomain - domain

    • getAuthUsername

      public String getAuthUsername()
      Gets the authentication username.
      Returns:
      username

    • setAuthUsername

      public void setAuthUsername(String authUsername)
      Sets the authentication username.
      Parameters:
      authUsername - username

    • getAuthPassword

      public String getAuthPassword()
      Gets the authentication password.
      Returns:
      the password

    • setAuthPassword

      public void setAuthPassword(String authPassword)
      Sets the authentication password.
      Parameters:
      authPassword - password

    • getAuthPasswordKey

      public EncryptionKey getAuthPasswordKey()
      Gets the authentication password encryption key.
      Returns:
      the password key or null if the password is not encrypted.
      See Also:

    • setAuthPasswordKey

      public void setAuthPasswordKey(EncryptionKey authPasswordKey)
      Sets the authentication password encryption key. Only required when the password is encrypted.
      Parameters:
      authPasswordKey - password key
      See Also:

    • getFtpConnectTimeout

      public Integer getFtpConnectTimeout()

    • setFtpConnectTimeout

      public void setFtpConnectTimeout(Integer ftpConnectTimeout)

    • getFtpControlEncoding

      public String getFtpControlEncoding()

    • setFtpControlEncoding

      public void setFtpControlEncoding(String ftpControlEncoding)

    • getFtpDataTimeout

      public Integer getFtpDataTimeout()

    • setFtpDataTimeout

      public void setFtpDataTimeout(Integer ftpDataTimeout)

    • getFtpDefaultDateFormat

      public String getFtpDefaultDateFormat()

    • setFtpDefaultDateFormat

      public void setFtpDefaultDateFormat(String ftpDefaultDateFormat)

    • getFtpFileType

      public org.apache.commons.vfs2.provider.ftp.FtpFileType getFtpFileType()

    • setFtpFileType

      public void setFtpFileType(org.apache.commons.vfs2.provider.ftp.FtpFileType ftpFileType)

    • isFtpPassiveMode

      public boolean isFtpPassiveMode()

    • setFtpPassiveMode

      public void setFtpPassiveMode(boolean ftpPassiveMode)

    • getFtpRecentDateFormat

      public String getFtpRecentDateFormat()

    • setFtpRecentDateFormat

      public void setFtpRecentDateFormat(String ftpRecentDateFormat)

    • isFtpRemoteVerification

      public boolean isFtpRemoteVerification()

    • setFtpRemoteVerification

      public void setFtpRemoteVerification(boolean ftpRemoteVerification)

    • getFtpServerLanguageCode

      public String getFtpServerLanguageCode()

    • setFtpServerLanguageCode

      public void setFtpServerLanguageCode(String ftpServerLanguageCode)

    • getFtpServerTimeZoneId

      public String getFtpServerTimeZoneId()

    • setFtpServerTimeZoneId

      public void setFtpServerTimeZoneId(String ftpServerTimeZoneId)

    • getFtpShortMonthNames

      public String[] getFtpShortMonthNames()

    • setFtpShortMonthNames

      public void setFtpShortMonthNames(String... ftpShortMonthNames)

    • getFtpSoTimeout

      public Integer getFtpSoTimeout()

    • setFtpSoTimeout

      public void setFtpSoTimeout(Integer ftpSoTimeout)

    • isFtpUserDirIsRoot

      public boolean isFtpUserDirIsRoot()

    • setFtpUserDirIsRoot

      public void setFtpUserDirIsRoot(boolean ftpUserDirIsRoot)

    • isFtpSecure

      public boolean isFtpSecure()

    • setFtpSecure

      public void setFtpSecure(boolean ftpSecure)

    • getHdfsConfigName

      public String getHdfsConfigName()

    • setHdfsConfigName

      public void setHdfsConfigName(String hdfsConfigName)

    • getHdfsConfigPath

      public org.apache.hadoop.fs.Path getHdfsConfigPath()

    • setHdfsConfigPath

      public void setHdfsConfigPath(org.apache.hadoop.fs.Path hdfsConfigPath)

    • getHdfsConfigURL

      public URL getHdfsConfigURL()

    • setHdfsConfigURL

      public void setHdfsConfigURL(URL hdfsConfigURL)

    • getHttpConnectionTimeout

      public int getHttpConnectionTimeout()

    • setHttpConnectionTimeout

      public void setHttpConnectionTimeout(int httpConnectionTimeout)

    • isHttpFollowRedirect

      public boolean isHttpFollowRedirect()

    • setHttpFollowRedirect

      public void setHttpFollowRedirect(boolean httpFollowRedirect)

    • getHttpMaxConnectionsPerHost

      public int getHttpMaxConnectionsPerHost()

    • setHttpMaxConnectionsPerHost

      public void setHttpMaxConnectionsPerHost(int httpMaxConnectionsPerHost)

    • getHttpMaxTotalConnections

      public int getHttpMaxTotalConnections()

    • setHttpMaxTotalConnections

      public void setHttpMaxTotalConnections(int httpMaxTotalConnections)

    • isHttpPreemptiveAuth

      public boolean isHttpPreemptiveAuth()

    • setHttpPreemptiveAuth

      public void setHttpPreemptiveAuth(boolean httpPreemptiveAuth)

    • getHttpSoTimeout

      public int getHttpSoTimeout()

    • setHttpSoTimeout

      public void setHttpSoTimeout(int httpSoTimeout)

    • getHttpUrlCharset

      public String getHttpUrlCharset()

    • setHttpUrlCharset

      public void setHttpUrlCharset(String httpUrlCharset)

    • getHttpUserAgent

      public String getHttpUserAgent()

    • setHttpUserAgent

      public void setHttpUserAgent(String httpUserAgent)

    • isHttpWebdav

      public boolean isHttpWebdav()

    • setHttpWebdav

      public void setHttpWebdav(boolean httpWebdav)

    • getHttpWebdavCreatorName

      public String getHttpWebdavCreatorName()

    • setHttpWebdavCreatorName

      public void setHttpWebdavCreatorName(String httpWebdavCreatorName)

    • isHttpWebdavVersioning

      public boolean isHttpWebdavVersioning()

    • setHttpWebdavVersioning

      public void setHttpWebdavVersioning(boolean httpWebdavVersioning)

    • getRamMaxSize

      public long getRamMaxSize()

    • setRamMaxSize

      public void setRamMaxSize(long ramMaxSize)

    • getSftpCompression

      public String getSftpCompression()

    • setSftpCompression

      public void setSftpCompression(String sftpCompression)

    • getSftpFileNameEncoding

      public String getSftpFileNameEncoding()

    • setSftpFileNameEncoding

      public void setSftpFileNameEncoding(String sftpFileNameEncoding)

    • getSftpKnownHosts

      public File getSftpKnownHosts()

    • setSftpKnownHosts

      public void setSftpKnownHosts(File sftpKnownHosts)

    • getSftpPreferredAuthentications

      public String getSftpPreferredAuthentications()

    • setSftpPreferredAuthentications

      public void setSftpPreferredAuthentications(String sftpPreferredAuthentications)

    • getSftpStrictHostKeyChecking

      public String getSftpStrictHostKeyChecking()

    • setSftpStrictHostKeyChecking

      public void setSftpStrictHostKeyChecking(String sftpStrictHostKeyChecking)

    • getSftpTimeout

      public int getSftpTimeout()

    • setSftpTimeout

      public void setSftpTimeout(int sftpTimeout)

    • isSftpUserDirIsRoot

      public boolean isSftpUserDirIsRoot()

    • setSftpUserDirIsRoot

      public void setSftpUserDirIsRoot(boolean sftpUserDirIsRoot)

    • getCmisRepositoryId

      public String getCmisRepositoryId()
      Gets CMIS repository ID.
      Returns:
      repository id
      Since:
      2.9.0

    • setCmisRepositoryId

      public void setCmisRepositoryId(String cmisRepositoryId)
      Sets CMIS repository ID.
      Parameters:
      cmisRepositoryId - repository ID
      Since:
      2.9.0

    • getCmisXmlTargetField

      public String getCmisXmlTargetField()
      Sets the name of the field where the raw XML obtained from the CMIS REST API will be stored. Default is null (does not store the raw XML).
      Returns:
      field name
      Since:
      2.9.0

    • setCmisXmlTargetField

      public void setCmisXmlTargetField(String cmisXmlTargetField)
      Sets the name of the field where the raw XML obtained from the CMIS REST API will be stored.
      Parameters:
      cmisXmlTargetField - target field
      Since:
      2.9.0

    • getFilesystemOptions

      public org.apache.commons.vfs2.FileSystemOptions getFilesystemOptions(org.apache.commons.vfs2.FileObject fileObject)
      Description copied from interface: IFilesystemOptionsProvider
      Provide file system options associated with a file object.
      Specified by:
      getFilesystemOptions in interface IFilesystemOptionsProvider
      Parameters:
      fileObject - the file for which to get options
      Returns:
      options

    • loadFromXML

      public void loadFromXML(Reader in) throws IOException
      Specified by:
      loadFromXML in interface IXMLConfigurable
      Throws:
      IOException

    • saveToXML

      public void saveToXML(Writer out) throws IOException
      Specified by:
      saveToXML in interface IXMLConfigurable
      Throws:
      IOException

    • equals

      public boolean equals(Object other)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • toString

      public String toString()
      Overrides:
      toString in class Object