GenericDocumentFetcher (Norconex HTTP Collector 2.9.1 API)

java.lang.Object
- com.norconex.collector.http.fetch.impl.GenericDocumentFetcher

All Implemented Interfaces:

IHttpDocumentFetcher, IXMLConfigurable
```
public class GenericDocumentFetcher
extends Object
implements IHttpDocumentFetcher, IXMLConfigurable
```
Default implementation of IHttpDocumentFetcher.

Content type and character encoding

The default behavior of the HTTP Collector to identify the content type and character encoding of a document is to rely on the "Content-Type" HTTP response header. Web servers can sometimes return invalid or missing content type and character encoding information. Since 2.7.0, you can optionally decide not to trust web servers HTTP responses and have the collector perform its own content type and encoding detection. Such detection can be enabled with setDetectContentType(boolean) and setDetectCharset(boolean).

XML configuration usage:
```
  <documentFetcher
      class="com.norconex.collector.http.fetch.impl.GenericDocumentFetcher"
      detectContentType="[false|true]" detectCharset="[false|true]">
    <validStatusCodes>(defaults to 200)</validStatusCodes>
    <notFoundStatusCodes>(defaults to 404)</notFoundStatusCodes>
    <headersPrefix>(string to prefix headers)</headersPrefix>
  </documentFetcher>
 
```
The "validStatusCodes" and "notFoundStatusCodes" elements expect a coma-separated list of HTTP response code. If a code is added in both elements, the valid list takes precedence.

The "notFoundStatusCodes" element was added in 2.2.0.

Usage example:

The following configures the document fetcher to not trust HTTP response headers to identify the content type and encoding, but try to detect them instead.
```
  <documentFetcher detectContentType="true" detectCharset="true"/>
 
```
Author:

Pascal Essiembre

Constructor Summary

Constructors
Constructor and Description

GenericDocumentFetcher()

GenericDocumentFetcher(int[] validStatusCodes)

Constructors
Constructor and Description
`GenericDocumentFetcher()`
`GenericDocumentFetcher(int[] validStatusCodes)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected org.apache.http.client.methods.HttpRequestBase`	`createUriRequest(HttpDocument doc)` Creates the HTTP request to be executed.
`boolean`	`equals(Object other)`
`HttpFetchResponse`	`fetchDocument(org.apache.http.client.HttpClient httpClient, HttpDocument doc)` Fetches HTTP document and saves it to a local file
`String`	`getHeadersPrefix()`
`int[]`	`getNotFoundStatusCodes()` Gets HTTP status codes to be considered as "Not found" state.
`int[]`	`getValidStatusCodes()`
`int`	`hashCode()`
`boolean`	`isDetectCharset()` Gets whether character encoding is detected instead of relying on HTTP response header.
`boolean`	`isDetectContentType()` Gets whether content type is detected instead of relying on HTTP response header.
`void`	`loadFromXML(Reader in)`
`void`	`saveToXML(Writer out)`
`void`	`setDetectCharset(boolean detectCharset)` Sets whether character encoding is detected instead of relying on HTTP response header.
`void`	`setDetectContentType(boolean detectContentType)` Sets whether content type is detected instead of relying on HTTP response header.
`void`	`setHeadersPrefix(String headersPrefix)`
`void`	`setNotFoundStatusCodes(int... notFoundStatusCodes)` Sets HTTP status codes to be considered as "Not found" state.
`void`	`setValidStatusCodes(int... validStatusCodes)`
`String`	`toString()`

Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

- Constructor Detail
  - GenericDocumentFetcher
```
public GenericDocumentFetcher()
```
  - GenericDocumentFetcher
```
public GenericDocumentFetcher(int[] validStatusCodes)
```
- Method Detail
  - getValidStatusCodes
```
public int[] getValidStatusCodes()
```
  - setValidStatusCodes
```
public final void setValidStatusCodes(int... validStatusCodes)
```
  - getNotFoundStatusCodes
```
public int[] getNotFoundStatusCodes()
```
    Gets HTTP status codes to be considered as "Not found" state. Default is 404.
    
    Returns:
    
    "Not found" codes
    
    Since:
    
    2.2.0
  - setNotFoundStatusCodes
```
public final void setNotFoundStatusCodes(int... notFoundStatusCodes)
```
    Sets HTTP status codes to be considered as "Not found" state.
    
    Parameters:
    
    notFoundStatusCodes - "Not found" codes
    
    Since:
    
    2.2.0
  - getHeadersPrefix
```
public String getHeadersPrefix()
```
  - setHeadersPrefix
```
public void setHeadersPrefix(String headersPrefix)
```
  - isDetectContentType
```
public boolean isDetectContentType()
```
    Gets whether content type is detected instead of relying on HTTP response header.
    
    Returns:
    
    true to enable detection
    
    Since:
    
    2.7.0
  - setDetectContentType
```
public void setDetectContentType(boolean detectContentType)
```
    Sets whether content type is detected instead of relying on HTTP response header.
    
    Parameters:
    
    detectContentType - true to enable detection
    
    Since:
    
    2.7.0
  - isDetectCharset
```
public boolean isDetectCharset()
```
    Gets whether character encoding is detected instead of relying on HTTP response header.
    
    Returns:
    
    true to enable detection
    
    Since:
    
    2.7.0
  - setDetectCharset
```
public void setDetectCharset(boolean detectCharset)
```
    Sets whether character encoding is detected instead of relying on HTTP response header.
    
    Parameters:
    
    detectCharset - true to enable detection
    
    Since:
    
    2.7.0
  - fetchDocument
```
public HttpFetchResponse fetchDocument(org.apache.http.client.HttpClient httpClient,
                                       HttpDocument doc)
```
    Description copied from interface: IHttpDocumentFetcher
    
    Fetches HTTP document and saves it to a local file
    
    Specified by:
    
    fetchDocument in interface IHttpDocumentFetcher
    
    Parameters:
    
    httpClient - the HTTP client
    
    doc - the document to fetch and save
    
    Returns:
    
    fetch response
  - createUriRequest
```
protected org.apache.http.client.methods.HttpRequestBase createUriRequest(HttpDocument doc)
```
    Creates the HTTP request to be executed. Default implementation returns an HttpGet request around the document reference. This method can be overwritten to return another type of request, add HTTP headers, etc.
    
    Parameters:
    
    doc - document to fetch
    
    Returns:
    
    HTTP request
  - loadFromXML
```
public void loadFromXML(Reader in)
```
    Specified by:
    
    loadFromXML in interface IXMLConfigurable
  - saveToXML
```
public void saveToXML(Writer out)
               throws IOException
```
    Specified by:
    
    saveToXML in interface IXMLConfigurable
    
    Throws:
    
    IOException
  - equals
```
public boolean equals(Object other)
```
    Overrides:
    
    equals in class Object
  - hashCode
```
public int hashCode()
```
    Overrides:
    
    hashCode in class Object
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object

Class GenericDocumentFetcher

Content type and character encoding

XML configuration usage:

Usage example:

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Constructor Detail

GenericDocumentFetcher

GenericDocumentFetcher

Method Detail

getValidStatusCodes

setValidStatusCodes

getNotFoundStatusCodes

setNotFoundStatusCodes

getHeadersPrefix

setHeadersPrefix

isDetectContentType

setDetectContentType

isDetectCharset

setDetectCharset

fetchDocument

createUriRequest

loadFromXML

saveToXML

equals

hashCode

toString