RegexLinkExtractor (Norconex HTTP Collector 2.9.1 API)

java.lang.Object
- com.norconex.collector.http.url.impl.RegexLinkExtractor

All Implemented Interfaces:

ILinkExtractor, IXMLConfigurable
```
public class RegexLinkExtractor
extends Object
implements ILinkExtractor, IXMLConfigurable
```
Link extractor using regular expressions to extract links found in text documents. Relative links are resolved to the document URL. For HTML documents, it is best advised to use the GenericLinkExtractor, which addresses many cases specific to HTML.

Applicable documents

By default, this extractor will extract URLs only in documents having their content type matching this regular expression:
```
 text/.*
 
```
You can specify your own content types or reference restriction patterns using setApplyToContentTypePattern(String) or setApplyToReferencePattern(String), but make sure they represent text files. When both methods are used, a document should be be matched by both to be accepted.

Referrer data

The following referrer information is stored as metadata in each document represented by the extracted URLs:
- Referrer reference: The reference (URL) of the page where the link to a document was found. Metadata value is HttpMetadata.COLLECTOR_REFERRER_REFERENCE.
Character encoding

This extractor will by default attempt to detect the encoding of the a page when extracting links and referrer information. If no charset could be detected, it falls back to UTF-8. It is also possible to dictate which encoding to use with setCharset(String).

XML configuration usage:
```
  <extractor class="com.norconex.collector.http.url.impl.RegexLinkExtractor"
          maxURLLength="(maximum URL length. Default is 2048)" 
          charset="(supported character encoding)" >
      <applyToContentTypePattern>
          (Regular expression matching content types this extractor 
           should apply to. Default accepts "text/.*".)
      </applyToContentTypePattern>
      <applyToReferencePattern>
          (Regular expression matching references this extractor should
           apply to. Default accepts all references.)
      </applyToReferencePattern>
      
      
      <linkExtractionPatterns>
          <pattern>
            <match>(regular expression)</match>
            <replace>(optional regex replacement)</replace>
          </pattern>
          
      </linkExtractionPatterns>
  </extractor>
 
```
Usage example:

The following extracts page "ids" contained in square brackets and add them to a custom URL.
```
  <extractor class="com.norconex.collector.http.url.impl.RegexLinkExtractor">
      <linkExtractionPatterns>
          <pattern>
              <match>\[(\d+)\]<match>
              <replace>http://www.example.com/page?id=$1<replace>
          </pattern>
      </linkExtractionPatterns>
  </extractor>
 
```
Since:

2.7.0

Author:

Pascal Essiembre

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`DEFAULT_CONTENT_TYPE_PATTERN`
`static int`	`DEFAULT_MAX_URL_LENGTH` Default maximum length a URL can have.
`static int`	`MAX_BUFFER_SIZE`
`static int`	`OVERLAP_SIZE`

Constructor Summary

Constructors
Constructor and Description

RegexLinkExtractor()

Constructors
Constructor and Description
`RegexLinkExtractor()`

Method Summary

All Methods Instance Methods Concrete Methods Deprecated Methods
Modifier and Type	Method and Description
`boolean`	`accepts(String url, ContentType contentType)` Whether this link extraction should be executed for the given URL and/or content type.
`void`	`addPattern(String pattern)`
`void`	`addPattern(String pattern, int matchGroup)` Deprecated. Since 2.8.0, use `addPattern(String, String)` instead.
`void`	`addPattern(String pattern, String replacement)` Adds a URL pattern, with an optional replacement.
`void`	`clearPatterns()`
`boolean`	`equals(Object other)`
`Set<Link>`	`extractLinks(InputStream input, String reference, ContentType contentType)` Extracts links from a document.
`String`	`getApplyToContentTypePattern()`
`String`	`getApplyToReferencePattern()`
`String`	`getCharset()` Gets the character set of pages on which link extraction is performed.
`int`	`getMaxURLLength()` Gets the maximum supported URL length.
`int`	`getPatternMatchGroup(String pattern)` Deprecated. Since 2.8.0, use #getPatternReplacement(String) instead. It will return the group id if the "replacement" value only contains a group replacement (e.g. $1), else, it will always return -1.
`String`	`getPatternReplacement(String pattern)` Gets a pattern replacement.
`List<String>`	`getPatterns()`
`int`	`hashCode()`
`void`	`loadFromXML(Reader in)`
`void`	`saveToXML(Writer out)`
`void`	`setApplyToContentTypePattern(String applyToContentTypePattern)`
`void`	`setApplyToReferencePattern(String applyToReferencePattern)`
`void`	`setCharset(String charset)` Sets the character set of pages on which link extraction is performed.
`void`	`setMaxURLLength(int maxURLLength)` Sets the maximum supported URL length.
`String`	`toString()`

Methods inherited from class java.lang.Object
clone, finalize, getClass, notify, notifyAll, wait, wait, wait

- Field Detail
  - DEFAULT_CONTENT_TYPE_PATTERN
```
public static final String DEFAULT_CONTENT_TYPE_PATTERN
```
    See Also:
    
    Constant Field Values
  - MAX_BUFFER_SIZE
```
public static final int MAX_BUFFER_SIZE
```
    See Also:
    
    Constant Field Values
  - OVERLAP_SIZE
```
public static final int OVERLAP_SIZE
```
    See Also:
    
    Constant Field Values
  - DEFAULT_MAX_URL_LENGTH
```
public static final int DEFAULT_MAX_URL_LENGTH
```
    Default maximum length a URL can have.
    
    See Also:
    
    Constant Field Values
- Constructor Detail
  - RegexLinkExtractor
```
public RegexLinkExtractor()
```
- Method Detail
  - extractLinks
```
public Set<Link> extractLinks(InputStream input,
                              String reference,
                              ContentType contentType)
                       throws IOException
```
    Description copied from interface: ILinkExtractor
    
    Extracts links from a document.
    
    Specified by:
    
    extractLinks in interface ILinkExtractor
    
    Parameters:
    
    input - the document input stream
    
    reference - document reference (URL)
    
    contentType - the document content type
    
    Returns:
    
    a set of links
    
    Throws:
    
    IOException - problem reading the document
  - accepts
```
public boolean accepts(String url,
                       ContentType contentType)
```
    Description copied from interface: ILinkExtractor
    
    Whether this link extraction should be executed for the given URL and/or content type.
    
    Specified by:
    
    accepts in interface ILinkExtractor
    
    Parameters:
    
    url - the url
    
    contentType - the content type
    
    Returns:
    
    true if the given URL is accepted
  - getMaxURLLength
```
public int getMaxURLLength()
```
    Gets the maximum supported URL length.
    
    Returns:
    
    maximum URL length
  - setMaxURLLength
```
public void setMaxURLLength(int maxURLLength)
```
    Sets the maximum supported URL length.
    
    Parameters:
    
    maxURLLength - maximum URL length
  - getCharset
```
public String getCharset()
```
    Gets the character set of pages on which link extraction is performed. Default is null (charset detection will be attempted).
    
    Returns:
    
    character set to use, or null
  - setCharset
```
public void setCharset(String charset)
```
    Sets the character set of pages on which link extraction is performed. Not specifying any (null) will attempt charset detection.
    
    Parameters:
    
    charset - character set to use, or null
  - getApplyToContentTypePattern
```
public String getApplyToContentTypePattern()
```
  - setApplyToContentTypePattern
```
public void setApplyToContentTypePattern(String applyToContentTypePattern)
```
  - getApplyToReferencePattern
```
public String getApplyToReferencePattern()
```
  - setApplyToReferencePattern
```
public void setApplyToReferencePattern(String applyToReferencePattern)
```
  - getPatterns
```
public List<String> getPatterns()
```
  - getPatternReplacement
```
public String getPatternReplacement(String pattern)
```
    Gets a pattern replacement.
    
    Parameters:
    
    pattern - the pattern for which to obtain its replacement
    
    Returns:
    
    pattern replacement or null (no replacement)
    
    Since:
    
    2.8.0
  - getPatternMatchGroup
```
@Deprecated
public int getPatternMatchGroup(String pattern)
```
    Deprecated. Since 2.8.0, use #getPatternReplacement(String) instead. It will return the group id if the "replacement" value only contains a group replacement (e.g. $1), else, it will always return -1.
    
    Parameters:
    
    pattern - the pattern for which to obtain its replacement
    
    Returns:
    
    the matching group.
  - clearPatterns
```
public void clearPatterns()
```
  - addPattern
```
public void addPattern(String pattern)
```
  - addPattern
```
public void addPattern(String pattern,
                       String replacement)
```
    Adds a URL pattern, with an optional replacement.
    
    Parameters:
    
    pattern - a regular expression
    
    replacement - a regular expression replacement
    
    Since:
    
    2.8.0
  - addPattern
```
@Deprecated
public void addPattern(String pattern,
                                    int matchGroup)
```
    Deprecated. Since 2.8.0, use addPattern(String, String) instead.
    
    Parameters:
    
    pattern - a regular expression
    
    matchGroup - regular expression match group
  - loadFromXML
```
public void loadFromXML(Reader in)
```
    Specified by:
    
    loadFromXML in interface IXMLConfigurable
  - saveToXML
```
public void saveToXML(Writer out)
               throws IOException
```
    Specified by:
    
    saveToXML in interface IXMLConfigurable
    
    Throws:
    
    IOException
  - toString
```
public String toString()
```
    Overrides:
    
    toString in class Object
  - equals
```
public boolean equals(Object other)
```
    Overrides:
    
    equals in class Object
  - hashCode
```
public int hashCode()
```
    Overrides:
    
    hashCode in class Object

Class RegexLinkExtractor

Applicable documents

Referrer data

Character encoding

XML configuration usage:

Usage example:

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

DEFAULT_CONTENT_TYPE_PATTERN

MAX_BUFFER_SIZE

OVERLAP_SIZE

DEFAULT_MAX_URL_LENGTH

Constructor Detail

RegexLinkExtractor

Method Detail

extractLinks

accepts

getMaxURLLength

setMaxURLLength

getCharset

setCharset

getApplyToContentTypePattern

setApplyToContentTypePattern

getApplyToReferencePattern

setApplyToReferencePattern

getPatterns

getPatternReplacement

getPatternMatchGroup

clearPatterns

addPattern

addPattern

addPattern

loadFromXML

saveToXML

toString

equals

hashCode