public interface IRobotsTxtProvider
Modifier and Type | Method and Description |
---|---|
RobotsTxt |
getRobotsTxt(HttpFetchClient fetchClient,
String url)
Gets robots.txt rules.
|
RobotsTxt getRobotsTxt(HttpFetchClient fetchClient, String url)
fetchClient
- http fetcher executor to grab robots.txturl
- the URL to derive the robots.txt fromCopyright © 2009–2023 Norconex Inc.. All rights reserved.