Package com.norconex.collector.http.robot.impl
package com.norconex.collector.http.robot.impl
-
ClassesClassDescriptionImplementation of
IRobotsMetaProvideras per X-Robots-Tag and ROBOTS standards.Implementation ofIRobotsTxtProvideras per the robots.txt standard described at http://www.robotstxt.org/robotstxt.html.