Interface IDelayResolver
-
- All Known Implementing Classes:
AbstractDelayResolver
,GenericDelayResolver
,ReferenceDelayResolver
public interface IDelayResolver
Resolves and creates intentional "delays" to increase document download time intervals. This interface does not dictate how delays are resolved. It is left to implementors to put in place their own strategy (e.g. pause all threads, delay multiple crawls on the same website domain only, etc). Try to be "nice" to the web sites you crawl.- Author:
- Pascal Essiembre
-
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description void
delay(RobotsTxt robotsTxt, String url)
Delay crawling activities (if applicable).
-