Do you sometimes wonder what your crawler is doing?
Knowing more about the sequence of events taking place can help you
better configure your crawling solution.
The following flowcharts detail how each URL encountered is processed.
While they do not cover all available features, they should give
you a better idea of what's going on.
The upper flowchart shows how URLs are "prepared" before being queued for
processing by the next available thread. The second flowchart shows what
happens when a thread gets the next URL from the queue and
"processes" it.
Click on a shape to get related information and links to more documentation.