Can I read Design of a Crawler with Bounded Bandwidth on EtoBox?

Design of a Crawler with Bounded Bandwidth by Michelangelo Diligenti; Marco Maggini; Filippo Maria Pucci; Franco Scarselli is a scholarly article available to read on EtoBox.

What is Design of a Crawler with Bounded Bandwidth about?

This paper presents an algorithm to bound the bandwidth of a Web crawler. The crawler collects statistics on the transfer rate of each server to predict the expected bandwidth use for future downloads. The prediction allows us to activate the optimal number of fetcher threads in order to exploit the assigned bandwidth. The experimental results show the effectiveness of the proposed technique.

Author

Michelangelo Diligenti; Marco Maggini; Filippo Maria Pucci; Franco Scarselli

Publisher

ACM Press

Published

2004

Language

EN