Georgia Tech: Networking & Telecommunications Group


 
Title Data Streaming Algorithms for Estimating Entropy of Network Traffic
Speaker Ashwin Lall
Abstract

Using entropy of traffic distributions has been shown to aid a wide variety of network monitoring applications such as anomaly detection, clustering to reveal interesting patterns, and traffic classification. However, realizing this potential benefit in practice requires accurate algorithms that can operate on high-speed links, with low CPU and memory requirements. Estimating the entropy in a streaming model to enable such fine-grained traffic analysis is a challenging problem. We give lower bounds for this problem, showing that neither approximation nor randomization alone will let us compute the entropy efficiently. We then present two algorithms for randomly approximating the entropy in a time- and space-efficient manner, applicable for use on very high speed (greater than OC-48) links. Our first algorithm for entropy estimation, inspired by the seminal work of Alon et al. for estimating frequency moments, has strong theoretical guarantees on the error and resource usage. Our second algorithm utilizes the observation that the efficiency can be substantially enhanced by separating the high-frequency items (or elephants), from the low-frequency items (or mice).

Bio

Ashwin Lall is a fourth year Ph.D. student at the University of Rochester. His research interests include streaming algorithms (both from a networking and a theoretical perspective), sublinear space algorithms, and performance modeling and simulation.