NetQuest: Scalable and Flexible Internet Measurement

The Internet connects over 200 million hosts and nearly a billion users worldwide. How to better understand and thereby control this enormous, decentralized, and constantly evolving infrastructure is a major challenge faced by today's researchers, engineers, and network operators. As an essential means of achieving such better understanding, network measurement is becoming crucial to a variety of existing and emerging network applications, such as IP performance management, fault diagnosis, traffic engineering, content distribution, overlay routing, multihoming, and peer-to-peer applications. These applications often require the ability to continuously monitor the current network state and react to changes in a timely fashion. Despite considerable recent progress in network measurement, there are still several gaps towards fulfilling such requirements.

First, existing network monitoring techniques often focus on accurately measuring the properties of individual links or paths and therefore lack scalability. They rarely scale beyond 100 nodes. In contrast, large-scale network management applications often require the ability to monitor the whole network which may contain orders of magnitude more nodes. Second, robustness issues have not been sufficiently investigated in the context of large-scale network monitoring and inference. Large-scale Internet measurements are inherently noisy due to failures, measurement artifacts, missing data, and even malicious behaviors. How to draw meaningful conclusions based on such noisy data remains a challenging problem that must be addressed in order to make large-scale network monitoring useful in practice. Third, existing techniques are often tailored to monitor specific types of network properties, and thus lack the flexibility to accommodate applications with even slightly different requirements and different performance metrics.

Our research aims to address the above challenges, and develop innovative techniques to design measurement experiments, conduct measurement, and analyze the data. We build our research on top of solid theoretical foundation in statistics, information theory, and optimization, and also leverage considerable practical engineering experience in measuring and managing large operational IP networks. 

People

Faculty

• Lili Qiu
• Yin Zhang

Students

• Han Hee Song

Papers

• Han Hee Song, Lili Qiu, and Yin Zhang. NetQuest: A Flexible Framework for Large-Scale Network Measurement. In Proc. of ACM SIGMETRICS, June 2006.
• Han Hee Song, Praveen Yalagandula, Lili Qiu and Yin Zhang, Real-time End-to-end Network Monitoring in Large Distributed Systems, to appear in Proceedings of the 2nd International Conference on COMmunication Systems softWAre and middlewaRE (COMSWARE 2007) (invited paper), Bangalore, India, January 2007.
• Yin Zhang, Zihui Ge, Matthew Roughan and Albert Greenberg, Network Anomography , in Proceedings of the Internet Measurement Conference (IMC '05), Berkeley, CA, USA, October 2005.
• Venkata N. Padmanabhan, Lili Qiu, and Helen Wang. Server-based Inference of Internet Performance. In Proc. of IEEE INFOCOM, San Francisco, CA, March 2003.