Network Data Streaming: Algorithms for Network Measurement and Monitoring

High-speed networks;Network algorithms;Expectation maximization;Data streaming;Network monitoring;Computer networks

Kumar, Abhishek ; Computing

University:Georgia Institute of Technology

Department:Computing

关键词: High-speed networks;    Network algorithms;    Expectation maximization;    Data streaming;    Network monitoring;    Computer networks;   

Others  :  https://smartech.gatech.edu/bitstream/1853/7516/1/kumar_abhishek_200512_phd.pdf

美国|英语

来源: SMARTech Repository

【 摘 要 】

With the emergence of computer networks as one of the primary modes ofcommunication, and with their adoption for an increasingly wide rangeof applications, there is a growing need to understand andcharacterize the traffic they carry. The rise of large scalenetwork attacks adds urgency to this need. However, the large size,high speed and increasing complexity of these networks imply thattracking and characterizing the traffic they carry is an increasinglydifficult problem. Dealing with higher level aggregates, such as flowsinstead of packets, does not solve the problem because theseaggregates tend to be quite numerous and exhibit dynamics of theirown. In this thesis, we investigate a novel approach to deal with theimmense amounts of data associated with problems in networkmeasurement and monitoring.Building upon the paradigm of DataStreaming, which processes a large stream of data using a smallworking memory to answer a class of queries, we develop anarchitecture for Network Data Streaming that can accommodateadditional constraints imposed in the context of network monitoring.Using this architecture, we design algorithms for monitoringproperties of network traffic that have traditionally been consideredtoo difficult to monitor at high speed network links and routers. Ourfirst algorithm provides the ability to accurately estimate the sizeof individual flows. A second algorithm to estimate the distribution offlow sizes enables network operators to monitor anomalies in thetraffic. Incorporating the use of packet sampling, we can extend thelatter algorithm to estimate the flow size distribution of arbitrarysubpopulations. Finally, we apply the tools of Network Data Streaming to the operationof packet sampling itself. Using the ability to efficiently estimateflow-statistics such as approximate per-flow size, we design a familyof mechanisms where the sampling decision is guided by this knowledge.The individual solutions developed in this thesis share a commonarchitectural theme, supporting the monitoring of highly dynamicpopulations. Integrating this with the traditional sampling basedframework for network monitoring will enable a broad range ofapplications for accurate and comprehensive monitoring of networktraffic.