Online Internet Traffic Monitoring System Using Spark Streaming
Owing to the explosive growth of Internet traffic, network operators must be able to monitor the entire network situation and efficiently manage their network resources. Traditional network analysis methods that usually work on a single machine are no longer suitable for huge traffic data owing to t...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Tsinghua University Press
2018-03-01
|
| Series: | Big Data Mining and Analytics |
| Subjects: | |
| Online Access: | https://www.sciopen.com/article/10.26599/BDMA.2018.9020005 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832572845514817536 |
|---|---|
| author | Baojun Zhou Jie Li Xiaoyan Wang Yu Gu Li Xu Yongqiang Hu Lihua Zhu |
| author_facet | Baojun Zhou Jie Li Xiaoyan Wang Yu Gu Li Xu Yongqiang Hu Lihua Zhu |
| author_sort | Baojun Zhou |
| collection | DOAJ |
| description | Owing to the explosive growth of Internet traffic, network operators must be able to monitor the entire network situation and efficiently manage their network resources. Traditional network analysis methods that usually work on a single machine are no longer suitable for huge traffic data owing to their poor processing ability. Big data frameworks, such as Hadoop and Spark, can handle such analysis jobs even for a large amount of network traffic. However, Hadoop and Spark are inherently designed for offline data analysis. To cope with streaming data, various stream-processing-based frameworks have been proposed, such as Storm, Flink, and Spark Streaming. In this study, we propose an online Internet traffic monitoring system based on Spark Streaming. The system comprises three parts, namely, the collector, messaging system, and stream processor. We considered the TCP performance monitoring as a special use case of showing how network monitoring can be performed with our proposed system. We conducted typical experiments with a cluster in standalone mode, which showed that our system performs well for large Internet traffic measurement and monitoring. |
| format | Article |
| id | doaj-art-e6bfcbdec10b47ef945a0ba76f8e8db8 |
| institution | Kabale University |
| issn | 2096-0654 |
| language | English |
| publishDate | 2018-03-01 |
| publisher | Tsinghua University Press |
| record_format | Article |
| series | Big Data Mining and Analytics |
| spelling | doaj-art-e6bfcbdec10b47ef945a0ba76f8e8db82025-02-02T06:49:44ZengTsinghua University PressBig Data Mining and Analytics2096-06542018-03-0111475610.26599/BDMA.2018.9020005Online Internet Traffic Monitoring System Using Spark StreamingBaojun Zhou0Jie Li1Xiaoyan Wang2Yu Gu3Li Xu4Yongqiang Hu5Lihua Zhu6<institution content-type="dept">Department of Computer Science</institution>, <institution>University of Tsukuba</institution>, <city>Tsukuba</city> <postal-code>305-8577</postal-code>, <country>Japan</country>.<institution content-type="dept">Department of Computer Science</institution>, <institution>University of Tsukuba</institution>, <city>Tsukuba</city> <postal-code>305-8577</postal-code>, <country>Japan</country>.<institution content-type="dept">College of Engineering</institution>, <institution>Ibaraki University</institution>, <city>Hitachi</city> <postal-code>316-8511</postal-code>, <country>Japan</country>.<institution content-type="dept">School of Computer and Information</institution>, <institution>Hefei University of Technology</institution>, <city>Hefei</city> <postal-code>230601</postal-code>, <country>China</country>.<institution content-type="dept">College of Mathematics and Computer Science</institution>, <institution>Fujian Normal University</institution>, <city>Fuzhou</city> <postal-code>350007</postal-code>, <country>China</country>.<institution>Institute of Scientific and Technical Information of Qinghai</institution>, <city>Xining</city> <postal-code>810008</postal-code>, <country>China</country>.<institution>Institute of Scientific and Technical Information of Qinghai</institution>, <city>Xining</city> <postal-code>810008</postal-code>, <country>China</country>.Owing to the explosive growth of Internet traffic, network operators must be able to monitor the entire network situation and efficiently manage their network resources. Traditional network analysis methods that usually work on a single machine are no longer suitable for huge traffic data owing to their poor processing ability. Big data frameworks, such as Hadoop and Spark, can handle such analysis jobs even for a large amount of network traffic. However, Hadoop and Spark are inherently designed for offline data analysis. To cope with streaming data, various stream-processing-based frameworks have been proposed, such as Storm, Flink, and Spark Streaming. In this study, we propose an online Internet traffic monitoring system based on Spark Streaming. The system comprises three parts, namely, the collector, messaging system, and stream processor. We considered the TCP performance monitoring as a special use case of showing how network monitoring can be performed with our proposed system. We conducted typical experiments with a cluster in standalone mode, which showed that our system performs well for large Internet traffic measurement and monitoring.https://www.sciopen.com/article/10.26599/BDMA.2018.9020005spark streamingnetwork monitoringbig datatcp performance monitoring |
| spellingShingle | Baojun Zhou Jie Li Xiaoyan Wang Yu Gu Li Xu Yongqiang Hu Lihua Zhu Online Internet Traffic Monitoring System Using Spark Streaming Big Data Mining and Analytics spark streaming network monitoring big data tcp performance monitoring |
| title | Online Internet Traffic Monitoring System Using Spark Streaming |
| title_full | Online Internet Traffic Monitoring System Using Spark Streaming |
| title_fullStr | Online Internet Traffic Monitoring System Using Spark Streaming |
| title_full_unstemmed | Online Internet Traffic Monitoring System Using Spark Streaming |
| title_short | Online Internet Traffic Monitoring System Using Spark Streaming |
| title_sort | online internet traffic monitoring system using spark streaming |
| topic | spark streaming network monitoring big data tcp performance monitoring |
| url | https://www.sciopen.com/article/10.26599/BDMA.2018.9020005 |
| work_keys_str_mv | AT baojunzhou onlineinternettrafficmonitoringsystemusingsparkstreaming AT jieli onlineinternettrafficmonitoringsystemusingsparkstreaming AT xiaoyanwang onlineinternettrafficmonitoringsystemusingsparkstreaming AT yugu onlineinternettrafficmonitoringsystemusingsparkstreaming AT lixu onlineinternettrafficmonitoringsystemusingsparkstreaming AT yongqianghu onlineinternettrafficmonitoringsystemusingsparkstreaming AT lihuazhu onlineinternettrafficmonitoringsystemusingsparkstreaming |