Flow-network based auto rescale strategy for Flink
In order to solve the problem that the load of big data stream computing platform is increasing with fluctuation while the cluster was not able to rescale efficiently,the Flow-network based auto rescale strategy for Flink was proposed.Firstly,the flow-network model was set up and the capacity of eac...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Editorial Department of Journal on Communications
2019-08-01
|
| Series: | Tongxin xuebao |
| Subjects: | |
| Online Access: | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2019173/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841539337318039552 |
|---|---|
| author | Ziyang LI Jiong YU Chen BIAN Yitian ZHANG Yonglin PU Yuefei WANG Liang LU |
| author_facet | Ziyang LI Jiong YU Chen BIAN Yitian ZHANG Yonglin PU Yuefei WANG Liang LU |
| author_sort | Ziyang LI |
| collection | DOAJ |
| description | In order to solve the problem that the load of big data stream computing platform is increasing with fluctuation while the cluster was not able to rescale efficiently,the Flow-network based auto rescale strategy for Flink was proposed.Firstly,the flow-network model was set up and the capacity of each edge that was calculated by self-learning algorithm.Secondly,the bottleneck of the cluster was acquired by maximum-flow algorithm and the resource rescheduling plan was drawn up.Finally,the resource rescheduling plan was executed and the stateful data was migrated efficiently by the data migration algorithm based on the strategy of data partitioning by bulk and bucket.The experimental results show that the strategy can effectively provide performance promotion in the application with complex stateful data.It improved the throughput of the cluster and reduced the time overhead of the data migration on the premise of satisfying the latency constrain of the application,which means that the strategy promotes the scalability of the cluster efficiently. |
| format | Article |
| id | doaj-art-3527e96e8cb1411ea24cd7a61d373199 |
| institution | Kabale University |
| issn | 1000-436X |
| language | zho |
| publishDate | 2019-08-01 |
| publisher | Editorial Department of Journal on Communications |
| record_format | Article |
| series | Tongxin xuebao |
| spelling | doaj-art-3527e96e8cb1411ea24cd7a61d3731992025-01-14T07:17:31ZzhoEditorial Department of Journal on CommunicationsTongxin xuebao1000-436X2019-08-01408510159728960Flow-network based auto rescale strategy for FlinkZiyang LIJiong YUChen BIANYitian ZHANGYonglin PUYuefei WANGLiang LUIn order to solve the problem that the load of big data stream computing platform is increasing with fluctuation while the cluster was not able to rescale efficiently,the Flow-network based auto rescale strategy for Flink was proposed.Firstly,the flow-network model was set up and the capacity of each edge that was calculated by self-learning algorithm.Secondly,the bottleneck of the cluster was acquired by maximum-flow algorithm and the resource rescheduling plan was drawn up.Finally,the resource rescheduling plan was executed and the stateful data was migrated efficiently by the data migration algorithm based on the strategy of data partitioning by bulk and bucket.The experimental results show that the strategy can effectively provide performance promotion in the application with complex stateful data.It improved the throughput of the cluster and reduced the time overhead of the data migration on the premise of satisfying the latency constrain of the application,which means that the strategy promotes the scalability of the cluster efficiently.http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2019173/stream computingresource schedulingelastic clusterload migrationFlink |
| spellingShingle | Ziyang LI Jiong YU Chen BIAN Yitian ZHANG Yonglin PU Yuefei WANG Liang LU Flow-network based auto rescale strategy for Flink Tongxin xuebao stream computing resource scheduling elastic cluster load migration Flink |
| title | Flow-network based auto rescale strategy for Flink |
| title_full | Flow-network based auto rescale strategy for Flink |
| title_fullStr | Flow-network based auto rescale strategy for Flink |
| title_full_unstemmed | Flow-network based auto rescale strategy for Flink |
| title_short | Flow-network based auto rescale strategy for Flink |
| title_sort | flow network based auto rescale strategy for flink |
| topic | stream computing resource scheduling elastic cluster load migration Flink |
| url | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2019173/ |
| work_keys_str_mv | AT ziyangli flownetworkbasedautorescalestrategyforflink AT jiongyu flownetworkbasedautorescalestrategyforflink AT chenbian flownetworkbasedautorescalestrategyforflink AT yitianzhang flownetworkbasedautorescalestrategyforflink AT yonglinpu flownetworkbasedautorescalestrategyforflink AT yuefeiwang flownetworkbasedautorescalestrategyforflink AT lianglu flownetworkbasedautorescalestrategyforflink |