Progressive filling partitioning and mapping algorithm for Spark based on allocation fitness degree

The job execution mechanism of Spark was analyzed,task efficiency model and Shuffle model were established,then allocation fitness degree (AFD) was defined and the optimization goal was put forward.On the basis of the model definition,the progressive filling partitioning and mapping algorithm (PFPM)...

Full description

Saved in:
Bibliographic Details
Main Authors: Chen BIAN, Jiong1 YU, Wei-rong XIU, Bin LIAO, Chang-tian YING, Yu-rong QIAN
Format: Article
Language:zho
Published: Editorial Department of Journal on Communications 2017-09-01
Series:Tongxin xuebao
Subjects:
Online Access:http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2017188/
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The job execution mechanism of Spark was analyzed,task efficiency model and Shuffle model were established,then allocation fitness degree (AFD) was defined and the optimization goal was put forward.On the basis of the model definition,the progressive filling partitioning and mapping algorithm (PFPM) was proposed.PFPM established the data distribution scheme adapting Reducers’ computing ability to decrease synchronous latency during Shuffle process and increase cluster the computing efficiency.The experiments demonstrate that PFPM could improve the rationality of workload distribution in Shuffle and optimize the execution efficiency of Spark.
ISSN:1000-436X