Research on a real-time receiving scheme of streaming data
Discussing the common scenarios in modern data warehouse systems that need to receive a large amount of streaming data, connect it with the existing data on the disk, and then store it in the warehouse.By rationally setting disk paging and applying cache modules to disperse the disk I/O pressure, a...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | zho |
| Published: |
Editorial Department of Journal on Communications
2022-04-01
|
| Series: | Tongxin xuebao |
| Subjects: | |
| Online Access: | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2022080/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1841540000621002752 |
|---|---|
| author | Xiaoyan ZHANG Zhihao LIU Xiaofeng DU Tianbo LU |
| author_facet | Xiaoyan ZHANG Zhihao LIU Xiaofeng DU Tianbo LU |
| author_sort | Xiaoyan ZHANG |
| collection | DOAJ |
| description | Discussing the common scenarios in modern data warehouse systems that need to receive a large amount of streaming data, connect it with the existing data on the disk, and then store it in the warehouse.By rationally setting disk paging and applying cache modules to disperse the disk I/O pressure, a more efficient data receiving scheme was proposed based on the existing research, and a consistent Hash function was introduced and extended to distributed environment and a D-CACHEJOIN algorithm applied to distributed environment was proposed.The cost model of the algorithm was calculated by theory and simulation experiment was performed using data that obey the Zipfian distribution.The experiment results show that the proposed algorithm has higher efficiency than existing algorithms in practical application scenarios close to reality, and can be quickly and easily extended to distributed environments. |
| format | Article |
| id | doaj-art-e6cbf48007804821b34543003a7c6ea0 |
| institution | Kabale University |
| issn | 1000-436X |
| language | zho |
| publishDate | 2022-04-01 |
| publisher | Editorial Department of Journal on Communications |
| record_format | Article |
| series | Tongxin xuebao |
| spelling | doaj-art-e6cbf48007804821b34543003a7c6ea02025-01-14T06:30:19ZzhoEditorial Department of Journal on CommunicationsTongxin xuebao1000-436X2022-04-014315416359396932Research on a real-time receiving scheme of streaming dataXiaoyan ZHANGZhihao LIUXiaofeng DUTianbo LUDiscussing the common scenarios in modern data warehouse systems that need to receive a large amount of streaming data, connect it with the existing data on the disk, and then store it in the warehouse.By rationally setting disk paging and applying cache modules to disperse the disk I/O pressure, a more efficient data receiving scheme was proposed based on the existing research, and a consistent Hash function was introduced and extended to distributed environment and a D-CACHEJOIN algorithm applied to distributed environment was proposed.The cost model of the algorithm was calculated by theory and simulation experiment was performed using data that obey the Zipfian distribution.The experiment results show that the proposed algorithm has higher efficiency than existing algorithms in practical application scenarios close to reality, and can be quickly and easily extended to distributed environments.http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2022080/streaming datacachedistributed systemconsistent Hash function |
| spellingShingle | Xiaoyan ZHANG Zhihao LIU Xiaofeng DU Tianbo LU Research on a real-time receiving scheme of streaming data Tongxin xuebao streaming data cache distributed system consistent Hash function |
| title | Research on a real-time receiving scheme of streaming data |
| title_full | Research on a real-time receiving scheme of streaming data |
| title_fullStr | Research on a real-time receiving scheme of streaming data |
| title_full_unstemmed | Research on a real-time receiving scheme of streaming data |
| title_short | Research on a real-time receiving scheme of streaming data |
| title_sort | research on a real time receiving scheme of streaming data |
| topic | streaming data cache distributed system consistent Hash function |
| url | http://www.joconline.com.cn/zh/article/doi/10.11959/j.issn.1000-436x.2022080/ |
| work_keys_str_mv | AT xiaoyanzhang researchonarealtimereceivingschemeofstreamingdata AT zhihaoliu researchonarealtimereceivingschemeofstreamingdata AT xiaofengdu researchonarealtimereceivingschemeofstreamingdata AT tianbolu researchonarealtimereceivingschemeofstreamingdata |