In-Memory Versus Disk-Based Computing with Random Forest for Stock Analysis: A Comparative Study
Background: The advancement of big data analytics calls for careful selection of processing frameworks to optimize machine learning effectiveness. Choosing the appropriate framework can significantly influence the speed and accuracy of data analysis, ultimately leading to more informed decision maki...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Prague University of Economics and Business
2025-08-01
|
| Series: | Acta Informatica Pragensia |
| Online Access: | https://aip.vse.cz/artkey/aip-202503-0010_in-memory-versus-disk-based-computing-with-random-forest-for-stock-analysis-a-comparative-study.php |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849338729195372544 |
|---|---|
| author | Chitra Joshi Chitrakant Banchorr Omkaresh Kulkarni Kirti Wanjale |
| author_facet | Chitra Joshi Chitrakant Banchorr Omkaresh Kulkarni Kirti Wanjale |
| author_sort | Chitra Joshi |
| collection | DOAJ |
| description | Background: The advancement of big data analytics calls for careful selection of processing frameworks to optimize machine learning effectiveness. Choosing the appropriate framework can significantly influence the speed and accuracy of data analysis, ultimately leading to more informed decision making. In adapting to this changing landscape, businesses should focus on factors such as how well a system scales, how easily it can be used and how effectively it integrates with their existing tools. The effectiveness of these frameworks plays a crucial role in determining data processing speed, model training efficiency and predictive accuracy. As data become increasingly large, diverse and fast-moving, conventional processing systems often fall short of the performance required for modern analytics.Objective: This research seeks to thoroughly assess the performance of two prominent big data processing frameworks-Apache Spark (in-memory computing) and MapReduce (disk-based computing)-with a focus on applying random forest algorithms to predict stock prices. The primary objective is to assess and compare their effectiveness in handling large-scale financial datasets, focusing on key aspects such as predictive accuracy, processing speed and scalability.Methods: The investigation uses the MapReduce methodology and Apache Spark independently to analyse a substantial stock price dataset and to train a random forest regressor. Mean squared error (MSE) and root mean square error (RMSE) were employed to assess the primary performance indicators of the models, while mean absolute error (MAE) and the R-squared value were used to evaluate the goodness of fit of the models.Results: The RMSE, MAE and MSE obtained for the Spark-based implementation were lower, compared to the MapReduce-based implementation, although these low values indicate high prediction accuracy. It also had a big impact on the time it took to train and run models because of its optimized in-memory processing. As opposed to this, the MapReduce approach had higher latency and lower accuracy, reflecting its disk-based constraints and reduced efficiency for iterative machine learning tasks.Conclusion: The conclusion supports the fact that Spark is the better option for complex machine learning tasks such as stock price prediction, as it is good for handling large amounts of data. MapReduce is still a reliable framework but not fast enough to process and not lightweight enough for analytics that are too rapid and iterative. The outcomes of this study are helpful for data scientists and financial analysts to choose the most appropriate framework for big data machine learning applications. |
| format | Article |
| id | doaj-art-e68b1325c23c4eb4a22f44c7d3c30741 |
| institution | Kabale University |
| issn | 1805-4951 |
| language | English |
| publishDate | 2025-08-01 |
| publisher | Prague University of Economics and Business |
| record_format | Article |
| series | Acta Informatica Pragensia |
| spelling | doaj-art-e68b1325c23c4eb4a22f44c7d3c307412025-08-20T03:44:19ZengPrague University of Economics and BusinessActa Informatica Pragensia1805-49512025-08-0114346047310.18267/j.aip.275aip-202503-0010In-Memory Versus Disk-Based Computing with Random Forest for Stock Analysis: A Comparative StudyChitra Joshi0https://orcid.org/0009-0000-3643-5557Chitrakant Banchorr1https://orcid.org/0000-0002-7287-0801Omkaresh Kulkarni2https://orcid.org/0000-0003-4775-1437Kirti Wanjale3https://orcid.org/0000-0003-4271-504XVishwakarma Institute of Information Technology, Pune, IndiaVishwakarma Institute of Information Technology, Pune, IndiaVishwakarma Institute of Information Technology, Pune, IndiaVishwakarma Institute of Information Technology, Pune, IndiaBackground: The advancement of big data analytics calls for careful selection of processing frameworks to optimize machine learning effectiveness. Choosing the appropriate framework can significantly influence the speed and accuracy of data analysis, ultimately leading to more informed decision making. In adapting to this changing landscape, businesses should focus on factors such as how well a system scales, how easily it can be used and how effectively it integrates with their existing tools. The effectiveness of these frameworks plays a crucial role in determining data processing speed, model training efficiency and predictive accuracy. As data become increasingly large, diverse and fast-moving, conventional processing systems often fall short of the performance required for modern analytics.Objective: This research seeks to thoroughly assess the performance of two prominent big data processing frameworks-Apache Spark (in-memory computing) and MapReduce (disk-based computing)-with a focus on applying random forest algorithms to predict stock prices. The primary objective is to assess and compare their effectiveness in handling large-scale financial datasets, focusing on key aspects such as predictive accuracy, processing speed and scalability.Methods: The investigation uses the MapReduce methodology and Apache Spark independently to analyse a substantial stock price dataset and to train a random forest regressor. Mean squared error (MSE) and root mean square error (RMSE) were employed to assess the primary performance indicators of the models, while mean absolute error (MAE) and the R-squared value were used to evaluate the goodness of fit of the models.Results: The RMSE, MAE and MSE obtained for the Spark-based implementation were lower, compared to the MapReduce-based implementation, although these low values indicate high prediction accuracy. It also had a big impact on the time it took to train and run models because of its optimized in-memory processing. As opposed to this, the MapReduce approach had higher latency and lower accuracy, reflecting its disk-based constraints and reduced efficiency for iterative machine learning tasks.Conclusion: The conclusion supports the fact that Spark is the better option for complex machine learning tasks such as stock price prediction, as it is good for handling large amounts of data. MapReduce is still a reliable framework but not fast enough to process and not lightweight enough for analytics that are too rapid and iterative. The outcomes of this study are helpful for data scientists and financial analysts to choose the most appropriate framework for big data machine learning applications.https://aip.vse.cz/artkey/aip-202503-0010_in-memory-versus-disk-based-computing-with-random-forest-for-stock-analysis-a-comparative-study.php |
| spellingShingle | Chitra Joshi Chitrakant Banchorr Omkaresh Kulkarni Kirti Wanjale In-Memory Versus Disk-Based Computing with Random Forest for Stock Analysis: A Comparative Study Acta Informatica Pragensia |
| title | In-Memory Versus Disk-Based Computing with Random Forest for Stock Analysis: A Comparative Study |
| title_full | In-Memory Versus Disk-Based Computing with Random Forest for Stock Analysis: A Comparative Study |
| title_fullStr | In-Memory Versus Disk-Based Computing with Random Forest for Stock Analysis: A Comparative Study |
| title_full_unstemmed | In-Memory Versus Disk-Based Computing with Random Forest for Stock Analysis: A Comparative Study |
| title_short | In-Memory Versus Disk-Based Computing with Random Forest for Stock Analysis: A Comparative Study |
| title_sort | in memory versus disk based computing with random forest for stock analysis a comparative study |
| url | https://aip.vse.cz/artkey/aip-202503-0010_in-memory-versus-disk-based-computing-with-random-forest-for-stock-analysis-a-comparative-study.php |
| work_keys_str_mv | AT chitrajoshi inmemoryversusdiskbasedcomputingwithrandomforestforstockanalysisacomparativestudy AT chitrakantbanchorr inmemoryversusdiskbasedcomputingwithrandomforestforstockanalysisacomparativestudy AT omkareshkulkarni inmemoryversusdiskbasedcomputingwithrandomforestforstockanalysisacomparativestudy AT kirtiwanjale inmemoryversusdiskbasedcomputingwithrandomforestforstockanalysisacomparativestudy |