A Reproducible Analysis of Sequential Recommender Systems
Sequential Recommender Systems (SRSs) have emerged as a highly efficient approach to recommendation systems. By leveraging sequential data, SRSs can identify temporal patterns in user behaviour, significantly improving recommendation accuracy and relevance. Ensuring the reproducibility of these mode...
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10813337/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1832586839745101824 |
|---|---|
| author | Filippo Betello Antonio Purificato Federico Siciliano Giovanni Trappolini Andrea Bacciu Nicola Tonellotto Fabrizio Silvestri |
| author_facet | Filippo Betello Antonio Purificato Federico Siciliano Giovanni Trappolini Andrea Bacciu Nicola Tonellotto Fabrizio Silvestri |
| author_sort | Filippo Betello |
| collection | DOAJ |
| description | Sequential Recommender Systems (SRSs) have emerged as a highly efficient approach to recommendation systems. By leveraging sequential data, SRSs can identify temporal patterns in user behaviour, significantly improving recommendation accuracy and relevance. Ensuring the reproducibility of these models is paramount for advancing research and facilitating comparisons between them. Existing works exhibit shortcomings in reproducibility and replicability of results, leading to inconsistent statements across papers. Our work fills these gaps by standardising data pre-processing and model implementations, providing a comprehensive code resource, including a framework for developing SRSs and establishing a foundation for consistent and reproducible experimentation. We conduct extensive experiments on several benchmark datasets, comparing various SRSs implemented in our resource. We challenge prevailing performance benchmarks, offering new insights into the SR domain. For instance, SASRec does not consistently outperform GRU4Rec. On the contrary, when the number of model parameters becomes substantial, SASRec starts to clearly dominate all the other SRSs. This discrepancy underscores the significant impact that experimental configuration has on the outcomes and the importance of setting it up to ensure precise and comprehensive results. Failure to do so can lead to significantly flawed conclusions, highlighting the need for rigorous experimental design and analysis in SRS research. Our code is available at <uri>https://github.com/federicosiciliano/easy_lightning</uri>. |
| format | Article |
| id | doaj-art-120c917bf79e4371be54a588c4b514cf |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-120c917bf79e4371be54a588c4b514cf2025-01-25T00:02:56ZengIEEEIEEE Access2169-35362025-01-01135762577210.1109/ACCESS.2024.352204910813337A Reproducible Analysis of Sequential Recommender SystemsFilippo Betello0https://orcid.org/0009-0006-0945-9688Antonio Purificato1https://orcid.org/0009-0009-3933-380XFederico Siciliano2https://orcid.org/0000-0003-1339-6983Giovanni Trappolini3https://orcid.org/0000-0002-5515-634XAndrea Bacciu4https://orcid.org/0009-0007-1322-343XNicola Tonellotto5https://orcid.org/0000-0002-7427-1001Fabrizio Silvestri6https://orcid.org/0000-0001-7669-9055Department of Computer, Control and Management Engineering, Sapienza University of Rome, Rome, ItalyDepartment of Computer, Control and Management Engineering, Sapienza University of Rome, Rome, ItalyDepartment of Computer, Control and Management Engineering, Sapienza University of Rome, Rome, ItalyDepartment of Computer, Control and Management Engineering, Sapienza University of Rome, Rome, ItalyDepartment of Computer, Control and Management Engineering, Sapienza University of Rome, Rome, ItalyInformation Engineering Department, University of Pisa, Pisa, ItalyDepartment of Computer, Control and Management Engineering, Sapienza University of Rome, Rome, ItalySequential Recommender Systems (SRSs) have emerged as a highly efficient approach to recommendation systems. By leveraging sequential data, SRSs can identify temporal patterns in user behaviour, significantly improving recommendation accuracy and relevance. Ensuring the reproducibility of these models is paramount for advancing research and facilitating comparisons between them. Existing works exhibit shortcomings in reproducibility and replicability of results, leading to inconsistent statements across papers. Our work fills these gaps by standardising data pre-processing and model implementations, providing a comprehensive code resource, including a framework for developing SRSs and establishing a foundation for consistent and reproducible experimentation. We conduct extensive experiments on several benchmark datasets, comparing various SRSs implemented in our resource. We challenge prevailing performance benchmarks, offering new insights into the SR domain. For instance, SASRec does not consistently outperform GRU4Rec. On the contrary, when the number of model parameters becomes substantial, SASRec starts to clearly dominate all the other SRSs. This discrepancy underscores the significant impact that experimental configuration has on the outcomes and the importance of setting it up to ensure precise and comprehensive results. Failure to do so can lead to significantly flawed conclusions, highlighting the need for rigorous experimental design and analysis in SRS research. Our code is available at <uri>https://github.com/federicosiciliano/easy_lightning</uri>.https://ieeexplore.ieee.org/document/10813337/Recommendationsequential recommendationreproducibilityreplicabilityresource |
| spellingShingle | Filippo Betello Antonio Purificato Federico Siciliano Giovanni Trappolini Andrea Bacciu Nicola Tonellotto Fabrizio Silvestri A Reproducible Analysis of Sequential Recommender Systems IEEE Access Recommendation sequential recommendation reproducibility replicability resource |
| title | A Reproducible Analysis of Sequential Recommender Systems |
| title_full | A Reproducible Analysis of Sequential Recommender Systems |
| title_fullStr | A Reproducible Analysis of Sequential Recommender Systems |
| title_full_unstemmed | A Reproducible Analysis of Sequential Recommender Systems |
| title_short | A Reproducible Analysis of Sequential Recommender Systems |
| title_sort | reproducible analysis of sequential recommender systems |
| topic | Recommendation sequential recommendation reproducibility replicability resource |
| url | https://ieeexplore.ieee.org/document/10813337/ |
| work_keys_str_mv | AT filippobetello areproducibleanalysisofsequentialrecommendersystems AT antoniopurificato areproducibleanalysisofsequentialrecommendersystems AT federicosiciliano areproducibleanalysisofsequentialrecommendersystems AT giovannitrappolini areproducibleanalysisofsequentialrecommendersystems AT andreabacciu areproducibleanalysisofsequentialrecommendersystems AT nicolatonellotto areproducibleanalysisofsequentialrecommendersystems AT fabriziosilvestri areproducibleanalysisofsequentialrecommendersystems AT filippobetello reproducibleanalysisofsequentialrecommendersystems AT antoniopurificato reproducibleanalysisofsequentialrecommendersystems AT federicosiciliano reproducibleanalysisofsequentialrecommendersystems AT giovannitrappolini reproducibleanalysisofsequentialrecommendersystems AT andreabacciu reproducibleanalysisofsequentialrecommendersystems AT nicolatonellotto reproducibleanalysisofsequentialrecommendersystems AT fabriziosilvestri reproducibleanalysisofsequentialrecommendersystems |