Automated Assessment of Quality of Jupyter Notebooks Using Artificial Intelligence and Big Code
We present in this paper an automated method to assess the quality of Jupyter notebooks. The quality of notebooks is assessed in terms of reproducibility and executability. Specifically, we automatically extract a number of expert-defined features for each notebook, perform a feature selection step,...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
LibraryPress@UF
2021-04-01
|
| Series: | Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| Subjects: | |
| Online Access: | https://journals.flvc.org/FLAIRS/article/view/128560 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849736440113528832 |
|---|---|
| author | Priti Oli Rabin Banjade Lasang Jimba Tamang Vasile Rus |
| author_facet | Priti Oli Rabin Banjade Lasang Jimba Tamang Vasile Rus |
| author_sort | Priti Oli |
| collection | DOAJ |
| description | We present in this paper an automated method to assess the quality of Jupyter notebooks. The quality of notebooks is assessed in terms of reproducibility and executability. Specifically, we automatically extract a number of expert-defined features for each notebook, perform a feature selection step, and then trained supervised binary classifiers to predict whether a notebook is reproducible and executable, respectively. We also experimented with semantic code embeddings to capture the notebooks' semantics. We have evaluated these methods on a dataset of 306,539 notebooks and achieved an F1 score of 0.87 for reproducibility and 0.96 for executability (using expert-defined features) and an F1 score of 0.81 for reproducibility and 0.78 for executability (using code embeddings). Our results suggest that semantic code embeddings can be used to determine with good performance the reproducibility and executability of Jupyter notebooks, and since they can be automatically derived, they have the advantage of no need for expert involvement to define features. |
| format | Article |
| id | doaj-art-2c2f9fe19fe646f9a81eb6750ce69b5b |
| institution | DOAJ |
| issn | 2334-0754 2334-0762 |
| language | English |
| publishDate | 2021-04-01 |
| publisher | LibraryPress@UF |
| record_format | Article |
| series | Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| spelling | doaj-art-2c2f9fe19fe646f9a81eb6750ce69b5b2025-08-20T03:07:16ZengLibraryPress@UFProceedings of the International Florida Artificial Intelligence Research Society Conference2334-07542334-07622021-04-013410.32473/flairs.v34i1.12856062949Automated Assessment of Quality of Jupyter Notebooks Using Artificial Intelligence and Big CodePriti Oli0Rabin Banjade1Lasang Jimba Tamang2Vasile Rus3University of MemphisUniversity of MemphisUniversity of MemphisUniversity of MemphisWe present in this paper an automated method to assess the quality of Jupyter notebooks. The quality of notebooks is assessed in terms of reproducibility and executability. Specifically, we automatically extract a number of expert-defined features for each notebook, perform a feature selection step, and then trained supervised binary classifiers to predict whether a notebook is reproducible and executable, respectively. We also experimented with semantic code embeddings to capture the notebooks' semantics. We have evaluated these methods on a dataset of 306,539 notebooks and achieved an F1 score of 0.87 for reproducibility and 0.96 for executability (using expert-defined features) and an F1 score of 0.81 for reproducibility and 0.78 for executability (using code embeddings). Our results suggest that semantic code embeddings can be used to determine with good performance the reproducibility and executability of Jupyter notebooks, and since they can be automatically derived, they have the advantage of no need for expert involvement to define features.https://journals.flvc.org/FLAIRS/article/view/128560jupyter notebooksquality assessmentbig codereproducibilityexecutabilitymachine learningdeep learning |
| spellingShingle | Priti Oli Rabin Banjade Lasang Jimba Tamang Vasile Rus Automated Assessment of Quality of Jupyter Notebooks Using Artificial Intelligence and Big Code Proceedings of the International Florida Artificial Intelligence Research Society Conference jupyter notebooks quality assessment big code reproducibility executability machine learning deep learning |
| title | Automated Assessment of Quality of Jupyter Notebooks Using Artificial Intelligence and Big Code |
| title_full | Automated Assessment of Quality of Jupyter Notebooks Using Artificial Intelligence and Big Code |
| title_fullStr | Automated Assessment of Quality of Jupyter Notebooks Using Artificial Intelligence and Big Code |
| title_full_unstemmed | Automated Assessment of Quality of Jupyter Notebooks Using Artificial Intelligence and Big Code |
| title_short | Automated Assessment of Quality of Jupyter Notebooks Using Artificial Intelligence and Big Code |
| title_sort | automated assessment of quality of jupyter notebooks using artificial intelligence and big code |
| topic | jupyter notebooks quality assessment big code reproducibility executability machine learning deep learning |
| url | https://journals.flvc.org/FLAIRS/article/view/128560 |
| work_keys_str_mv | AT pritioli automatedassessmentofqualityofjupyternotebooksusingartificialintelligenceandbigcode AT rabinbanjade automatedassessmentofqualityofjupyternotebooksusingartificialintelligenceandbigcode AT lasangjimbatamang automatedassessmentofqualityofjupyternotebooksusingartificialintelligenceandbigcode AT vasilerus automatedassessmentofqualityofjupyternotebooksusingartificialintelligenceandbigcode |