Scientometric Evaluation of Research Institutions: Identifying the Appropriate Dimensions and Attributes for Assessment
Many descriptions and evaluations of research institutions apply publication and citation indicators, which can also be found in most of the popular rankings of universities all over the world. When comparing different institutions, the question arises whether scientometric indicators, and instituti...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Korea Institute of Science and Technology Information
2025-06-01
|
| Series: | Journal of Information Science Theory and Practice |
| Subjects: | |
| Online Access: | https://data.doi.or.kr/10.1633/JISTaP.2025.13.2.4 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850100764458876928 |
|---|---|
| author | Christian Schlögl Wolfgang G. Stock Gerhard Reichmann |
| author_facet | Christian Schlögl Wolfgang G. Stock Gerhard Reichmann |
| author_sort | Christian Schlögl |
| collection | DOAJ |
| description | Many descriptions and evaluations of research institutions apply publication and citation indicators, which can also be found in most of the popular rankings of universities all over the world. When comparing different institutions, the question arises whether scientometric indicators, and institutional rankings derived from them, are really valid. In our paper we discuss various dimensions of scientometric analyses which have a more or less strong impact on the results of research evaluations on the scientometric meso-level (e.g., department or university level). Concerning research output (based upon publications), we found nine different dimensions, namely time period, size of the institution, representatives of the institution, data source, language, document types and their weighting, co-authorship, document length, and access option. Concerning research impact (based upon citations), there are 14 different dimensions. Using the simple example of two university departments, we present different rankings while varying a few attributes of the suggested dimensions. As the results show, there are large differences between the performed rankings. Each ranking stresses a different aspect; hence there is not the one and only valid ranking. All the popular global university rankings work with arbitrary selections of indicator combinations, making the results more or less arbitrary. The greatest issues are the rankings’ incomplete empirical bases, the non-consideration of fractional counting of authors, and – most importantly – the disregarding of the institutions’ sizes. |
| format | Article |
| id | doaj-art-73030ff5acf643d8956dc6816e1fc7f4 |
| institution | DOAJ |
| issn | 2287-9099 2287-4577 |
| language | English |
| publishDate | 2025-06-01 |
| publisher | Korea Institute of Science and Technology Information |
| record_format | Article |
| series | Journal of Information Science Theory and Practice |
| spelling | doaj-art-73030ff5acf643d8956dc6816e1fc7f42025-08-20T02:40:13ZengKorea Institute of Science and Technology InformationJournal of Information Science Theory and Practice2287-90992287-45772025-06-0113210.1633/JISTaP.2025.13.2.4JISTaP.2025.13.2.4Scientometric Evaluation of Research Institutions: Identifying the Appropriate Dimensions and Attributes for AssessmentChristian Schlögl0Wolfgang G. Stock1Gerhard Reichmann2Department of Operations and Information Systems, University of Graz, Graz, AustriaDepartment of Information Science, Heinrich Heine University Düsseldorf, Düsseldorf, Germany E-mail: wolfgang.stock@hhu.deDepartment of Operations and Information Systems, University of Graz, Graz, Austria E-mail: gerhard.reichmann@uni-graz.atMany descriptions and evaluations of research institutions apply publication and citation indicators, which can also be found in most of the popular rankings of universities all over the world. When comparing different institutions, the question arises whether scientometric indicators, and institutional rankings derived from them, are really valid. In our paper we discuss various dimensions of scientometric analyses which have a more or less strong impact on the results of research evaluations on the scientometric meso-level (e.g., department or university level). Concerning research output (based upon publications), we found nine different dimensions, namely time period, size of the institution, representatives of the institution, data source, language, document types and their weighting, co-authorship, document length, and access option. Concerning research impact (based upon citations), there are 14 different dimensions. Using the simple example of two university departments, we present different rankings while varying a few attributes of the suggested dimensions. As the results show, there are large differences between the performed rankings. Each ranking stresses a different aspect; hence there is not the one and only valid ranking. All the popular global university rankings work with arbitrary selections of indicator combinations, making the results more or less arbitrary. The greatest issues are the rankings’ incomplete empirical bases, the non-consideration of fractional counting of authors, and – most importantly – the disregarding of the institutions’ sizes.https://data.doi.or.kr/10.1633/JISTaP.2025.13.2.4research evaluationscientometricsrankingsmulti-dimensional evaluationuniversity departments |
| spellingShingle | Christian Schlögl Wolfgang G. Stock Gerhard Reichmann Scientometric Evaluation of Research Institutions: Identifying the Appropriate Dimensions and Attributes for Assessment Journal of Information Science Theory and Practice research evaluation scientometrics rankings multi-dimensional evaluation university departments |
| title | Scientometric Evaluation of Research Institutions: Identifying the Appropriate Dimensions and Attributes for Assessment |
| title_full | Scientometric Evaluation of Research Institutions: Identifying the Appropriate Dimensions and Attributes for Assessment |
| title_fullStr | Scientometric Evaluation of Research Institutions: Identifying the Appropriate Dimensions and Attributes for Assessment |
| title_full_unstemmed | Scientometric Evaluation of Research Institutions: Identifying the Appropriate Dimensions and Attributes for Assessment |
| title_short | Scientometric Evaluation of Research Institutions: Identifying the Appropriate Dimensions and Attributes for Assessment |
| title_sort | scientometric evaluation of research institutions identifying the appropriate dimensions and attributes for assessment |
| topic | research evaluation scientometrics rankings multi-dimensional evaluation university departments |
| url | https://data.doi.or.kr/10.1633/JISTaP.2025.13.2.4 |
| work_keys_str_mv | AT christianschlogl scientometricevaluationofresearchinstitutionsidentifyingtheappropriatedimensionsandattributesforassessment AT wolfganggstock scientometricevaluationofresearchinstitutionsidentifyingtheappropriatedimensionsandattributesforassessment AT gerhardreichmann scientometricevaluationofresearchinstitutionsidentifyingtheappropriatedimensionsandattributesforassessment |