Inter-rater reliability and validity of supervision performance assessment and recognition (SPARS) indicators to assess medicines management in public health facilities in Nepal
Background Nepal’s Ministry of Health and Population piloted the Supervision, Performance Assessment, and Recognition Strategy (SPARS), a multipronged strategy to improve medicines management practices in health facilities. Medicines management supervisors (MMS) periodically assess facility performa...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Taylor & Francis Group
2025-12-01
|
| Series: | Journal of Pharmaceutical Policy and Practice |
| Subjects: | |
| Online Access: | https://www.tandfonline.com/doi/10.1080/20523211.2025.2477098 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850099197609508864 |
|---|---|
| author | Santusta Adhikari Anup Bastola Reekesh Shrestha Narendra Kumar Khanal Birna Trap |
| author_facet | Santusta Adhikari Anup Bastola Reekesh Shrestha Narendra Kumar Khanal Birna Trap |
| author_sort | Santusta Adhikari |
| collection | DOAJ |
| description | Background Nepal’s Ministry of Health and Population piloted the Supervision, Performance Assessment, and Recognition Strategy (SPARS), a multipronged strategy to improve medicines management practices in health facilities. Medicines management supervisors (MMS) periodically assess facility performance using 25 indicators grouped into 5 domains – prescribing, dispensing, ordering and reporting quality and stock and storage management – and provide targeted on-the-job coaching. This study aimed to determine inter-rater reliability (IRR) and validity for SPARS indicators in Nepali context.Methods We assessed IRR and validity scores with three rater teams assessing medicines management in six public health posts. Each team of three MMS and one technical advisor evaluated two facilities, resulting in 24 total assessments. We calculated the mean and median validity and IRR scores for SPARS overall, and by domain and indicator. Acceptable scores were mean validity and IRR scores ≥75%, moderately acceptable were 50–75%, and unacceptable were <50%.Results The mean overall IRR (60%) and validity (74%) of all 25 SPARS indicators were significantly different (p < 0.003) but not by domain. Three domains related to dispensing, ordering and reporting, and storage had acceptable mean validity scores, while the prescribing quality domain had an unacceptable mean IRR score. We found significant difference (p < 0.0001) between the indicator IRR and validity acceptability scores. Four (16%) and 12 (48%) of the SPARS indicators had acceptable reproducibility and validity, respectively; 8 (33%) IRR scores and 1 (4%) validity score were unacceptable. Simple indicators (67%) had significantly (p < 0.02) higher reproducibility compared to complex indicators (42%), and higher validity of 80% and 72%, respectively, which was not significant.Conclusions This study demonstrated that the SPARS indicators produce moderately accurate findings. Validity was significantly higher than IRR and both need improvement, especially for complex indicators. As such, future SPARS implementation should endeavour to include validity and IRR assessments to help improve the indicators. |
| format | Article |
| id | doaj-art-36c609bb33f64178adc44aae8d59b9b3 |
| institution | DOAJ |
| issn | 2052-3211 |
| language | English |
| publishDate | 2025-12-01 |
| publisher | Taylor & Francis Group |
| record_format | Article |
| series | Journal of Pharmaceutical Policy and Practice |
| spelling | doaj-art-36c609bb33f64178adc44aae8d59b9b32025-08-20T02:40:32ZengTaylor & Francis GroupJournal of Pharmaceutical Policy and Practice2052-32112025-12-0118110.1080/20523211.2025.2477098Inter-rater reliability and validity of supervision performance assessment and recognition (SPARS) indicators to assess medicines management in public health facilities in NepalSantusta Adhikari0Anup Bastola1Reekesh Shrestha2Narendra Kumar Khanal3Birna Trap4USAID Medicines, Technologies, and Pharmaceutical Services (MTaPS), Kathmandu, NepalCurative Service Division, Department of Health Services, Ministry of Health and Population, Kathmandu, NepalUSAID Medicines, Technologies, and Pharmaceutical Services (MTaPS), Kathmandu, NepalCurative Service Division, Department of Health Services, Ministry of Health and Population, Kathmandu, NepalUSAID Medicines, Technologies, and Pharmaceutical Services (MTaPS), Kathmandu, NepalBackground Nepal’s Ministry of Health and Population piloted the Supervision, Performance Assessment, and Recognition Strategy (SPARS), a multipronged strategy to improve medicines management practices in health facilities. Medicines management supervisors (MMS) periodically assess facility performance using 25 indicators grouped into 5 domains – prescribing, dispensing, ordering and reporting quality and stock and storage management – and provide targeted on-the-job coaching. This study aimed to determine inter-rater reliability (IRR) and validity for SPARS indicators in Nepali context.Methods We assessed IRR and validity scores with three rater teams assessing medicines management in six public health posts. Each team of three MMS and one technical advisor evaluated two facilities, resulting in 24 total assessments. We calculated the mean and median validity and IRR scores for SPARS overall, and by domain and indicator. Acceptable scores were mean validity and IRR scores ≥75%, moderately acceptable were 50–75%, and unacceptable were <50%.Results The mean overall IRR (60%) and validity (74%) of all 25 SPARS indicators were significantly different (p < 0.003) but not by domain. Three domains related to dispensing, ordering and reporting, and storage had acceptable mean validity scores, while the prescribing quality domain had an unacceptable mean IRR score. We found significant difference (p < 0.0001) between the indicator IRR and validity acceptability scores. Four (16%) and 12 (48%) of the SPARS indicators had acceptable reproducibility and validity, respectively; 8 (33%) IRR scores and 1 (4%) validity score were unacceptable. Simple indicators (67%) had significantly (p < 0.02) higher reproducibility compared to complex indicators (42%), and higher validity of 80% and 72%, respectively, which was not significant.Conclusions This study demonstrated that the SPARS indicators produce moderately accurate findings. Validity was significantly higher than IRR and both need improvement, especially for complex indicators. As such, future SPARS implementation should endeavour to include validity and IRR assessments to help improve the indicators.https://www.tandfonline.com/doi/10.1080/20523211.2025.2477098SPARSinter-rater reliabilityIRRvalidityindicatorsmedicines management indicators |
| spellingShingle | Santusta Adhikari Anup Bastola Reekesh Shrestha Narendra Kumar Khanal Birna Trap Inter-rater reliability and validity of supervision performance assessment and recognition (SPARS) indicators to assess medicines management in public health facilities in Nepal Journal of Pharmaceutical Policy and Practice SPARS inter-rater reliability IRR validity indicators medicines management indicators |
| title | Inter-rater reliability and validity of supervision performance assessment and recognition (SPARS) indicators to assess medicines management in public health facilities in Nepal |
| title_full | Inter-rater reliability and validity of supervision performance assessment and recognition (SPARS) indicators to assess medicines management in public health facilities in Nepal |
| title_fullStr | Inter-rater reliability and validity of supervision performance assessment and recognition (SPARS) indicators to assess medicines management in public health facilities in Nepal |
| title_full_unstemmed | Inter-rater reliability and validity of supervision performance assessment and recognition (SPARS) indicators to assess medicines management in public health facilities in Nepal |
| title_short | Inter-rater reliability and validity of supervision performance assessment and recognition (SPARS) indicators to assess medicines management in public health facilities in Nepal |
| title_sort | inter rater reliability and validity of supervision performance assessment and recognition spars indicators to assess medicines management in public health facilities in nepal |
| topic | SPARS inter-rater reliability IRR validity indicators medicines management indicators |
| url | https://www.tandfonline.com/doi/10.1080/20523211.2025.2477098 |
| work_keys_str_mv | AT santustaadhikari interraterreliabilityandvalidityofsupervisionperformanceassessmentandrecognitionsparsindicatorstoassessmedicinesmanagementinpublichealthfacilitiesinnepal AT anupbastola interraterreliabilityandvalidityofsupervisionperformanceassessmentandrecognitionsparsindicatorstoassessmedicinesmanagementinpublichealthfacilitiesinnepal AT reekeshshrestha interraterreliabilityandvalidityofsupervisionperformanceassessmentandrecognitionsparsindicatorstoassessmedicinesmanagementinpublichealthfacilitiesinnepal AT narendrakumarkhanal interraterreliabilityandvalidityofsupervisionperformanceassessmentandrecognitionsparsindicatorstoassessmedicinesmanagementinpublichealthfacilitiesinnepal AT birnatrap interraterreliabilityandvalidityofsupervisionperformanceassessmentandrecognitionsparsindicatorstoassessmedicinesmanagementinpublichealthfacilitiesinnepal |