PvaPy streaming framework for real-time data processing
User facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynam...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
International Union of Crystallography
2025-05-01
|
| Series: | Journal of Synchrotron Radiation |
| Subjects: | |
| Online Access: | https://journals.iucr.org/paper?S1600577525002115 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850144185748815872 |
|---|---|
| author | Siniša Veseli John Hammonds Steven Henke Hannah Parraga Barbara Frosik Nicholas Schwarz |
| author_facet | Siniša Veseli John Hammonds Steven Henke Hannah Parraga Barbara Frosik Nicholas Schwarz |
| author_sort | Siniša Veseli |
| collection | DOAJ |
| description | User facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynamically and modify them in real time. In turn, this requires a computing framework for real-time processing capable of moving data quickly from the detector to local or remote computing resources, processing data, and returning results to users. In this paper, we discuss the streaming framework built on top of PvaPy, a Python API for the EPICS pvAccess protocol. We describe the framework architecture and capabilities, and discuss scientific use cases and applications that benefit from streaming workflows implemented on top of this framework. We also illustrate the framework's performance in terms of achievable data-processing rates for various detector image sizes. |
| format | Article |
| id | doaj-art-ed47fba054db4cfbb8ca33943f6df6b4 |
| institution | OA Journals |
| issn | 1600-5775 |
| language | English |
| publishDate | 2025-05-01 |
| publisher | International Union of Crystallography |
| record_format | Article |
| series | Journal of Synchrotron Radiation |
| spelling | doaj-art-ed47fba054db4cfbb8ca33943f6df6b42025-08-20T02:28:27ZengInternational Union of CrystallographyJournal of Synchrotron Radiation1600-57752025-05-0132382383610.1107/S1600577525002115yn5121PvaPy streaming framework for real-time data processingSiniša Veseli0John Hammonds1Steven Henke2Hannah Parraga3Barbara Frosik4Nicholas Schwarz5Argonne National Laboratory, 9700 South Cass Avenue, Lemont, IL 60439, USAArgonne National Laboratory, 9700 South Cass Avenue, Lemont, IL 60439, USAArgonne National Laboratory, 9700 South Cass Avenue, Lemont, IL 60439, USAArgonne National Laboratory, 9700 South Cass Avenue, Lemont, IL 60439, USAArgonne National Laboratory, 9700 South Cass Avenue, Lemont, IL 60439, USAArgonne National Laboratory, 9700 South Cass Avenue, Lemont, IL 60439, USAUser facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynamically and modify them in real time. In turn, this requires a computing framework for real-time processing capable of moving data quickly from the detector to local or remote computing resources, processing data, and returning results to users. In this paper, we discuss the streaming framework built on top of PvaPy, a Python API for the EPICS pvAccess protocol. We describe the framework architecture and capabilities, and discuss scientific use cases and applications that benefit from streaming workflows implemented on top of this framework. We also illustrate the framework's performance in terms of achievable data-processing rates for various detector image sizes.https://journals.iucr.org/paper?S1600577525002115data streamingreal-time data processingcomputing frameworkspvapyepics pvaccesspython applications |
| spellingShingle | Siniša Veseli John Hammonds Steven Henke Hannah Parraga Barbara Frosik Nicholas Schwarz PvaPy streaming framework for real-time data processing Journal of Synchrotron Radiation data streaming real-time data processing computing frameworks pvapy epics pvaccess python applications |
| title | PvaPy streaming framework for real-time data processing |
| title_full | PvaPy streaming framework for real-time data processing |
| title_fullStr | PvaPy streaming framework for real-time data processing |
| title_full_unstemmed | PvaPy streaming framework for real-time data processing |
| title_short | PvaPy streaming framework for real-time data processing |
| title_sort | pvapy streaming framework for real time data processing |
| topic | data streaming real-time data processing computing frameworks pvapy epics pvaccess python applications |
| url | https://journals.iucr.org/paper?S1600577525002115 |
| work_keys_str_mv | AT sinisaveseli pvapystreamingframeworkforrealtimedataprocessing AT johnhammonds pvapystreamingframeworkforrealtimedataprocessing AT stevenhenke pvapystreamingframeworkforrealtimedataprocessing AT hannahparraga pvapystreamingframeworkforrealtimedataprocessing AT barbarafrosik pvapystreamingframeworkforrealtimedataprocessing AT nicholasschwarz pvapystreamingframeworkforrealtimedataprocessing |