Hidden Brain State-Based Internal Evaluation Using Kernel Inverse Reinforcement Learning in Brain-Machine Interfaces
Reinforcement learning (RL)-based brain machine interfaces (BMIs) assist paralyzed people in controlling neural prostheses without the need for real limb movement as supervised signals. The design of reward signal significantly impacts the learning efficiency of the RL-based decoders. Existing rewar...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Transactions on Neural Systems and Rehabilitation Engineering |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10759843/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850066197694906368 |
|---|---|
| author | Jieyuan Tan Xiang Zhang Shenghui Wu Zhiwei Song Yiwen Wang |
| author_facet | Jieyuan Tan Xiang Zhang Shenghui Wu Zhiwei Song Yiwen Wang |
| author_sort | Jieyuan Tan |
| collection | DOAJ |
| description | Reinforcement learning (RL)-based brain machine interfaces (BMIs) assist paralyzed people in controlling neural prostheses without the need for real limb movement as supervised signals. The design of reward signal significantly impacts the learning efficiency of the RL-based decoders. Existing reward designs in the RL-based BMI framework rely on external rewards or manually labeled internal rewards, unable to accurately extract subjects’ internal evaluation. In this paper, we propose a hidden brain state-based kernel inverse reinforcement learning (HBS-KIRL) method to accurately infer the subject-specific internal evaluation from neural activity during the BMI task. The state-space model is applied to project the neural state into low-dimensional hidden brain state space, which greatly reduces the exploration dimension. Then the kernel method is applied to speed up the convergence of policy, reward, and Q-value networks in reproducing kernel Hilbert space (RKHS). We tested our proposed algorithm on the data collected from the medial prefrontal cortex (mPFC) of rats when they were performing a two-lever-discrimination task. We assessed the state-value estimation performance of our proposed method and compared it with naïve IRL and PCA-based IRL. To validate that the extracted internal evaluation could contribute to the decoder training, we compared the decoding performance of decoders trained by different reward models, including manually designed reward, naïve IRL, PCA-IRL, and our proposed HBS-KIRL. The results show that the HBS-KIRL method can give a stable and accurate estimation of state-value distribution with respect to behavior. Compared with other methods, the decoder guided by HBS-KIRL achieves consistent and better decoding performance over days. This study reveals the potential of applying the IRL method to better extract subject-specific evaluation and improve the BMI decoding performance. |
| format | Article |
| id | doaj-art-3e49e6bbe0544e09aec320491c34a027 |
| institution | DOAJ |
| issn | 1534-4320 1558-0210 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Transactions on Neural Systems and Rehabilitation Engineering |
| spelling | doaj-art-3e49e6bbe0544e09aec320491c34a0272025-08-20T02:48:49ZengIEEEIEEE Transactions on Neural Systems and Rehabilitation Engineering1534-43201558-02102024-01-01324219422910.1109/TNSRE.2024.350371310759843Hidden Brain State-Based Internal Evaluation Using Kernel Inverse Reinforcement Learning in Brain-Machine InterfacesJieyuan Tan0https://orcid.org/0000-0003-2782-298XXiang Zhang1https://orcid.org/0000-0002-7432-9904Shenghui Wu2https://orcid.org/0000-0001-5333-8804Zhiwei Song3https://orcid.org/0000-0003-0605-5136Yiwen Wang4https://orcid.org/0000-0002-1878-6182Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Water Bay Rd, Hong KongDepartment of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Water Bay Rd, Hong KongDepartment of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Water Bay Rd, Hong KongDepartment of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Water Bay Rd, Hong KongDepartment of Electronic and Computer Engineering, Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology, Water Bay Rd, Hong KongReinforcement learning (RL)-based brain machine interfaces (BMIs) assist paralyzed people in controlling neural prostheses without the need for real limb movement as supervised signals. The design of reward signal significantly impacts the learning efficiency of the RL-based decoders. Existing reward designs in the RL-based BMI framework rely on external rewards or manually labeled internal rewards, unable to accurately extract subjects’ internal evaluation. In this paper, we propose a hidden brain state-based kernel inverse reinforcement learning (HBS-KIRL) method to accurately infer the subject-specific internal evaluation from neural activity during the BMI task. The state-space model is applied to project the neural state into low-dimensional hidden brain state space, which greatly reduces the exploration dimension. Then the kernel method is applied to speed up the convergence of policy, reward, and Q-value networks in reproducing kernel Hilbert space (RKHS). We tested our proposed algorithm on the data collected from the medial prefrontal cortex (mPFC) of rats when they were performing a two-lever-discrimination task. We assessed the state-value estimation performance of our proposed method and compared it with naïve IRL and PCA-based IRL. To validate that the extracted internal evaluation could contribute to the decoder training, we compared the decoding performance of decoders trained by different reward models, including manually designed reward, naïve IRL, PCA-IRL, and our proposed HBS-KIRL. The results show that the HBS-KIRL method can give a stable and accurate estimation of state-value distribution with respect to behavior. Compared with other methods, the decoder guided by HBS-KIRL achieves consistent and better decoding performance over days. This study reveals the potential of applying the IRL method to better extract subject-specific evaluation and improve the BMI decoding performance.https://ieeexplore.ieee.org/document/10759843/Brain-machine interface (BMI)inverse reinforcement learningmedial prefrontal cortexinternal evaluation |
| spellingShingle | Jieyuan Tan Xiang Zhang Shenghui Wu Zhiwei Song Yiwen Wang Hidden Brain State-Based Internal Evaluation Using Kernel Inverse Reinforcement Learning in Brain-Machine Interfaces IEEE Transactions on Neural Systems and Rehabilitation Engineering Brain-machine interface (BMI) inverse reinforcement learning medial prefrontal cortex internal evaluation |
| title | Hidden Brain State-Based Internal Evaluation Using Kernel Inverse Reinforcement Learning in Brain-Machine Interfaces |
| title_full | Hidden Brain State-Based Internal Evaluation Using Kernel Inverse Reinforcement Learning in Brain-Machine Interfaces |
| title_fullStr | Hidden Brain State-Based Internal Evaluation Using Kernel Inverse Reinforcement Learning in Brain-Machine Interfaces |
| title_full_unstemmed | Hidden Brain State-Based Internal Evaluation Using Kernel Inverse Reinforcement Learning in Brain-Machine Interfaces |
| title_short | Hidden Brain State-Based Internal Evaluation Using Kernel Inverse Reinforcement Learning in Brain-Machine Interfaces |
| title_sort | hidden brain state based internal evaluation using kernel inverse reinforcement learning in brain machine interfaces |
| topic | Brain-machine interface (BMI) inverse reinforcement learning medial prefrontal cortex internal evaluation |
| url | https://ieeexplore.ieee.org/document/10759843/ |
| work_keys_str_mv | AT jieyuantan hiddenbrainstatebasedinternalevaluationusingkernelinversereinforcementlearninginbrainmachineinterfaces AT xiangzhang hiddenbrainstatebasedinternalevaluationusingkernelinversereinforcementlearninginbrainmachineinterfaces AT shenghuiwu hiddenbrainstatebasedinternalevaluationusingkernelinversereinforcementlearninginbrainmachineinterfaces AT zhiweisong hiddenbrainstatebasedinternalevaluationusingkernelinversereinforcementlearninginbrainmachineinterfaces AT yiwenwang hiddenbrainstatebasedinternalevaluationusingkernelinversereinforcementlearninginbrainmachineinterfaces |