Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning
Coverage path planning (CPP) is the problem of finding a path that covers the entire free space of a confined area, with applications ranging from robotic lawn mowing to search-and-rescue. While for known environments, offline methods can find provably complete paths, and in some cases optimal solut...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/11039829/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849473572420976640 |
|---|---|
| author | Arvi Jonnarth Ola Johansson Jie Zhao Michael Felsberg |
| author_facet | Arvi Jonnarth Ola Johansson Jie Zhao Michael Felsberg |
| author_sort | Arvi Jonnarth |
| collection | DOAJ |
| description | Coverage path planning (CPP) is the problem of finding a path that covers the entire free space of a confined area, with applications ranging from robotic lawn mowing to search-and-rescue. While for known environments, offline methods can find provably complete paths, and in some cases optimal solutions, unknown environments need to be planned online during mapping. We investigate the suitability of continuous-space reinforcement learning (RL) for this challenging problem, and propose a computationally feasible egocentric map representation based on frontiers, as well as a novel reward term based on total variation to promote complete coverage. Compared to existing classical methods, this approach allows for a flexible path space, and enables the agent to adapt to specific environment characteristics. Meanwhile, the deployment of RL models on real robot systems is difficult. Training from scratch may be infeasible due to slow convergence times, while transferring from simulation to reality, i.e. sim-to-real transfer, is a key challenge in itself. We bridge the sim-to-real gap through a semi-virtual environment, including a real robot and real-time aspects, while utilizing a simulated sensor and obstacles to enable environment randomization and automated episode resetting. We investigate what level of fine-tuning is needed for adapting to a realistic setting. Through extensive experiments, we show that our approach surpasses the performance of both previous RL-based approaches and highly specialized methods across multiple CPP variations in simulation. Meanwhile, our method successfully transfers to a real robot. Our code implementation can be found online (Link to code repository: <uri>https://github.com/arvijj/rl-cpp</uri>). |
| format | Article |
| id | doaj-art-e1d1508810ca4149b15daa34c5f91f14 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-e1d1508810ca4149b15daa34c5f91f142025-08-20T03:24:06ZengIEEEIEEE Access2169-35362025-01-011310688310690510.1109/ACCESS.2025.358103511039829Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path PlanningArvi Jonnarth0https://orcid.org/0000-0002-3434-2522Ola Johansson1https://orcid.org/0009-0001-1386-3230Jie Zhao2https://orcid.org/0000-0002-3924-5537Michael Felsberg3https://orcid.org/0000-0002-6096-3648Manta Systems, Linköping, SwedenDepartment of Electrical Engineering, Linköping University, Linköping, SwedenDepartment of Information and Communication Engineering, Dalian University of Technology, Dalian, ChinaDepartment of Electrical Engineering, Linköping University, Linköping, SwedenCoverage path planning (CPP) is the problem of finding a path that covers the entire free space of a confined area, with applications ranging from robotic lawn mowing to search-and-rescue. While for known environments, offline methods can find provably complete paths, and in some cases optimal solutions, unknown environments need to be planned online during mapping. We investigate the suitability of continuous-space reinforcement learning (RL) for this challenging problem, and propose a computationally feasible egocentric map representation based on frontiers, as well as a novel reward term based on total variation to promote complete coverage. Compared to existing classical methods, this approach allows for a flexible path space, and enables the agent to adapt to specific environment characteristics. Meanwhile, the deployment of RL models on real robot systems is difficult. Training from scratch may be infeasible due to slow convergence times, while transferring from simulation to reality, i.e. sim-to-real transfer, is a key challenge in itself. We bridge the sim-to-real gap through a semi-virtual environment, including a real robot and real-time aspects, while utilizing a simulated sensor and obstacles to enable environment randomization and automated episode resetting. We investigate what level of fine-tuning is needed for adapting to a realistic setting. Through extensive experiments, we show that our approach surpasses the performance of both previous RL-based approaches and highly specialized methods across multiple CPP variations in simulation. Meanwhile, our method successfully transfers to a real robot. Our code implementation can be found online (Link to code repository: <uri>https://github.com/arvijj/rl-cpp</uri>).https://ieeexplore.ieee.org/document/11039829/Coverage path planningend-to-end learningexplorationonlinereal-timereinforcement learning |
| spellingShingle | Arvi Jonnarth Ola Johansson Jie Zhao Michael Felsberg Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning IEEE Access Coverage path planning end-to-end learning exploration online real-time reinforcement learning |
| title | Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning |
| title_full | Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning |
| title_fullStr | Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning |
| title_full_unstemmed | Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning |
| title_short | Sim-to-Real Transfer of Deep Reinforcement Learning Agents for Online Coverage Path Planning |
| title_sort | sim to real transfer of deep reinforcement learning agents for online coverage path planning |
| topic | Coverage path planning end-to-end learning exploration online real-time reinforcement learning |
| url | https://ieeexplore.ieee.org/document/11039829/ |
| work_keys_str_mv | AT arvijonnarth simtorealtransferofdeepreinforcementlearningagentsforonlinecoveragepathplanning AT olajohansson simtorealtransferofdeepreinforcementlearningagentsforonlinecoveragepathplanning AT jiezhao simtorealtransferofdeepreinforcementlearningagentsforonlinecoveragepathplanning AT michaelfelsberg simtorealtransferofdeepreinforcementlearningagentsforonlinecoveragepathplanning |