A Continuous Space Path Planning Method for Unmanned Aerial Vehicle Based on Particle Swarm Optimization-Enhanced Deep Q-Network
In the field of unmanned aerial vehicle (UAV) path planning, the conventional deep Q-network (DQN) algorithm encounters the issue of action space discretization, which results in the generation of unsmooth and inefficient planned paths. To address this issue, we introduce the particle swarm optimiza...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-02-01
|
| Series: | Drones |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2504-446X/9/2/122 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850081254958956544 |
|---|---|
| author | Le Han Hui Zhang Nan An |
| author_facet | Le Han Hui Zhang Nan An |
| author_sort | Le Han |
| collection | DOAJ |
| description | In the field of unmanned aerial vehicle (UAV) path planning, the conventional deep Q-network (DQN) algorithm encounters the issue of action space discretization, which results in the generation of unsmooth and inefficient planned paths. To address this issue, we introduce the particle swarm optimization (PSO) algorithm into DQN to convert the discrete action space into a continuous one. This method divides the agent’s surrounding space into discrete and continuous action spaces. The PSO algorithm performs a global search in the continuous space to obtain a continuous candidate solution, while DQN learns a policy in the discrete space to obtain a discrete candidate solution. Then, the two candidate solutions are combined using a weighted vector method to determine a direction that balances global search and policy learning. Additionally, we introduce a novel feature matrix as the state space for DQN, providing more accurate environmental and positional representations. Furthermore, we incorporate a mechanism into the base prioritized experience replay (PER) and N-step updates, which combines the current temporal difference error (TD-error) with historical priorities and includes a policy entropy penalty term, thereby enhancing DQN’s ability to learn long-term dependencies. The performance of the PSO-DQN model is further improved through an enhanced <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>ε</mi></semantics></math></inline-formula>-greedy policy and learning rate decay strategy. Simulation results and experiments using the Flightmare simulator demonstrate that the proposed method generates smoother and more efficient paths for drones, exhibiting strong robustness in complex environments. |
| format | Article |
| id | doaj-art-fa37df680b3b4737b70bf1e05bc43332 |
| institution | DOAJ |
| issn | 2504-446X |
| language | English |
| publishDate | 2025-02-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Drones |
| spelling | doaj-art-fa37df680b3b4737b70bf1e05bc433322025-08-20T02:44:46ZengMDPI AGDrones2504-446X2025-02-019212210.3390/drones9020122A Continuous Space Path Planning Method for Unmanned Aerial Vehicle Based on Particle Swarm Optimization-Enhanced Deep Q-NetworkLe Han0Hui Zhang1Nan An2School of Electronic Information Engineering, Inner Mongolia University, Hohhot 010021, ChinaSchool of Electronic Information Engineering, Inner Mongolia University, Hohhot 010021, ChinaSchool of Electronic Information Engineering, Inner Mongolia University, Hohhot 010021, ChinaIn the field of unmanned aerial vehicle (UAV) path planning, the conventional deep Q-network (DQN) algorithm encounters the issue of action space discretization, which results in the generation of unsmooth and inefficient planned paths. To address this issue, we introduce the particle swarm optimization (PSO) algorithm into DQN to convert the discrete action space into a continuous one. This method divides the agent’s surrounding space into discrete and continuous action spaces. The PSO algorithm performs a global search in the continuous space to obtain a continuous candidate solution, while DQN learns a policy in the discrete space to obtain a discrete candidate solution. Then, the two candidate solutions are combined using a weighted vector method to determine a direction that balances global search and policy learning. Additionally, we introduce a novel feature matrix as the state space for DQN, providing more accurate environmental and positional representations. Furthermore, we incorporate a mechanism into the base prioritized experience replay (PER) and N-step updates, which combines the current temporal difference error (TD-error) with historical priorities and includes a policy entropy penalty term, thereby enhancing DQN’s ability to learn long-term dependencies. The performance of the PSO-DQN model is further improved through an enhanced <inline-formula><math xmlns="http://www.w3.org/1998/Math/MathML" display="inline"><semantics><mi>ε</mi></semantics></math></inline-formula>-greedy policy and learning rate decay strategy. Simulation results and experiments using the Flightmare simulator demonstrate that the proposed method generates smoother and more efficient paths for drones, exhibiting strong robustness in complex environments.https://www.mdpi.com/2504-446X/9/2/122UAVparticle swarm optimizationdeep Q-Networkpath planning |
| spellingShingle | Le Han Hui Zhang Nan An A Continuous Space Path Planning Method for Unmanned Aerial Vehicle Based on Particle Swarm Optimization-Enhanced Deep Q-Network Drones UAV particle swarm optimization deep Q-Network path planning |
| title | A Continuous Space Path Planning Method for Unmanned Aerial Vehicle Based on Particle Swarm Optimization-Enhanced Deep Q-Network |
| title_full | A Continuous Space Path Planning Method for Unmanned Aerial Vehicle Based on Particle Swarm Optimization-Enhanced Deep Q-Network |
| title_fullStr | A Continuous Space Path Planning Method for Unmanned Aerial Vehicle Based on Particle Swarm Optimization-Enhanced Deep Q-Network |
| title_full_unstemmed | A Continuous Space Path Planning Method for Unmanned Aerial Vehicle Based on Particle Swarm Optimization-Enhanced Deep Q-Network |
| title_short | A Continuous Space Path Planning Method for Unmanned Aerial Vehicle Based on Particle Swarm Optimization-Enhanced Deep Q-Network |
| title_sort | continuous space path planning method for unmanned aerial vehicle based on particle swarm optimization enhanced deep q network |
| topic | UAV particle swarm optimization deep Q-Network path planning |
| url | https://www.mdpi.com/2504-446X/9/2/122 |
| work_keys_str_mv | AT lehan acontinuousspacepathplanningmethodforunmannedaerialvehiclebasedonparticleswarmoptimizationenhanceddeepqnetwork AT huizhang acontinuousspacepathplanningmethodforunmannedaerialvehiclebasedonparticleswarmoptimizationenhanceddeepqnetwork AT nanan acontinuousspacepathplanningmethodforunmannedaerialvehiclebasedonparticleswarmoptimizationenhanceddeepqnetwork AT lehan continuousspacepathplanningmethodforunmannedaerialvehiclebasedonparticleswarmoptimizationenhanceddeepqnetwork AT huizhang continuousspacepathplanningmethodforunmannedaerialvehiclebasedonparticleswarmoptimizationenhanceddeepqnetwork AT nanan continuousspacepathplanningmethodforunmannedaerialvehiclebasedonparticleswarmoptimizationenhanceddeepqnetwork |