Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative Multi-agent Reinforcement Learning
The effectiveness of multi-agent reinforcement learning (MARL) hinges largely on the meticulous arrangement of objectives. Yet, conventional MARL methods might not completely harness the inherent structures present in environmental states and agent relationships for goal organization. This study is...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
LibraryPress@UF
2024-05-01
|
| Series: | Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| Online Access: | https://journals.flvc.org/FLAIRS/article/view/135588 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850271035476148224 |
|---|---|
| author | Lixing Liu Volkan Ustun Rajay Kumar |
| author_facet | Lixing Liu Volkan Ustun Rajay Kumar |
| author_sort | Lixing Liu |
| collection | DOAJ |
| description | The effectiveness of multi-agent reinforcement learning (MARL) hinges largely on the meticulous arrangement of objectives. Yet, conventional MARL methods might not completely harness the inherent structures present in environmental states and agent relationships for goal organization. This study is conducted within the domain of military training simulations, which are typically characterized by complexity, heterogeneity, non-stationary and doctrine-driven environments with a clear organizational hierarchy and a top-down chain of command. This research investigates the approximation and integration of the organizational hierarchy into MARL for cooperative training scenarios, with the goal of streamlining the processes of reward engineering and enhancing team coordination. In the preliminary experiments, we employed two-tiered commander-subordinate feudal hierarchical (CSFH) networks to separate the prioritized team goal and individual goals. The empirical results demonstrate that the proposed framework enhances learning efficiency. It guarantees the learning of a prioritized policy for the commander agent and encourages subordinate agents to explore areas of interest more frequently, guided by appropriate soft constraints imposed by the commander. |
| format | Article |
| id | doaj-art-54d79c3d982149ac91d069318437bc21 |
| institution | OA Journals |
| issn | 2334-0754 2334-0762 |
| language | English |
| publishDate | 2024-05-01 |
| publisher | LibraryPress@UF |
| record_format | Article |
| series | Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| spelling | doaj-art-54d79c3d982149ac91d069318437bc212025-08-20T01:52:22ZengLibraryPress@UFProceedings of the International Florida Artificial Intelligence Research Society Conference2334-07542334-07622024-05-013710.32473/flairs.37.1.13558871967Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative Multi-agent Reinforcement LearningLixing Liu0https://orcid.org/0009-0009-8648-2007Volkan Ustun1https://orcid.org/0000-0002-7090-4086Rajay Kumar2University of Southern CaliforniaInstitute for Creative Technologies, University of Southern CaliforniaInstitute for Creative Technologies, University of Southern CaliforniaThe effectiveness of multi-agent reinforcement learning (MARL) hinges largely on the meticulous arrangement of objectives. Yet, conventional MARL methods might not completely harness the inherent structures present in environmental states and agent relationships for goal organization. This study is conducted within the domain of military training simulations, which are typically characterized by complexity, heterogeneity, non-stationary and doctrine-driven environments with a clear organizational hierarchy and a top-down chain of command. This research investigates the approximation and integration of the organizational hierarchy into MARL for cooperative training scenarios, with the goal of streamlining the processes of reward engineering and enhancing team coordination. In the preliminary experiments, we employed two-tiered commander-subordinate feudal hierarchical (CSFH) networks to separate the prioritized team goal and individual goals. The empirical results demonstrate that the proposed framework enhances learning efficiency. It guarantees the learning of a prioritized policy for the commander agent and encourages subordinate agents to explore areas of interest more frequently, guided by appropriate soft constraints imposed by the commander.https://journals.flvc.org/FLAIRS/article/view/135588 |
| spellingShingle | Lixing Liu Volkan Ustun Rajay Kumar Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative Multi-agent Reinforcement Learning Proceedings of the International Florida Artificial Intelligence Research Society Conference |
| title | Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative Multi-agent Reinforcement Learning |
| title_full | Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative Multi-agent Reinforcement Learning |
| title_fullStr | Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative Multi-agent Reinforcement Learning |
| title_full_unstemmed | Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative Multi-agent Reinforcement Learning |
| title_short | Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative Multi-agent Reinforcement Learning |
| title_sort | leveraging organizational hierarchy to simplify reward design in cooperative multi agent reinforcement learning |
| url | https://journals.flvc.org/FLAIRS/article/view/135588 |
| work_keys_str_mv | AT lixingliu leveragingorganizationalhierarchytosimplifyrewarddesignincooperativemultiagentreinforcementlearning AT volkanustun leveragingorganizationalhierarchytosimplifyrewarddesignincooperativemultiagentreinforcementlearning AT rajaykumar leveragingorganizationalhierarchytosimplifyrewarddesignincooperativemultiagentreinforcementlearning |