LLM-Driven, Self-Improving Framework for Security Test Automation: Leveraging Karate DSL for Augmented API Resilience
Modern software architectures heavily rely on APIs, yet face significant security challenges, particularly with Broken Object Level Authorization (BOLA) vulnerabilities, which remain the most critical API security risk according to OWASP. This paper introduces Karate-BOLA-Guard, an innovative framew...
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10942340/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849763115238948864 |
|---|---|
| author | Emil Marian Pasca Daniela Delinschi Rudolf Erdei Oliviu Matei |
| author_facet | Emil Marian Pasca Daniela Delinschi Rudolf Erdei Oliviu Matei |
| author_sort | Emil Marian Pasca |
| collection | DOAJ |
| description | Modern software architectures heavily rely on APIs, yet face significant security challenges, particularly with Broken Object Level Authorization (BOLA) vulnerabilities, which remain the most critical API security risk according to OWASP. This paper introduces Karate-BOLA-Guard, an innovative framework leveraging Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) techniques to automate security-focused test case generation for APIs. Our approach integrates vector databases for context retrieval, multiple LLM models for test generation, and observability tools for process monitoring. Initial experiments were carried out on three deliberately vulnerable APIs (VAmPI, Crapi, and OWASP Juice Shop), with subsequent validation on fifteen additional production APIs spanning diverse domains including social media, version control systems, financial services, and transportation services. Our evaluation metrics show Llama 3 8B achieving consistent performance (Accuracy: 3.1-3.4, Interoperability: 3.7-4.3) with an average processing time of 143.76 seconds on GPU. Performance analysis revealed significant GPU acceleration benefits, with 20-25x improvement over CPU processing times. Smaller models demonstrated efficient processing, with Phi-3 Mini averaging 69.58 seconds and Mistral 72.14 seconds, while maintaining acceptable accuracy scores. Token utilization patterns showed Llama 3 8B using an average of 36,591 tokens per session, compared to Mistral’s 25,225 and Phi-3 Mini’s 31,007. Our framework’s effectiveness varied across APIs, with notably strong performance in complex platforms (Instagram: A = 4.3, I = 4.4) while maintaining consistent functionality in simpler implementations (VAmPI: A = 3.6, I = 4.3). The iterative refinement process, evaluated through comprehensive metrics including Accuracy (A), Complexity (C), and Interoperability (I), represents a significant advancement in automated API security testing, offering an efficient, accurate, and adaptable approach to detecting BOLA vulnerabilities across diverse API architectures. |
| format | Article |
| id | doaj-art-4203a5ca69274ac0b0b2d2cb75812fb4 |
| institution | DOAJ |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-4203a5ca69274ac0b0b2d2cb75812fb42025-08-20T03:05:31ZengIEEEIEEE Access2169-35362025-01-0113568615688610.1109/ACCESS.2025.355496010942340LLM-Driven, Self-Improving Framework for Security Test Automation: Leveraging Karate DSL for Augmented API ResilienceEmil Marian Pasca0https://orcid.org/0000-0002-0216-6499Daniela Delinschi1https://orcid.org/0000-0001-8582-5842Rudolf Erdei2Oliviu Matei3https://orcid.org/0000-0002-3496-3513Department of Electrical, Electronic and Computer Engineering, Technical University of Cluj Napoca, North University Centre of Baia Mare, Baia Mare, RomaniaDepartment of Electrical, Electronic and Computer Engineering, Technical University of Cluj Napoca, North University Centre of Baia Mare, Baia Mare, RomaniaDepartment of Electrical, Electronic and Computer Engineering, Technical University of Cluj Napoca, North University Centre of Baia Mare, Baia Mare, RomaniaDepartment of Electrical, Electronic and Computer Engineering, Technical University of Cluj Napoca, North University Centre of Baia Mare, Baia Mare, RomaniaModern software architectures heavily rely on APIs, yet face significant security challenges, particularly with Broken Object Level Authorization (BOLA) vulnerabilities, which remain the most critical API security risk according to OWASP. This paper introduces Karate-BOLA-Guard, an innovative framework leveraging Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) techniques to automate security-focused test case generation for APIs. Our approach integrates vector databases for context retrieval, multiple LLM models for test generation, and observability tools for process monitoring. Initial experiments were carried out on three deliberately vulnerable APIs (VAmPI, Crapi, and OWASP Juice Shop), with subsequent validation on fifteen additional production APIs spanning diverse domains including social media, version control systems, financial services, and transportation services. Our evaluation metrics show Llama 3 8B achieving consistent performance (Accuracy: 3.1-3.4, Interoperability: 3.7-4.3) with an average processing time of 143.76 seconds on GPU. Performance analysis revealed significant GPU acceleration benefits, with 20-25x improvement over CPU processing times. Smaller models demonstrated efficient processing, with Phi-3 Mini averaging 69.58 seconds and Mistral 72.14 seconds, while maintaining acceptable accuracy scores. Token utilization patterns showed Llama 3 8B using an average of 36,591 tokens per session, compared to Mistral’s 25,225 and Phi-3 Mini’s 31,007. Our framework’s effectiveness varied across APIs, with notably strong performance in complex platforms (Instagram: A = 4.3, I = 4.4) while maintaining consistent functionality in simpler implementations (VAmPI: A = 3.6, I = 4.3). The iterative refinement process, evaluated through comprehensive metrics including Accuracy (A), Complexity (C), and Interoperability (I), represents a significant advancement in automated API security testing, offering an efficient, accurate, and adaptable approach to detecting BOLA vulnerabilities across diverse API architectures.https://ieeexplore.ieee.org/document/10942340/API securityautomation testing toolscybersecurityrestful APIsoftware testing |
| spellingShingle | Emil Marian Pasca Daniela Delinschi Rudolf Erdei Oliviu Matei LLM-Driven, Self-Improving Framework for Security Test Automation: Leveraging Karate DSL for Augmented API Resilience IEEE Access API security automation testing tools cybersecurity restful API software testing |
| title | LLM-Driven, Self-Improving Framework for Security Test Automation: Leveraging Karate DSL for Augmented API Resilience |
| title_full | LLM-Driven, Self-Improving Framework for Security Test Automation: Leveraging Karate DSL for Augmented API Resilience |
| title_fullStr | LLM-Driven, Self-Improving Framework for Security Test Automation: Leveraging Karate DSL for Augmented API Resilience |
| title_full_unstemmed | LLM-Driven, Self-Improving Framework for Security Test Automation: Leveraging Karate DSL for Augmented API Resilience |
| title_short | LLM-Driven, Self-Improving Framework for Security Test Automation: Leveraging Karate DSL for Augmented API Resilience |
| title_sort | llm driven self improving framework for security test automation leveraging karate dsl for augmented api resilience |
| topic | API security automation testing tools cybersecurity restful API software testing |
| url | https://ieeexplore.ieee.org/document/10942340/ |
| work_keys_str_mv | AT emilmarianpasca llmdrivenselfimprovingframeworkforsecuritytestautomationleveragingkaratedslforaugmentedapiresilience AT danieladelinschi llmdrivenselfimprovingframeworkforsecuritytestautomationleveragingkaratedslforaugmentedapiresilience AT rudolferdei llmdrivenselfimprovingframeworkforsecuritytestautomationleveragingkaratedslforaugmentedapiresilience AT oliviumatei llmdrivenselfimprovingframeworkforsecuritytestautomationleveragingkaratedslforaugmentedapiresilience |