Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback
Evaluating query-passage relevance is a crucial task in information retrieval (IR), where the performance of large language models (LLMs) greatly depends on the quality of prompts. Current prompt optimization methods typically require multiple candidate generations or iterative refinements, resultin...
Saved in:
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-05-01
|
| Series: | Applied Sciences |
| Subjects: | |
| Online Access: | https://www.mdpi.com/2076-3417/15/9/5198 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850032115186401280 |
|---|---|
| author | Jaekeol Choi |
| author_facet | Jaekeol Choi |
| author_sort | Jaekeol Choi |
| collection | DOAJ |
| description | Evaluating query-passage relevance is a crucial task in information retrieval (IR), where the performance of large language models (LLMs) greatly depends on the quality of prompts. Current prompt optimization methods typically require multiple candidate generations or iterative refinements, resulting in significant computational overhead and limited practical applicability. In this paper, we propose a novel prompt optimization method that leverages LLM-based confusion matrix feedback to efficiently optimize prompts for the relevance evaluation task. Unlike previous approaches, our method systematically analyzes LLM predictions—both correct and incorrect—using a confusion matrix, enabling prompt refinement through a single-step update. Our experiments in realistic IR scenarios demonstrate that our method achieves competitive or superior performance compared to existing methods while drastically reducing computational costs, highlighting its potential as a practical and scalable solution. |
| format | Article |
| id | doaj-art-94b093175ccf456a89799f7b71e5c81f |
| institution | DOAJ |
| issn | 2076-3417 |
| language | English |
| publishDate | 2025-05-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Applied Sciences |
| spelling | doaj-art-94b093175ccf456a89799f7b71e5c81f2025-08-20T02:58:44ZengMDPI AGApplied Sciences2076-34172025-05-01159519810.3390/app15095198Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix FeedbackJaekeol Choi0Division of AI Data Convergence, Hankuk University of Foreign Studies, Yongin 17035, Republic of KoreaEvaluating query-passage relevance is a crucial task in information retrieval (IR), where the performance of large language models (LLMs) greatly depends on the quality of prompts. Current prompt optimization methods typically require multiple candidate generations or iterative refinements, resulting in significant computational overhead and limited practical applicability. In this paper, we propose a novel prompt optimization method that leverages LLM-based confusion matrix feedback to efficiently optimize prompts for the relevance evaluation task. Unlike previous approaches, our method systematically analyzes LLM predictions—both correct and incorrect—using a confusion matrix, enabling prompt refinement through a single-step update. Our experiments in realistic IR scenarios demonstrate that our method achieves competitive or superior performance compared to existing methods while drastically reducing computational costs, highlighting its potential as a practical and scalable solution.https://www.mdpi.com/2076-3417/15/9/5198prompt optimizationlarge language modelsrelevance evaluationconfusion matrix feedback |
| spellingShingle | Jaekeol Choi Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback Applied Sciences prompt optimization large language models relevance evaluation confusion matrix feedback |
| title | Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback |
| title_full | Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback |
| title_fullStr | Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback |
| title_full_unstemmed | Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback |
| title_short | Efficient Prompt Optimization for Relevance Evaluation via LLM-Based Confusion Matrix Feedback |
| title_sort | efficient prompt optimization for relevance evaluation via llm based confusion matrix feedback |
| topic | prompt optimization large language models relevance evaluation confusion matrix feedback |
| url | https://www.mdpi.com/2076-3417/15/9/5198 |
| work_keys_str_mv | AT jaekeolchoi efficientpromptoptimizationforrelevanceevaluationviallmbasedconfusionmatrixfeedback |