FPGA Acceleration With Hessian-Based Comprehensive Intra-Layer Mixed-Precision Quantization for Transformer Models
Recent advancements in using FPGAs as co-processors for language model acceleration, particularly for energy efficiency and flexibility, face challenges due to limited memory capacity. This limitation hinders the deployment of transformer-based language models. To address this challenge, we propose...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2025-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10973048/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849310457318342656 |
|---|---|
| author | Woohong Byun Jongseok Woo Saibal Mukhopadhyay |
| author_facet | Woohong Byun Jongseok Woo Saibal Mukhopadhyay |
| author_sort | Woohong Byun |
| collection | DOAJ |
| description | Recent advancements in using FPGAs as co-processors for language model acceleration, particularly for energy efficiency and flexibility, face challenges due to limited memory capacity. This limitation hinders the deployment of transformer-based language models. To address this challenge, we propose a novel software-hardware co-optimization framework that integrates Hessian-based intra-layer mixed-precision quantization with a runtime bit-configurable FPGA accelerator. Our proposed Hessian-based row-wise weight quantization addresses hardware inefficiencies in traditional parameter-wise and channel-wise approaches by enabling mixed-precision weight matrices to be split into two uniform-precision matrices, thereby simplifying hardware requirements. Additionally, our Query-Key coupled attention activation quantization optimally aligns precision within each outer product pair in attention calculations, reducing hardware complexity and memory management overhead. Our concurrent quantization method balances the benefits of row-wise weight quantization and Query-Key coupled activation quantization while maximizing energy efficiency through multi-precision optimization. To support this algorithm, we design a multi-precision FPGA accelerator capable of handling both 2n-based and non-2n mixed-precision operations. It is implemented on a single Xilinx ZCU102 FPGA board, operating at 200MHz with a power consumption of 15.08W during inference on the 110-million-parameter BERT-Base and 345-million-parameter GPT-2 Medium transformer models. Coupled with the proposed algorithm and dataflow optimization, it enables on-chip storage of all necessary parameters, minimizing off-chip memory access. Experimental results demonstrate that our FPGA accelerator significantly outperforms existing solutions, achieving energy efficiency improvements ranging from <inline-formula> <tex-math notation="LaTeX">$2.22\times $ </tex-math></inline-formula> to <inline-formula> <tex-math notation="LaTeX">$17.23\times $ </tex-math></inline-formula> over state-of-the-art FPGA accelerators. |
| format | Article |
| id | doaj-art-da9ffb2491b54cb28451692fa6253c77 |
| institution | Kabale University |
| issn | 2169-3536 |
| language | English |
| publishDate | 2025-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-da9ffb2491b54cb28451692fa6253c772025-08-20T03:53:42ZengIEEEIEEE Access2169-35362025-01-0113702827029710.1109/ACCESS.2025.356319610973048FPGA Acceleration With Hessian-Based Comprehensive Intra-Layer Mixed-Precision Quantization for Transformer ModelsWoohong Byun0https://orcid.org/0009-0003-5087-4582Jongseok Woo1Saibal Mukhopadhyay2https://orcid.org/0000-0002-8894-3390School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USAMobile Display Design Team, Samsung Display Company Ltd., Yongin, South KoreaSchool of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USARecent advancements in using FPGAs as co-processors for language model acceleration, particularly for energy efficiency and flexibility, face challenges due to limited memory capacity. This limitation hinders the deployment of transformer-based language models. To address this challenge, we propose a novel software-hardware co-optimization framework that integrates Hessian-based intra-layer mixed-precision quantization with a runtime bit-configurable FPGA accelerator. Our proposed Hessian-based row-wise weight quantization addresses hardware inefficiencies in traditional parameter-wise and channel-wise approaches by enabling mixed-precision weight matrices to be split into two uniform-precision matrices, thereby simplifying hardware requirements. Additionally, our Query-Key coupled attention activation quantization optimally aligns precision within each outer product pair in attention calculations, reducing hardware complexity and memory management overhead. Our concurrent quantization method balances the benefits of row-wise weight quantization and Query-Key coupled activation quantization while maximizing energy efficiency through multi-precision optimization. To support this algorithm, we design a multi-precision FPGA accelerator capable of handling both 2n-based and non-2n mixed-precision operations. It is implemented on a single Xilinx ZCU102 FPGA board, operating at 200MHz with a power consumption of 15.08W during inference on the 110-million-parameter BERT-Base and 345-million-parameter GPT-2 Medium transformer models. Coupled with the proposed algorithm and dataflow optimization, it enables on-chip storage of all necessary parameters, minimizing off-chip memory access. Experimental results demonstrate that our FPGA accelerator significantly outperforms existing solutions, achieving energy efficiency improvements ranging from <inline-formula> <tex-math notation="LaTeX">$2.22\times $ </tex-math></inline-formula> to <inline-formula> <tex-math notation="LaTeX">$17.23\times $ </tex-math></inline-formula> over state-of-the-art FPGA accelerators.https://ieeexplore.ieee.org/document/10973048/AcceleratoractivationBERTFPGAhardware accelerationHessian |
| spellingShingle | Woohong Byun Jongseok Woo Saibal Mukhopadhyay FPGA Acceleration With Hessian-Based Comprehensive Intra-Layer Mixed-Precision Quantization for Transformer Models IEEE Access Accelerator activation BERT FPGA hardware acceleration Hessian |
| title | FPGA Acceleration With Hessian-Based Comprehensive Intra-Layer Mixed-Precision Quantization for Transformer Models |
| title_full | FPGA Acceleration With Hessian-Based Comprehensive Intra-Layer Mixed-Precision Quantization for Transformer Models |
| title_fullStr | FPGA Acceleration With Hessian-Based Comprehensive Intra-Layer Mixed-Precision Quantization for Transformer Models |
| title_full_unstemmed | FPGA Acceleration With Hessian-Based Comprehensive Intra-Layer Mixed-Precision Quantization for Transformer Models |
| title_short | FPGA Acceleration With Hessian-Based Comprehensive Intra-Layer Mixed-Precision Quantization for Transformer Models |
| title_sort | fpga acceleration with hessian based comprehensive intra layer mixed precision quantization for transformer models |
| topic | Accelerator activation BERT FPGA hardware acceleration Hessian |
| url | https://ieeexplore.ieee.org/document/10973048/ |
| work_keys_str_mv | AT woohongbyun fpgaaccelerationwithhessianbasedcomprehensiveintralayermixedprecisionquantizationfortransformermodels AT jongseokwoo fpgaaccelerationwithhessianbasedcomprehensiveintralayermixedprecisionquantizationfortransformermodels AT saibalmukhopadhyay fpgaaccelerationwithhessianbasedcomprehensiveintralayermixedprecisionquantizationfortransformermodels |