A Novel Sentence-Level Visual Speech Recognition System for Vietnamese Language Using ResNet3D and Zipformer - Phat Nguyen Huu; Thach Ho Sy (Modelling and Simulation in Engineering)
This paper presents the first sentence-level visual speech recognition (VSR) system specifically designed for the Vietnamese language. We have developed a unique dataset comprising 115 h of video recordings from over 100 speakers, focusing on single-speaker scenarios. The proposed VSR system utilizes a ResNet3D architecture as the visual frontend, paired with a neural transducer framework featuring a Zipformer speech encoder. It incorporates a stateless decoder that considers two preceding tokens and is optimized with a pruned-RNNT loss function. Experimental results show that our system achieves a word error rate (WER) of 27.14% and a character error rate (CER) of 20.45% on single-speaker tasks, demonstrating significant progress in VSR for Vietnamese.