AirStrum: A virtual guitar using real-time hand gesture recognition and strumming technique - Beulah ARUL; Shashank PANDA; Tushar NAIR (Revista Română de Informatică și Automatică)
The efforts in the field of virtualizing the guitar into well-modelled software systems have faced a lot of practical limitations. The existing guitar simulation programs require additional devices such as Electromyography (EMG) controllers, or Musical Instrument Digital Interface (MIDI)-based recording devices. The EMG-based device is still a work in progress, the device is expensive and it was not very well received by the instrumentalists. There exists a gap in the bridge that joins physical instruments to their software counterparts. In this context, this paper aims to significantly remove the inaccuracies and drawbacks related to the existing solutions by accounting for the individual roles that each hand plays in the act of guitar strumming and consolidating them into a single system. The design of the proposed AirStrum system involves a multi-step process. Initially, a dataset is created by recording images of hand gestures corresponding to the playing of various chords on a guitar. The palm is detected, and its related skeleton image is generated using MediaPipe. Subsequently, a model based on a Convolutional Neural Network (CNN) is trained and validated using the employed dataset to adeptly recognize and classify guitar chords. Additionally, this model incorporates a velocity detection function for the strumming hand. Finally, the proposed system can play different sounds by inferring both the played chord and the strumming velocity from human actions. This comprehensive approach enables a sophisticated virtual guitar experience based on a system that responds dynamically to the users' gestures and strumming techniques. The conducted experiments demonstrate that AirStrum achieves an accuracy of 95.92%, and a brief preliminary survey related to its perceived utility and usability received a positive feedback rate of 58% from eight guitar players.