A Process for Measuring Lip Kinematics Using Participants’ Webcams during Linguistic Experiments Conducted Online
Recent advances in automated face-tracking have made it possible for laboratory phonologists to measure lip motion using technology no more advanced than a common webcam. The present paper introduces a lip-tracking approach specifically designed for use in web-based research. The central innovation...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Open Library of Humanities
2024-05-01
|
| Series: | Laboratory Phonology |
| Subjects: | |
| Online Access: | https://www.journal-labphon.org/article/id/10483/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850031623175667712 |
|---|---|
| author | Erik Hunt Peter A. Krause Ryan James Pili |
| author_facet | Erik Hunt Peter A. Krause Ryan James Pili |
| author_sort | Erik Hunt |
| collection | DOAJ |
| description | Recent advances in automated face-tracking have made it possible for laboratory phonologists to measure lip motion using technology no more advanced than a common webcam. The present paper introduces a lip-tracking approach specifically designed for use in web-based research. The central innovation is a custom extension written for jsPsych, an existing JavaScript framework for running behavioral experiments online. This extension gathers data from the participant’s webcam and processes them through FaceMesh, an open-source, JavaScript face-tracker. Face-tracking happens on the fly inside the participant’s browser. Only time-stamped vertical or horizontal lip apertures are saved to the experiment server. That is, this extension allows experiments implemented in jsPsych to collect de-identified lip kinematics from participants seated at their own home computers. After explaining the core functionality of the extension, this paper presents two validation experiments. The first establishes that utterances with different lip kinematics result in very different lip aperture trajectories, and that the timing of a key kinematic landmark agrees with the timing of acoustic landmarks obtained for the same utterances. The second experiment replicates a vowel-priming result previously demonstrated using a prior face-tracking system and saved facial video. All associated scripts have been made publicly available. |
| format | Article |
| id | doaj-art-6645618bbef549d3aefa6afb707ee0d1 |
| institution | DOAJ |
| issn | 1868-6354 |
| language | English |
| publishDate | 2024-05-01 |
| publisher | Open Library of Humanities |
| record_format | Article |
| series | Laboratory Phonology |
| spelling | doaj-art-6645618bbef549d3aefa6afb707ee0d12025-08-20T02:58:55ZengOpen Library of HumanitiesLaboratory Phonology1868-63542024-05-0115110.16995/labphon.10483A Process for Measuring Lip Kinematics Using Participants’ Webcams during Linguistic Experiments Conducted OnlineErik Hunt0Peter A. Krause1https://orcid.org/0000-0002-8718-9824Ryan James Pili2Psychology, California State University Channel IslandsPsychology, California State University Channel IslandsPsychology, University of California Santa CruzRecent advances in automated face-tracking have made it possible for laboratory phonologists to measure lip motion using technology no more advanced than a common webcam. The present paper introduces a lip-tracking approach specifically designed for use in web-based research. The central innovation is a custom extension written for jsPsych, an existing JavaScript framework for running behavioral experiments online. This extension gathers data from the participant’s webcam and processes them through FaceMesh, an open-source, JavaScript face-tracker. Face-tracking happens on the fly inside the participant’s browser. Only time-stamped vertical or horizontal lip apertures are saved to the experiment server. That is, this extension allows experiments implemented in jsPsych to collect de-identified lip kinematics from participants seated at their own home computers. After explaining the core functionality of the extension, this paper presents two validation experiments. The first establishes that utterances with different lip kinematics result in very different lip aperture trajectories, and that the timing of a key kinematic landmark agrees with the timing of acoustic landmarks obtained for the same utterances. The second experiment replicates a vowel-priming result previously demonstrated using a prior face-tracking system and saved facial video. All associated scripts have been made publicly available.https://www.journal-labphon.org/article/id/10483/articulatory measurementlip trackingonline data collectionphoneticspsycholinguistics |
| spellingShingle | Erik Hunt Peter A. Krause Ryan James Pili A Process for Measuring Lip Kinematics Using Participants’ Webcams during Linguistic Experiments Conducted Online Laboratory Phonology articulatory measurement lip tracking online data collection phonetics psycholinguistics |
| title | A Process for Measuring Lip Kinematics Using Participants’ Webcams during Linguistic Experiments Conducted Online |
| title_full | A Process for Measuring Lip Kinematics Using Participants’ Webcams during Linguistic Experiments Conducted Online |
| title_fullStr | A Process for Measuring Lip Kinematics Using Participants’ Webcams during Linguistic Experiments Conducted Online |
| title_full_unstemmed | A Process for Measuring Lip Kinematics Using Participants’ Webcams during Linguistic Experiments Conducted Online |
| title_short | A Process for Measuring Lip Kinematics Using Participants’ Webcams during Linguistic Experiments Conducted Online |
| title_sort | process for measuring lip kinematics using participants webcams during linguistic experiments conducted online |
| topic | articulatory measurement lip tracking online data collection phonetics psycholinguistics |
| url | https://www.journal-labphon.org/article/id/10483/ |
| work_keys_str_mv | AT erikhunt aprocessformeasuringlipkinematicsusingparticipantswebcamsduringlinguisticexperimentsconductedonline AT peterakrause aprocessformeasuringlipkinematicsusingparticipantswebcamsduringlinguisticexperimentsconductedonline AT ryanjamespili aprocessformeasuringlipkinematicsusingparticipantswebcamsduringlinguisticexperimentsconductedonline AT erikhunt processformeasuringlipkinematicsusingparticipantswebcamsduringlinguisticexperimentsconductedonline AT peterakrause processformeasuringlipkinematicsusingparticipantswebcamsduringlinguisticexperimentsconductedonline AT ryanjamespili processformeasuringlipkinematicsusingparticipantswebcamsduringlinguisticexperimentsconductedonline |