Fine-Grained Length Controllable Video Captioning With Ordinal Embeddings
This paper proposes a method for video captioning that controls the length of generated captions. Previous work on length control often had few levels for expressing length. In this study, we propose two methods of length embedding for fine-grained length control. A traditional embedding method is l...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
IEEE
2024-01-01
|
| Series: | IEEE Access |
| Subjects: | |
| Online Access: | https://ieeexplore.ieee.org/document/10767711/ |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850052974204682240 |
|---|---|
| author | Tomoya Nitta Takumi Fukuzawa Toru Tamaki |
| author_facet | Tomoya Nitta Takumi Fukuzawa Toru Tamaki |
| author_sort | Tomoya Nitta |
| collection | DOAJ |
| description | This paper proposes a method for video captioning that controls the length of generated captions. Previous work on length control often had few levels for expressing length. In this study, we propose two methods of length embedding for fine-grained length control. A traditional embedding method is linear, using a one-hot vector and an embedding matrix. In this study, we propose methods that represent length in multi-hot vectors. One is bit embedding that expresses length in bit representation, and the other is ordinal embedding that uses the binary representation often used in ordinal regression. These length representations of multi-hot vectors are converted into length embedding by a nonlinear MLP. This method allows for not only the length control of caption sentences but also the control of the time when reading the caption. Experiments using ActivityNet Captions and Spoken Moments in Time show that the proposed method effectively controls the length of the generated captions. Analysis of the embedding vectors with Independent Component Analysis (ICA) shows that length and semantics were learned separately, demonstrating the effectiveness of the proposed embedding methods. Our code and online demo are available at <uri>https://huggingface.co/spaces/fztkm/length_controllable_video_captioning</uri>. |
| format | Article |
| id | doaj-art-6dee2fe280a34124a598531ff97df3d6 |
| institution | DOAJ |
| issn | 2169-3536 |
| language | English |
| publishDate | 2024-01-01 |
| publisher | IEEE |
| record_format | Article |
| series | IEEE Access |
| spelling | doaj-art-6dee2fe280a34124a598531ff97df3d62025-08-20T02:52:41ZengIEEEIEEE Access2169-35362024-01-011218966718968810.1109/ACCESS.2024.350675110767711Fine-Grained Length Controllable Video Captioning With Ordinal EmbeddingsTomoya Nitta0Takumi Fukuzawa1Toru Tamaki2https://orcid.org/0000-0001-9712-7777Toshiba, Kawasaki, JapanNagoya Institute of Technology, Nagoya, JapanNagoya Institute of Technology, Nagoya, JapanThis paper proposes a method for video captioning that controls the length of generated captions. Previous work on length control often had few levels for expressing length. In this study, we propose two methods of length embedding for fine-grained length control. A traditional embedding method is linear, using a one-hot vector and an embedding matrix. In this study, we propose methods that represent length in multi-hot vectors. One is bit embedding that expresses length in bit representation, and the other is ordinal embedding that uses the binary representation often used in ordinal regression. These length representations of multi-hot vectors are converted into length embedding by a nonlinear MLP. This method allows for not only the length control of caption sentences but also the control of the time when reading the caption. Experiments using ActivityNet Captions and Spoken Moments in Time show that the proposed method effectively controls the length of the generated captions. Analysis of the embedding vectors with Independent Component Analysis (ICA) shows that length and semantics were learned separately, demonstrating the effectiveness of the proposed embedding methods. Our code and online demo are available at <uri>https://huggingface.co/spaces/fztkm/length_controllable_video_captioning</uri>.https://ieeexplore.ieee.org/document/10767711/Video captioninglength controllable generationordinal embedding |
| spellingShingle | Tomoya Nitta Takumi Fukuzawa Toru Tamaki Fine-Grained Length Controllable Video Captioning With Ordinal Embeddings IEEE Access Video captioning length controllable generation ordinal embedding |
| title | Fine-Grained Length Controllable Video Captioning With Ordinal Embeddings |
| title_full | Fine-Grained Length Controllable Video Captioning With Ordinal Embeddings |
| title_fullStr | Fine-Grained Length Controllable Video Captioning With Ordinal Embeddings |
| title_full_unstemmed | Fine-Grained Length Controllable Video Captioning With Ordinal Embeddings |
| title_short | Fine-Grained Length Controllable Video Captioning With Ordinal Embeddings |
| title_sort | fine grained length controllable video captioning with ordinal embeddings |
| topic | Video captioning length controllable generation ordinal embedding |
| url | https://ieeexplore.ieee.org/document/10767711/ |
| work_keys_str_mv | AT tomoyanitta finegrainedlengthcontrollablevideocaptioningwithordinalembeddings AT takumifukuzawa finegrainedlengthcontrollablevideocaptioningwithordinalembeddings AT torutamaki finegrainedlengthcontrollablevideocaptioningwithordinalembeddings |