Data augmentation-assisted deep learning of hand-drawn partially colored sketches for visual search.
In recent years, image databases are growing at exponential rates, making their management, indexing, and retrieval, very challenging. Typical image retrieval systems rely on sample images as queries. However, in the absence of sample query images, hand-drawn sketches are also used. The recent adopt...
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2017-01-01
|
| Series: | PLoS ONE |
| Online Access: | https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0183838&type=printable |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849715570227806208 |
|---|---|
| author | Jamil Ahmad Khan Muhammad Sung Wook Baik |
| author_facet | Jamil Ahmad Khan Muhammad Sung Wook Baik |
| author_sort | Jamil Ahmad |
| collection | DOAJ |
| description | In recent years, image databases are growing at exponential rates, making their management, indexing, and retrieval, very challenging. Typical image retrieval systems rely on sample images as queries. However, in the absence of sample query images, hand-drawn sketches are also used. The recent adoption of touch screen input devices makes it very convenient to quickly draw shaded sketches of objects to be used for querying image databases. This paper presents a mechanism to provide access to visual information based on users' hand-drawn partially colored sketches using touch screen devices. A key challenge for sketch-based image retrieval systems is to cope with the inherent ambiguity in sketches due to the lack of colors, textures, shading, and drawing imperfections. To cope with these issues, we propose to fine-tune a deep convolutional neural network (CNN) using augmented dataset to extract features from partially colored hand-drawn sketches for query specification in a sketch-based image retrieval framework. The large augmented dataset contains natural images, edge maps, hand-drawn sketches, de-colorized, and de-texturized images which allow CNN to effectively model visual contents presented to it in a variety of forms. The deep features extracted from CNN allow retrieval of images using both sketches and full color images as queries. We also evaluated the role of partial coloring or shading in sketches to improve the retrieval performance. The proposed method is tested on two large datasets for sketch recognition and sketch-based image retrieval and achieved better classification and retrieval performance than many existing methods. |
| format | Article |
| id | doaj-art-85e6c87185004ebea9b39c0cb48ac903 |
| institution | DOAJ |
| issn | 1932-6203 |
| language | English |
| publishDate | 2017-01-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS ONE |
| spelling | doaj-art-85e6c87185004ebea9b39c0cb48ac9032025-08-20T03:13:19ZengPublic Library of Science (PLoS)PLoS ONE1932-62032017-01-01128e018383810.1371/journal.pone.0183838Data augmentation-assisted deep learning of hand-drawn partially colored sketches for visual search.Jamil AhmadKhan MuhammadSung Wook BaikIn recent years, image databases are growing at exponential rates, making their management, indexing, and retrieval, very challenging. Typical image retrieval systems rely on sample images as queries. However, in the absence of sample query images, hand-drawn sketches are also used. The recent adoption of touch screen input devices makes it very convenient to quickly draw shaded sketches of objects to be used for querying image databases. This paper presents a mechanism to provide access to visual information based on users' hand-drawn partially colored sketches using touch screen devices. A key challenge for sketch-based image retrieval systems is to cope with the inherent ambiguity in sketches due to the lack of colors, textures, shading, and drawing imperfections. To cope with these issues, we propose to fine-tune a deep convolutional neural network (CNN) using augmented dataset to extract features from partially colored hand-drawn sketches for query specification in a sketch-based image retrieval framework. The large augmented dataset contains natural images, edge maps, hand-drawn sketches, de-colorized, and de-texturized images which allow CNN to effectively model visual contents presented to it in a variety of forms. The deep features extracted from CNN allow retrieval of images using both sketches and full color images as queries. We also evaluated the role of partial coloring or shading in sketches to improve the retrieval performance. The proposed method is tested on two large datasets for sketch recognition and sketch-based image retrieval and achieved better classification and retrieval performance than many existing methods.https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0183838&type=printable |
| spellingShingle | Jamil Ahmad Khan Muhammad Sung Wook Baik Data augmentation-assisted deep learning of hand-drawn partially colored sketches for visual search. PLoS ONE |
| title | Data augmentation-assisted deep learning of hand-drawn partially colored sketches for visual search. |
| title_full | Data augmentation-assisted deep learning of hand-drawn partially colored sketches for visual search. |
| title_fullStr | Data augmentation-assisted deep learning of hand-drawn partially colored sketches for visual search. |
| title_full_unstemmed | Data augmentation-assisted deep learning of hand-drawn partially colored sketches for visual search. |
| title_short | Data augmentation-assisted deep learning of hand-drawn partially colored sketches for visual search. |
| title_sort | data augmentation assisted deep learning of hand drawn partially colored sketches for visual search |
| url | https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0183838&type=printable |
| work_keys_str_mv | AT jamilahmad dataaugmentationassisteddeeplearningofhanddrawnpartiallycoloredsketchesforvisualsearch AT khanmuhammad dataaugmentationassisteddeeplearningofhanddrawnpartiallycoloredsketchesforvisualsearch AT sungwookbaik dataaugmentationassisteddeeplearningofhanddrawnpartiallycoloredsketchesforvisualsearch |