Research Article
Mobile App for Text-to-Image Synthesis
@INPROCEEDINGS{10.1007/978-3-030-28468-8_3, author={Ryan Kang and Athira Sunil and Min Chen}, title={Mobile App for Text-to-Image Synthesis}, proceedings={Mobile Computing, Applications, and Services. 10th EAI International Conference, MobiCASE 2019, Hangzhou, China, June 14--15, 2019, Proceedings}, proceedings_a={MOBICASE}, year={2019}, month={9}, keywords={Text-to-Image Image-to-Text Mobile application ImageNet WordNet RESTful API Speech recognition English language education}, doi={10.1007/978-3-030-28468-8_3} }- Ryan Kang
Athira Sunil
Min Chen
Year: 2019
Mobile App for Text-to-Image Synthesis
MOBICASE
Springer
DOI: 10.1007/978-3-030-28468-8_3
Abstract
Generating visual representation of textual information is a challenging yet interesting topic with many potential applications. In this paper, we propose a novel approach to visualize natural language sentences using ImageNet to enhance language education. Currently the focus is to assist English language learners in building their vocabulary of common nouns and developing an in-depth understanding of the various prepositions of locations. To achieve this goal, real-world images representing nouns are obtained from ImageNet and their foreground objects of interest are extracted using image segmentation. The objects are then re-arranged on a canvas based on their spatial relationship specified in the sentence. To demonstrate the effectiveness of the proposed approach, we have developed a mobile application that uses the RESTful API to retrieve the images from the web service that operate the image generation program. The prototype mobile application can create visual representations of natural language sentences and a text description of the spatial relationship of objects to assist in learning new vocabulary and spatial prepositions during language education.