Referring expression is a kind of language expression that used for referring to particular objects. To make the expression without ambiguation, people often use attributes to describe the particular object. In this paper, we explore the role of attributes by incorporating them into both referring expression generation and comprehension. We first train an attribute learning model from visual objects and their paired descriptions. Then in the generation task, we take the learned attributes as the input into the generation model, thus the expressions are generated driven by both attributes and the previous words. For comprehension, we embed the learned attributes with visual features and semantics into the common space model, then the target object is retrieved based on its ranking distance in the common space. Experimental results on the three standard datasets, RefCOCO, RefCOCO+, and RefCOCOg show significant improvements over the baseline model, demonstrating that our method is effective for both tasks.
|Title of host publication||Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017|
|Publisher||Institute of Electrical and Electronics Engineers Inc.|
|Number of pages||9|
|Publication status||Published - 2017 Dec 22|
|Event||16th IEEE International Conference on Computer Vision, ICCV 2017 - Venice, Italy|
Duration: 2017 Oct 22 → 2017 Oct 29
|Name||Proceedings of the IEEE International Conference on Computer Vision|
|Other||16th IEEE International Conference on Computer Vision, ICCV 2017|
|Period||17/10/22 → 17/10/29|
Bibliographical noteFunding Information:
This work is supported by 1) the NSF CAREER Grant #1149783, gifts from Adobe and NVIDIA. 2) National Key Research and Development Program of China (2016YFB1001000), National Natural Science Foundation of China (61525306, 61633021, 61572504, 61420106015), Strategic Priority Research Program of the CAS (XDB02070001), and Beijing Natural Science Foundation (4162058). 3) grants from NVIDIA and the NVIDIA DGX-1 AI Supercomputer.
© 2017 IEEE.
All Science Journal Classification (ASJC) codes
- Computer Vision and Pattern Recognition